Apply OpenAPI and SwaggerUI to Phoenix Web

Documentation matters

Documentation is a love-hate relationship for developers. For the developer doing the documentation, it’s a chore that often takes up more time than writing code, doesn’t reward them in the short term, and isn’t something they can motivate themselves to do unless someone (usually a manager) encourages or forces them to do it. On the other hand, if you’re a developer who’s been thrown into an ongoing project and has to learn a lot of new things, or if you’ve suddenly taken over someone else’s work and have to maintain it, good documentation can feel like an oasis in the desert. Especially when you’re communicating between different areas, documentation is essential. For example, if a mobile client developer collaborates with a backend server developer, they’ll often develop based on API documentation.

Docs as Code

Having said that, documentation like this is hard to do diligently unless the person writing it is particularly motivated, and even then it’s not an easy task. Poorly written documentation can actually confuse the reader.

At Google, they said they’ve found that the most effective way to improve this critical piece of documentation is to “treat it like code” and integrate it into the engineering workflow. By treating writing and managing documentation the same as writing and maintaining code, they were able to weave documentation into the development process.

One way to manage documentation like code is to automatically documents the comments that the developer writes on code. This is a great way to integrate documentation into the development process, as you naturally modify comments as you maintain annotated modules and functions. However, it’s not very effective if you modify a function but don’t modify the function’s comments, or if you don’t realize you’ve documented it incorrectly. To avoid this, there are ways to combine documentation with testing. For example, the Elixir’s documentation tool, ex_doc, provides the ability to validate example code in function comments against tests via a macro called doctest.

Another area where combining code and documentation works well might be HTTP API documentation, because the API documentation is somewhat formatted and has a lot of overlap with the code, so there are a lot of related tools out there. The most famous tool would be (formerly Swagger) OpenAPI Specification(OAS).

OpenAPI Specification

OpenAPI or OpenAPI Specification (OAS) is the name given to Swagger when it passed to the OpenAPI Initiative in 2015. You could say that Swagger 2.0 led to OpenAPI 3.0, and as it addresses issues with Swagger 2.0, it serves as a very convenient tool for documenting APIs. While the OAS is an API documentation convention, tools like SwaggerUI can present it as a web document and even provide features like API test requests on the fly.

Web-Framework-Specific OAS Library

The original OAS specification is in the form of a json or yaml file, and each programming language has libraries that combine this with specific web frameworks to seamlessly combine code and specs. The advantage of combining code and API documentation in this way is that documentation management can go hand in hand with code management. This means that as you modify your API logic in branches and commits, you can also modify your documentation.

Such a library exists in Spring, for example, with SpringDoc. You can see how they use annotations to combine specs into existing controller method, as shown following code. Typically, the library will convert these combined specs into JSON format specs. As long as you have the converted specification, it’s very simple to visualize the API documentation web page with swagger-ui.

@RestController
@RequestMapping("/api/auth")
@Tag(name = "auth", description = "the authentication api")
public class AuthController {

    ...

    @Operation(summary = "login", description = "login by id password")
    @ApiResponses(value = {
        @ApiResponse(responseCode = "200", description = "successful operation", 
                  content = @Content(schema = @Schema(implementation = LoginResponse.class))),
        @ApiResponse(responseCode = "400", description = "bad request operation", 
                  content = @Content(schema = @Schema(implementation = LoginResponse.class)))
    })
    @PostMapping(value = "login", produces = { "application/json" })
    public ResponseEntity<LoginResponse> login(LoginRequest loginRequest) {
        LoginResponse loginResponse = new LoginResponse("accessTokenValue", "refreshTokenValue");
        return ResponseEntity.ok().body(loginResponse);
    }
}

open_api_spex library of Elixir

Elixir also has such a library, open_api_spex. It’s a library that combines with Phoenix, Elixir’s flagship web framework, and it’s actually the one I’ve adopted for my team. Using that library, the same controller above could be represented like:

defmodule MyAppWeb.AuthController do
  use MyAppWeb, :controller
  use OpenApiSpex.ControllerSpec

  alias MyAppWeb.Schemas.{LoginRequest, LoginResponse, ErrorResponse}

  tags ["auth"]

  operation :login,
    summary: "login",
    description: "login by id password",
    request_body: {"login params", "application/json", LoginRequest},
    responses: [
      ok: {"successful operation", "application/json", LoginResponse},
      bad_request: {"bad request operation", "application/json", ErrorResponse}
    ]

  def login(conn, _params) do
    ...
    render(conn, login_response)
  end
end

The open_api_spex library uses macros to combine controllers and specifications. The tags and operation macros are both syntaxes defined by the library to represent the OpenAPI Spec.

You can read more about the library’s usage in its documentation. The names of the macros and their respective terms mostly follow the OAS official documentation.

open_api_spex has two special abilities

In addition to this ability to combine documentation with code, open_api_spex provides two more special features. One is the ability to automatically perform validation on request parameters defined by the specification, and the other is the ability to add response validation to your tests to verify that the specification in the documentation matches the actual logic results. Let’s take a closer look at these two features.

Ability of CastAndValidate plug

First up is the ability to validate parameters. I mentioned earlier that HTTP API documentation is great to combine with code, and this is evident in parameter validation, where you can define the specification of a parameter in OAS format, and it will appear in the swagger-ui documentation, while at the same time validating it with the same conditions when processing the actual web request. For example, you can specify which parameters are required, where the range of an integer parameter starts and ends, the maximum length of a string parameter, and so on, and these become the validation check conditions for the controller action. The open_api_spex provides this validation and also a casting function that is very useful in dynamic languages like elixir. Casting is simply identifying and converting the type of each parameter field, such as string, integer, boolean, etc. It can also identify maps, arrays, and put in custom castings, so it is more powerful.

Let’s take a look at an example. Without open_api_spex in Phoenix project, we used to put parameter validation in part of each controller’s logic as shown below. Of course, it’s common practice to collect these helper functions in a utility module for reusing.

defmodule MyAppWeb.UsersController do
  use MyAppWeb, :controller

  def show(conn, %{"id" => id}) do
    with {:ok, id} <- validate_integer(id),
      ... do
      render(conn, %{user: user})
    else 
      err ->
        render_error(conn, err)
    end
  end
  
  # if `id` is a query parameter, this validation of missing parameters is needed
  def show(conn, _params) do
    render_error(conn, {:missing_parameter, "id is missing"})
  end

  defp validate_integer(id) do
    id = String.to_integer(id)
    if id > 0 do
      {:ok, id}
    else
      {:invalid_parameter, "#{id} must greater than 0"}
    end
  rescue 
    ArgumentError -> {:invalid_parameter, "#{id} is not valid integer"}
  end
end

While this isn’t necessarily inconvenient for simple parameter validation, it can become progressively more logic heavy and unmanageable for complex parameter validation. In particular, it has the potential to not match the API documentation exactly.

The CastAndValidate plug in open_api_spex allows you to perform parameter validation with the declared specification before the controller’s action is called, and to call a given error handler if the validation fails. Let’s see an example.

defmodule MyAppWeb.UsersController do
  use MyAppWeb, :controller
  use OpenApiSpex.ControllerSpec 

  alias MyAppWeb.Schemas.Commons.ID
  alias MyAppWeb.Schemas.UserResponse

  plug OpenApiSpex.Plug.CastAndValidate, json_render_error_v2: true

  operation :show,
    summary: "get a user",
    parameters: [
      id: [in: :path, description: "user ID", schema: ID]
    ],
    responses: %{
      200 => {"User", "application/json", UserResponse},
      400 -> OpenApiSpex.JsonErrorResponse.response()
    }

  def show(conn, %{id: id}) do
    with {:ok, user} <- ... do
      render(conn, %{user: user})
    else
      err ->
        render_error(conn, err)
    end
  end
end

For commonly used ID type parameter, we’ve defined them as schema components. The specification in the schema is then validated by the CastAndValidate plug. If you declare a schema like this, even a complex JSON body can be validated against the specification.

defmodule MyAppWeb.Schemas.UpdateUserParams do
  require OpenApiSpex
  alias OpenApiSpex.Schema

  OpenApiSpex.schema(%{
    title: "UpdateUserParams",
    required: [:name, :email],
    properties: %{
      id: %Schema{type: :integer}
      name: %Schema{type: :string, pattern: ~r/[a-zA-Z][a-zA-Z0-9_]+/},
      email: %Schema{type: :string, format: :email}
    }
  })
end

Ability of validation of response Spec

Now that we’ve seen validation for parameters, let’s take a look at validation for API responses. Unlike validation of parameters, validation of response values is not done in a production environment. It is not necessary to validate the response schema for each API request. Instead, the response schema can be validated in unit tests. Where you would normally write unit tests for controllers in Phoenix, you can use the schema validation tools provided by the library to verify that the API’s response matches the schema.

use MyAppWeb.ConnCase
import OpenApiSpex.TestAssertions

test "UserController produces a UsersResponse", %{conn: conn} do
  json =
    conn
    |> get(user_path(conn, :index))
    |> json_response(200)


  api_spec = MyAppWeb.ApiSpec.spec()
  assert_schema(json, "UsersResponse", api_spec)
end

By adding tests that validate these responses, you can ensure that your API documentation is always up to date by ensuring that if changes are made to the API in the future, the tests will fail if the schema is missing those changes. This is a practical example of how documentation and testing can be combined to ensure that your documentation is up to date.

These two special abilities are why open_api_spex is such a useful tool. It simplifies the logic of the controller by letting you document the specification you declare to the client through parameter validation, while automatically giving error responses for invalid arguments. This also aligns with the pragmatic programmer‘s “DRY” principle. In the book The Pragmatic Programmer, the DRY principle is stated as follows.

every piece of knowledge must have a single, unambiguous, authoritative representation within a system

The Pragmatic Programmer expands on this principle in the revised edition, noting that DRY is not just about duplication of code. If you change one aspect of your code and you change it in multiple places, it’s not DRY. With open_api_spex, you’re representing a single piece of knowledge - the API specification - in a single place, and you can control the parts of your code that relate to that knowledge at once.

Customize open_api_spex: schema definition

While open_api_spex is a great tool for adhering to these DRY principles, as an open-source library, there is still a lot of room for improvement. In particular, I don’t think the syntax is very comfortable, and while the library provides some optional syntaxes for defining specifications, I kept thinking that it could be easier and more intuitive to use.

Elixir’s ecto and absinthe libraries provide similar syntax for expressing database schema and graphql schema, respectively, and have very intuitive syntax. The schema representation for ecto is as follows.

defmodule User do
  use Ecto.Schema

  schema "users" do
    field :name, :string
    field :age, :integer
  end
end

It would be nice to be able to write these syntaxes similarly, so I tried defining them myself via macros. For example, the following schema represented above,

defmodule MyAppWeb.Schemas.UpdateUserParams do
  require OpenApiSpex
  alias OpenApiSpex.Schema

  OpenApiSpex.schema(%{
    title: "UpdateUserParams",
    required: [:name, :email],
    properties: %{
      id: %Schema{type: :integer}
      name: %Schema{type: :string, pattern: ~r/[a-zA-Z][a-zA-Z0-9_]+/},
      email: %Schema{type: :string, format: :email}
    }
  })
end

could be like this:

defmodule MyAppWeb.Schemas.UpdateUserParams do
  use OpenApiSpex.Schemax

  @required [:name, :email]
  schema "UpdateUserParams" do
    property :id, :integer
    property :name, :string, pattern: ~r/[a-zA-Z][a-zA-Z0-9_]+/
    property :email, :string, format: :email
  end
end

While it may not seem like much of a difference, I think it serves the purpose of making it look more like the schema syntax of other existing libraries to eliminate disparity and improve readability, and I’ve actually had people on my team tell me that they use the syntax I defined and they think it looks better.

I’ve also defined some of my own usage with macros to guide my team. For example, showing operations together in controller didn’t distribute the logic properly in huge legacy controllers, resulting in too much code, so I’ve taken steps to separate the controller and operation logic for readability. The beauty of elixir is that, when used properly, these macros allow you to bend the arbitrary syntax of the library to you will.

Conclusion

In this article, I introduced Docs as Code, a way to help developers with their love-hate relationship with documentation, and an example of how to combine API documentation with code. With a structured specification tool like OpenApi Spec, and a library like open_api_spex with a special feature to provide schema validation, developers can “DRY” and efficiently develop APIs. I also customized the library’s syntax to be similar to other schema syntaxes through a handy macro in Elixir to further improve usability.

The open_api_spex library is actually a library that I carefully selected and introduced to the team, and I tried to devise various macros to improve its usability. Now, the team members are gradually recognizing its convenience and usefulness, and I want to use it better as I have been thinking about it for a long time, so I will try various things in the future.

According to the book Software Engineering at Google, the state of documentation in the late 2010s, when the book was written, was similar to the state of software testing in the late 1980s.

Everyone recognizes that more effort needs to be made to improve it, but there is not yet organizational recognition of its critical benefits.

I believe that documentation is part of the development process and that there are increasingly optimal ways to make it happen efficiently in parallel with development. This article is one of them, and it’s part of the process of becoming a pragmatic programmer.