All articles
Product

Catching breaking changes with apigenie diff

Andriy Mykulyak · · 6 min read


Why I care about breaking changes

When you work API-first, the description of the API is the contract. Clients are generated from it, mocks answer from it, and other teams plan their work around it. That is the whole point, and it is also the risk: a small edit to the description can break code you have never seen.

The edits that do the damage rarely look dangerous. Someone removes an operation nobody seems to use. Someone makes a query parameter required, or renames a Protobuf field, or tightens a GraphQL argument type. In review, these are one-line changes in a long YAML or .proto file, and they are easy to approve.

I wanted a tool that reads both versions of the description, tells me exactly what changed, and says which of those changes will hurt consumers. That tool is apigenie diff.

What it compares

The command takes two files, the old one and the new one:

apigenie diff [OPTIONS] SOURCE_FILE DEST_FILE

Both files must be of the same kind: both OpenAPI 3.x documents, both Swagger 2.0 documents, both AsyncAPI 3.x documents, both Protobuf files or both GraphQL files. The differ picks the format from the file itself: a swagger field at the root means Swagger, an asyncapi field means AsyncAPI, a .proto extension means Protobuf, and .graphql, .graphqls or .gql means GraphQL.

Each format is compared the way its consumers see it:

  • OpenAPI and Swagger. Paths, operations, parameters, request bodies, responses, schemas, security and so on. The severity of a schema change depends on the direction of the data: a change in a request is judged by what it does to callers, a change in a response by what it does to readers. Read-only properties don’t matter in requests.
  • AsyncAPI. Servers, channels, operations, messages with their payloads and headers, security schemes, bindings and components. Here the direction is decided by who sends the message. A new required payload property breaks the senders of a message the application receives, while the receivers of a message the application sends can ignore it.
  • Protobuf. Messages, enums, services and extend blocks, down to fields, enum values and methods. Fields and enum values are matched by their numbers, because the numbers identify them in the encoded data. So a renamed field is a modified field, and a field that got another number is a removed one plus an added one. Removing a field is non-breaking when the new file reserves its number, which is how Protobuf retires fields.
  • GraphQL. The type system: root operation types, types of every kind with their fields, arguments and enum values, directive definitions and applied directives. Type extensions are merged into what they extend, so moving a field into an extend type block is not a change. Breaking changes follow what graphql-js’s findBreakingChanges() decides.

The diff reference describes the matching rules of each format in detail.

Severities

Every change gets one of three severities:

  • breaking: existing consumers stop working, for example a removed operation or a field whose type changed incompatibly;
  • non-breaking: the API changes, but existing consumers keep working, for example an added operation or an added optional parameter;
  • non-functional: nothing changes for the programs involved, for example descriptions in GraphQL or comments in Protobuf.

The defaults come from the recommended rules of each format. They are reasonable, but every team has its own contract with its consumers, so the rules are configurable in the commands.diff block of the configuration file. A property can be set to breaking, non_breaking, non_functional, none to ignore the change, or nested to take the highest severity of the changes inside. Rules can depend on conditions and can be overridden for specific locations. For example, this lets deprecated operations be removed without failing the build:

version: '1.0'
commands:
  diff:
    extends: [recommended]
    rules:
      pathItem:
        operations:
          remove_item:
            if: deprecated
            then: non_breaking
            else: breaking

The swagger, asyncapi, protobuf and graphql sections of the same block configure the other formats.

Reading the output

By default, apigenie diff prints the changelog report with the stylish formatter. Here is the beginning of what it prints for two versions of the Petstore example:

$ apigenie diff petstore.source.yaml petstore.dest.yaml
petstore.dest.yaml
   1  1      breaking  Metadata changed
  98  7  non-breaking  Added operation PUT /pets/{petId} (updatePet)
  87  7      breaking  Removed operation DELETE /pets/{petId} (deletePet)
  25  7      breaking  Modified operation GET /pets (listPets)
  25  7      breaking    Added operation.parameters['query:status']
  ...

Each line has a position in the destination file, a severity and a description. Changes inside a modified operation are indented under it. For Protobuf and GraphQL, what has been removed is reported at the closest enclosing declaration that is still there, as in this Protobuf example from the reference:

shop.proto
  19  1      breaking  Modified message 'Customer'
  19  1  non-breaking    Removed field 'fax'
  25  3      breaking    Modified field 'full_name'
  25  3      breaking      Modified field name

Two options control the output. --report chooses what to print: status, changelog (the default) or detailed. --formatter chooses how: html, json, markdown, stylish (the default), unix or visualstudio. A Markdown changelog is handy for release notes, JSON for further processing:

apigenie diff --report detailed --formatter markdown v1/openapi.yaml v2/openapi.yaml

Failing CI on breaking changes

The exit status is what makes the command useful in a pipeline. It exits with status 1 when there are breaking changes, and 0 otherwise. Any CI system can use that directly:

git show main:api/openapi.yaml > /tmp/openapi.base.yaml
apigenie diff --config apigenie.yaml /tmp/openapi.base.yaml api/openapi.yaml

The exit status considers all changes, whatever the report shows, so choosing a shorter report doesn’t hide a breaking change.

There is also --update-version. Instead of printing a report, it updates info.version of the destination file to the smallest version compatible with the changes: a new major version for breaking changes, minor for non-breaking and patch for non-functional ones. It keeps the rest of the file intact, prints nothing and exits with 0. Protobuf and GraphQL files have no version, so the option is rejected for them.

In the GitHub Action

If your API lives on GitHub, the API Genie GitHub Action runs the differ for you:

on: pull_request
jobs:
  api:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: mykulyak/api-genie-action@v0
        with:
          command: diff
          files: api/openapi.yaml

With command: diff, each file is compared with its version at the base commit of the pull request, or the commit before a push, and the step fails on breaking changes. Set base to compare with another commit, branch or tag, or base-file to compare two files of the repository.

The changes show up as annotations on the files of the pull request, so the reviewer sees “breaking” next to the line that caused it, and a report is added to the job summary. Annotations come from the default unix formatter, so leave formatter alone if you want them. The report input picks the report, as --report does on the command line. On pull requests, the action can also post the report as a comment and update it in place on later runs; that needs the pull-requests: write permission.

If you would rather decide yourself what a breaking change means for the build, set fail-on: never and read the status output, which is none, non-breaking or breaking.

The history view: apigenie changelog

apigenie diff compares two files you give it. Often the question is different: what happened to this API over the last releases? The changelog command answers it from git history. It compares the consecutive committed versions of an OpenAPI, Swagger, AsyncAPI, Protobuf or GraphQL file and rates each change with the same rules as diff, including your commands.diff configuration.

apigenie changelog api/openapi.yaml --revisions tags --tag-pattern 'v*' --unreleased --formatter markdown

This prints one entry per release tag, plus the changes made since the last release, in Markdown that can be published as is. With --by operation, the changes are grouped by operation instead, so you can see the history of a single endpoint. Unlike diff, changelog exits with 0 even when there are breaking changes: it describes history, it doesn’t gate it.

Where to start

Take the last released version of your API description and the current one, and run apigenie diff on them. If the output surprises you, that is exactly the change that would have surprised your consumers. Then put the same check in CI, so the next one is caught in review instead of in production.

Andriy Mykulyak builds API Genie.LinkedInXcontact@apigenie.pl
Was this helpful?

Try API Genie on your own schema

Run it with Docker or download a binary for your platform.

Get API Genie