Linter performance

This page gives how long apigenie lint, Spectral and vacuum take to check three OpenAPI documents with four rulesets, and their memory use. The measurements were made on 2026-11-08 with a Python script. A kit, a Node.js script with the same rulesets and command lines, now replaces it. The kit is in API Genie’s source repository, which is not public: Time the linters on your documents gives the command lines it runs.

What was measured

The three linters, at these versions:

Linter Version Runs as
apigenie lint the development version after 0.12.0 (it reports 0.12.0), built from source at commit f854324b9 with the macos-release preset a native arm64 binary
Spectral 6.17.1 spectral on Node.js 24.11.1
vacuum 0.39.3 the native arm64 binary of its release

vacuum’s npm package starts the binary from a Node.js script, which would add the start of Node.js to each run.

The machine: a MacBook Pro with an Apple M1 (8 cores) and 16 GiB of memory, on macOS 27.0.1, on AC power.

The documents:

Document Source Size
small petstore-with-fake-endpoints-models-for-testing.yaml, a test document of OpenAPI Generator 52 KiB
medium the Spotify Web API, from APIs.guru 282 KiB
large the GitHub REST API: api.github.com.yaml of 2023-12-22, from GitHub’s rest-api-description 8.2 MiB

The measured copy of the large document has six more lines: an x-apigenie extension that names two properties, +1 and -1, for generated code.

The rulesets:

  • default: each linter’s default rules: spectral:oas, vacuum’s recommended ruleset, and the recommended ruleset of apigenie lint. Spectral runs no rules without a ruleset file, so the kit passes it one that extends spectral:oas. They are different rules, so this row shows each linter with its own default, not the same work.
  • builtin: 26 rules of spectral:oas that the three linters implement natively. apigenie lint runs them as 24 rules under its own names, as Migrate from Spectral maps them.
  • declarative: 10 custom rules written with the core functions truthy, falsy, defined, casing, pattern, length and enumeration, including $.. paths that walk the whole document. The three linters read the same ruleset file.
  • javascript: the same 10 rules, with each check written as a custom JavaScript function. Spectral and apigenie lint load the same ES modules; vacuum loads the same functions written for its function API.

Each case ran 7 times after 1 untimed run; the large document ran 3 times, with no untimed run. The wall time table gives the median.

Wall time

Median wall time in seconds:

Document Ruleset Spectral vacuum apigenie lint
small default 0.688 0.053 0.020
small builtin 0.727 0.050 0.023
small declarative 0.635 0.051 0.021
small javascript 0.670 0.098 0.030
medium default 1.148 0.115 0.026
medium builtin 0.795 0.118 0.028
medium declarative 0.730 0.101 0.059
medium javascript 0.728 0.285 0.158
large default 23.951 3.562 0.244
large builtin 3.897 1.993 0.149
large declarative 3.479 4.457 1.123
large javascript 3.820 5.203 2.234

In the large builtin row, apigenie lint checks 243 operations that Spectral and vacuum skip: Issues reported says which.

Spectral’s median on the small document is 0.635 s or more with each ruleset. The start of Node.js and the loading of the ruleset were not timed apart.

With its default ruleset, Spectral takes 23.951 s on the large document, against 3.897 s with the builtin rules. Among the rules the default ruleset adds are oas3-valid-media-example and oas3-valid-schema-example, which validate the examples. That run was not profiled.

apigenie lint takes longer with custom rules than with its own. On the large document, it takes 1.123 s with the declarative rules and 2.234 s with the JavaScript functions, against 0.149 s with the builtin rules. Spectral takes about the same time with the declarative rules and with the JavaScript functions.

vacuum takes 2 to 3 times longer with the JavaScript functions than with the declarative rules on the small and medium documents. It runs a function’s script once for each node the rule selects (js_plugin.go). Its batch mode (batch: true in functionOptions) runs it once for all nodes; the kit doesn’t use it.

Memory and CPU

Peak memory on the large document, over the four rulesets:

Linter Peak memory
apigenie lint 87 to 115 MiB
Spectral 570 to 1,860 MiB
vacuum 1.3 to 2.2 GiB

Spectral and apigenie lint use about one core. vacuum runs on several. On the large document, its CPU time is 2 to 2.7 times its wall time, up to 13.3 s of CPU with the JavaScript functions.

Issues reported

The issue counts show whether the linters did comparable work. With the builtin rules, the totals of Spectral, vacuum and apigenie lint are close on the small (87, 81, 96) and medium (282, 304, 273) documents. On the large one, they are not:

Builtin rules, large document Spectral vacuum apigenie lint
Issues 100 556 2,064
Of them, in the 243 webhooks under x-webhooks 0 0 1,955
Of them, components only the webhooks use, reported as unused 0 261 0
Of them, request bodies without a description 0 241 0
Of them, enums with a value not of the schema’s type 0 0 4

The GitHub REST API describes its webhooks under x-webhooks, an extension of OpenAPI 3.0. apigenie lint checks the webhooks as operations. It reports 1,701 header parameters without a description, 241 webhooks without tags and 13 without a description. Spectral and vacuum don’t check the webhooks. vacuum reports the 249 schemas and 12 examples that only the webhooks use as unused. vacuum’s operation-description also reports 241 request bodies without a description. schema-enum-types reports 4 enums that typed-enum passes in Spectral and vacuum. They list strings in a schema without type. Before 0.13.0, it also reported 84 enums that list null in a schema with nullable: true, which OpenAPI 3.0 allows.

With the custom rules, the totals of Spectral, vacuum and apigenie lint on the small document are 63, 60 and 61 (declarative) and 63, 48 and 61 (javascript). Spectral’s 63 include one parser error, on a response code written as a number. vacuum’s casing accepts the property _class, which the other two report. With the JavaScript functions, vacuum reports the casing of property names once per properties object: 23 issues, against 37 in Spectral and 36 in apigenie lint. The kit’s function for vacuum reports at the properties object, with a message that does not name the property, and vacuum reports such results once.

Notes on the runs

  • Spectral 6.17.1 stops on the large document with the path $..[?(@.properties)], at a null in the GitHub document: Cannot read properties of null (reading 'properties'). The kit uses $..[?(@ && @.properties)], and the large declarative and javascript rows come from a second run with that path. In the first run, with the declarative rules, vacuum took 4.575 s and apigenie lint 1.150 s. With the JavaScript functions, they took 4.988 s and 2.263 s.
  • In the second run, one of vacuum’s 3 runs of the large document with the JavaScript functions exited with status 2. Its cause wasn’t recorded. Its time is one of the 3 samples of the median.
  • VS Code and a Safari tab each used about one core during the runs. The 1-minute load average was about 6 at the start. The runs of a case are close together, but the shortest times can carry a few milliseconds of noise.
  • The counts per rule under Issues reported, and the custom-rule totals on the small document, come from a rerun with the kit on 2026-11-09, except the apigenie lint counts of the large document with the builtin rules (next note). It ran in a Linux cloud container with the npm packages of Spectral, vacuum and API Genie 0.12.0. It ran each small and medium case 2 times, and the large document with the builtin rules once. Its builtin totals match those of the timed runs on the Mac on the three documents.
  • The apigenie lint counts of the large document with the builtin rules come from a run on the Mac on 2026-11-09. It used a build at commit 063c2ff96, with the schema-enum-types fix that 0.13.0 brings. Its counts are 84 lower than the 2,148 of the timed runs. The small and medium counts didn’t change. The times weren’t measured again.
  • A run on Linux in a cloud container on 2026-11-08 agreed on the small and medium documents. Its large runs of Spectral and vacuum failed, and it is not used here.

Time the linters on your documents

These are the command lines the kit runs. With the three documents above, the first group runs the default rows again. The other rows need the kit’s builtin, declarative and javascript rulesets, which this page doesn’t give. Use the same command lines to time the three linters on a document of yours.

The first group runs each linter’s default rules. Spectral runs no rules without a ruleset file, so default.yaml holds the one line extends: spectral:oas. vacuum runs its recommended ruleset, and apigenie lint its recommended ruleset:

spectral lint -q -f json -o spectral.json -r default.yaml openapi.yaml
vacuum lint -b -q --no-update-check --format json -o vacuum.json openapi.yaml
apigenie lint --formatter json --no-config openapi.yaml > apigenie.json

The second group runs a Spectral ruleset of yours, ruleset.yaml:

spectral lint -q -f json -o spectral.json -r ruleset.yaml openapi.yaml
vacuum lint -b -q --no-update-check --format json -o vacuum.json -r ruleset.yaml openapi.yaml
apigenie lint --formatter json --no-config --ruleset ruleset.yaml openapi.yaml > apigenie.json

If the ruleset calls JavaScript functions, Spectral and apigenie lint load them from its functionsDir. vacuum doesn’t load Spectral’s functions: the kit gives it the same checks written for vacuum’s function API, with -f and their directory. For the builtin rows, the kit passes apigenie lint a configuration file with --config. The file turns the same rules on under their API Genie names.

The kit takes the wall time of each run itself. On macOS, and on Linux with GNU time installed, it starts the linter with /usr/bin/time for the CPU time and the peak memory. The option is -l on macOS and -v on Linux. Put the same prefix in front of a command line to read them for a run of yours:

/usr/bin/time -v apigenie lint --formatter json --no-config openapi.yaml > apigenie.json

On the large document, in a 4-core Linux cloud container with API Genie 0.12.0, it prints the report below. The tables come from the Mac, not from this run:

Command exited with non-zero status 1
	Command being timed: "apigenie lint --formatter json --no-config openapi.yaml"
	User time (seconds): 0.70
	System time (seconds): 0.13
	Percent of CPU this job got: 99%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.84
	...
	Maximum resident set size (kbytes): 99792
	...
	Exit status: 1

GNU time prints its report to standard error. User time and System time add up to the CPU time. Maximum resident set size is the peak memory, in kbytes; macOS time -l gives it in bytes. Each linter exits with 1 when it reports an error. GNU time then starts its report with Command exited with non-zero status 1.

apigenie from Homebrew or a release archive is the native executable. The npm package installs a Node.js launcher instead. Time the executable of its platform package: @mykulyak/api-genie-<platform>/bin/apigenie (apigenie.exe on Windows) in the node_modules it was installed to, where <platform> is darwin-arm64, linux-x64 or win32-x64. Time vacuum’s release binary too, not its npm package, which starts the binary from a Node.js script.

Last updated on View as Markdown
Was this helpful?