Linter performance
This page gives how long apigenie lint, Spectral and vacuum take to check three OpenAPI documents
with four rulesets, and their memory use. The measurements were made on 2026-11-08 with a Python
script. A kit, a Node.js script with the same rulesets and command lines, now replaces it. The kit
is in API Genie’s source repository, which is not public:
Time the linters on your documents gives the command lines
it runs.
What was measured
The three linters, at these versions:
| Linter | Version | Runs as |
|---|---|---|
apigenie lint |
the development version after 0.12.0 (it reports 0.12.0), built from source at commit f854324b9 with the macos-release preset |
a native arm64 binary |
| Spectral | 6.17.1 | spectral on Node.js 24.11.1 |
| vacuum | 0.39.3 | the native arm64 binary of its release |
vacuum’s npm package starts the binary from a Node.js script, which would add the start of Node.js to each run.
The machine: a MacBook Pro with an Apple M1 (8 cores) and 16 GiB of memory, on macOS 27.0.1, on AC power.
The documents:
| Document | Source | Size |
|---|---|---|
| small | petstore-with-fake-endpoints-models-for-testing.yaml, a test document of OpenAPI Generator |
52 KiB |
| medium | the Spotify Web API, from APIs.guru | 282 KiB |
| large | the GitHub REST API: api.github.com.yaml of 2023-12-22, from GitHub’s rest-api-description |
8.2 MiB |
The measured copy of the large document has six more lines: an x-apigenie extension that names
two properties, +1 and -1, for generated code.
The rulesets:
- default: each linter’s default rules:
spectral:oas, vacuum’s recommended ruleset, and therecommendedruleset ofapigenie lint. Spectral runs no rules without a ruleset file, so the kit passes it one that extendsspectral:oas. They are different rules, so this row shows each linter with its own default, not the same work. - builtin: 26 rules of
spectral:oasthat the three linters implement natively.apigenie lintruns them as 24 rules under its own names, as Migrate from Spectral maps them. - declarative: 10 custom rules written with the core functions
truthy,falsy,defined,casing,pattern,lengthandenumeration, including$..paths that walk the whole document. The three linters read the same ruleset file. - javascript: the same 10 rules, with each check written as a custom JavaScript function.
Spectral and
apigenie lintload the same ES modules; vacuum loads the same functions written for its function API.
Each case ran 7 times after 1 untimed run; the large document ran 3 times, with no untimed run. The wall time table gives the median.
Wall time
Median wall time in seconds:
| Document | Ruleset | Spectral | vacuum | apigenie lint |
|---|---|---|---|---|
| small | default | 0.688 | 0.053 | 0.020 |
| small | builtin | 0.727 | 0.050 | 0.023 |
| small | declarative | 0.635 | 0.051 | 0.021 |
| small | javascript | 0.670 | 0.098 | 0.030 |
| medium | default | 1.148 | 0.115 | 0.026 |
| medium | builtin | 0.795 | 0.118 | 0.028 |
| medium | declarative | 0.730 | 0.101 | 0.059 |
| medium | javascript | 0.728 | 0.285 | 0.158 |
| large | default | 23.951 | 3.562 | 0.244 |
| large | builtin | 3.897 | 1.993 | 0.149 |
| large | declarative | 3.479 | 4.457 | 1.123 |
| large | javascript | 3.820 | 5.203 | 2.234 |
In the large builtin row, apigenie lint checks 243 operations that Spectral and vacuum skip:
Issues reported says which.
Spectral’s median on the small document is 0.635 s or more with each ruleset. The start of Node.js and the loading of the ruleset were not timed apart.
With its default ruleset, Spectral takes 23.951 s on the large document, against 3.897 s with the
builtin rules. Among the rules the default ruleset adds are oas3-valid-media-example and
oas3-valid-schema-example, which validate the examples. That run was not profiled.
apigenie lint takes longer with custom rules than with its own. On the large document, it takes
1.123 s with the declarative rules and 2.234 s with the JavaScript functions, against 0.149 s with
the builtin rules. Spectral takes about the same time with the declarative rules and with the
JavaScript functions.
vacuum takes 2 to 3 times longer with the JavaScript functions than with the declarative rules on
the small and medium documents.
It runs a function’s script once for each node the rule selects
(js_plugin.go).
Its batch mode (batch: true in functionOptions) runs it once for all nodes; the kit doesn’t use
it.
Memory and CPU
Peak memory on the large document, over the four rulesets:
| Linter | Peak memory |
|---|---|
apigenie lint |
87 to 115 MiB |
| Spectral | 570 to 1,860 MiB |
| vacuum | 1.3 to 2.2 GiB |
Spectral and apigenie lint use about one core. vacuum runs on several. On the large document,
its CPU time is 2 to 2.7 times its wall time, up to 13.3 s of CPU with the JavaScript functions.
Issues reported
The issue counts show whether the linters did comparable work. With the builtin rules, the totals
of Spectral, vacuum and apigenie lint are close on the small (87, 81, 96) and medium (282, 304,
273) documents. On the large one, they are not:
| Builtin rules, large document | Spectral | vacuum | apigenie lint |
|---|---|---|---|
| Issues | 100 | 556 | 2,064 |
Of them, in the 243 webhooks under x-webhooks |
0 | 0 | 1,955 |
| Of them, components only the webhooks use, reported as unused | 0 | 261 | 0 |
| Of them, request bodies without a description | 0 | 241 | 0 |
| Of them, enums with a value not of the schema’s type | 0 | 0 | 4 |
The GitHub REST API describes its webhooks under x-webhooks, an extension of OpenAPI 3.0.
apigenie lint checks the webhooks as operations. It reports 1,701 header parameters without a
description, 241 webhooks without tags and 13 without a description. Spectral and vacuum don’t
check the webhooks. vacuum reports the 249 schemas and 12 examples that only the webhooks use as
unused. vacuum’s operation-description also reports 241 request bodies without a description.
schema-enum-types reports 4 enums that typed-enum passes in Spectral and vacuum. They list
strings in a schema without type. Before 0.13.0, it also reported 84 enums that list null in a
schema with nullable: true, which OpenAPI 3.0 allows.
With the custom rules, the totals of Spectral, vacuum and apigenie lint on the small document
are 63, 60 and 61 (declarative) and 63, 48 and 61 (javascript). Spectral’s 63 include one parser
error, on a response code written as a number. vacuum’s casing accepts the property _class,
which the other two report. With the JavaScript functions, vacuum reports the
casing of property names once per properties object: 23 issues, against 37 in Spectral and 36 in apigenie lint. The kit’s
function for vacuum reports at the properties object, with a message that does not name the
property, and vacuum reports such results once.
Notes on the runs
- Spectral 6.17.1 stops on the large document with the path
$..[?(@.properties)], at anullin the GitHub document:Cannot read properties of null (reading 'properties'). The kit uses$..[?(@ && @.properties)], and the large declarative and javascript rows come from a second run with that path. In the first run, with the declarative rules, vacuum took 4.575 s andapigenie lint1.150 s. With the JavaScript functions, they took 4.988 s and 2.263 s. - In the second run, one of vacuum’s 3 runs of the large document with the JavaScript functions exited with status 2. Its cause wasn’t recorded. Its time is one of the 3 samples of the median.
- VS Code and a Safari tab each used about one core during the runs. The 1-minute load average was about 6 at the start. The runs of a case are close together, but the shortest times can carry a few milliseconds of noise.
- The counts per rule under Issues reported, and the custom-rule totals on the
small document, come from a rerun with the kit on 2026-11-09, except the
apigenie lintcounts of the large document with the builtin rules (next note). It ran in a Linux cloud container with the npm packages of Spectral, vacuum and API Genie 0.12.0. It ran each small and medium case 2 times, and the large document with the builtin rules once. Its builtin totals match those of the timed runs on the Mac on the three documents. - The
apigenie lintcounts of the large document with the builtin rules come from a run on the Mac on 2026-11-09. It used a build at commit 063c2ff96, with theschema-enum-typesfix that 0.13.0 brings. Its counts are 84 lower than the 2,148 of the timed runs. The small and medium counts didn’t change. The times weren’t measured again. - A run on Linux in a cloud container on 2026-11-08 agreed on the small and medium documents. Its large runs of Spectral and vacuum failed, and it is not used here.
Time the linters on your documents
These are the command lines the kit runs. With the three documents above, the first group runs the default rows again. The other rows need the kit’s builtin, declarative and javascript rulesets, which this page doesn’t give. Use the same command lines to time the three linters on a document of yours.
The first group runs each linter’s default rules. Spectral runs no rules without a ruleset file,
so default.yaml holds the one line extends: spectral:oas. vacuum runs its recommended ruleset,
and apigenie lint its recommended ruleset:
spectral lint -q -f json -o spectral.json -r default.yaml openapi.yaml
vacuum lint -b -q --no-update-check --format json -o vacuum.json openapi.yaml
apigenie lint --formatter json --no-config openapi.yaml > apigenie.json
The second group runs a Spectral ruleset of yours, ruleset.yaml:
spectral lint -q -f json -o spectral.json -r ruleset.yaml openapi.yaml
vacuum lint -b -q --no-update-check --format json -o vacuum.json -r ruleset.yaml openapi.yaml
apigenie lint --formatter json --no-config --ruleset ruleset.yaml openapi.yaml > apigenie.json
If the ruleset calls JavaScript functions, Spectral and apigenie lint load them from its
functionsDir. vacuum doesn’t load Spectral’s functions: the kit gives it the same checks written
for vacuum’s function API, with -f and their directory. For the builtin rows, the kit passes
apigenie lint a configuration file with --config. The file turns the same rules on under their
API Genie names.
The kit takes the wall time of each run itself. On macOS, and on Linux with GNU time installed,
it starts the linter with /usr/bin/time for the CPU time and the peak memory. The option is -l
on macOS and -v on Linux. Put the same prefix in front of a command line to read them for a run
of yours:
/usr/bin/time -v apigenie lint --formatter json --no-config openapi.yaml > apigenie.json
On the large document, in a 4-core Linux cloud container with API Genie 0.12.0, it prints the report below. The tables come from the Mac, not from this run:
Command exited with non-zero status 1
Command being timed: "apigenie lint --formatter json --no-config openapi.yaml"
User time (seconds): 0.70
System time (seconds): 0.13
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.84
...
Maximum resident set size (kbytes): 99792
...
Exit status: 1
GNU time prints its report to standard error. User time and System time add up to the CPU
time. Maximum resident set size is the peak memory, in kbytes; macOS time -l gives it in bytes.
Each linter exits with 1 when it reports an error. GNU time then starts its report with
Command exited with non-zero status 1.
apigenie from Homebrew or a release archive is the native executable. The npm package installs
a Node.js launcher instead. Time the executable of its platform package:
@mykulyak/api-genie-<platform>/bin/apigenie (apigenie.exe on Windows) in the node_modules it
was installed to, where <platform> is darwin-arm64, linux-x64 or win32-x64. Time vacuum’s
release binary too, not its npm package, which starts the binary from a Node.js script.