apigenie test
Runs the requests of .http files and requests generated from an OpenAPI or AsyncAPI document, a
GraphQL schema or a .proto file against a live server, checks their assertions and scripts, and
validates every request, response and message against the document or the schema with the checks
of the validating proxy. HTTP requests are checked against an OpenAPI
document, GraphQL requests against a GraphQL schema,
GRPC requests against a .proto file, and
WebSocket and MQTT,
Kafka and AMQP requests and the
requests of HTTP channels against
an AsyncAPI document.
Topics
- Request files: Regions, URLs and TLS, and the directives a request takes.
- Variables and OAuth 2: Environment files,
{{name}}, the responses of earlier requests, and tokens from a token endpoint. - Assertions and scripts:
??lines, and the JavaScript that runs before and after a request. - Validation: What
--specchecks in requests, responses and messages, for each kind of document. - Generated requests:
--generate: one request per operation, root field, method or channel of--spec. - GraphQL requests: Queries, mutations and subscriptions over graphql-ws.
- WebSocket and MQTT requests:
WSandMQTTregions, the broker settings and MQTT 5 properties. - Kafka requests:
KAFKAregions: the records produced and read. - AMQP requests:
AMQPregions: the messages published and consumed. - gRPC requests:
GRPCrequests to the methods of a.protofile, streams included. - Importing collections:
apigenie test import: a Postman collection as request files. - httpYac compatibility: What of the httpYac format is read, and what is not yet.
Usage
apigenie test [OPTIONS] FILES...
apigenie test --spec SPEC --generate [--base-url URL] [OPTIONS] [FILES...]
apigenie test --spec schema.graphql [--generate --base-url URL] [OPTIONS] [FILES...]
apigenie test --spec service.proto [--proto-path DIR ...] FILES...
apigenie test --spec ASYNCAPI --generate [--base-url ws://HOST:PORT] [--mqtt-url mqtt://HOST:PORT] [OPTIONS]
apigenie test import COLLECTION -o DIR [--environment FILE ...] [--spec SPEC] [--force]
The request files are written in the httpYac flavour of the .http format (see
Request files). Three requests, the second reading the
response of the first, and the run that checks them against an OpenAPI document:
@host = http://localhost:4010/v1
### List pets
# @name list
GET {{host}}/pets?limit=2
Accept: application/json
?? status == 200
?? body $ isArray
### Read the first one
GET {{host}}/pets/{{list.response.body.$[0].id}}
{{
exports.petName = response.parsedBody.name;
test("has a name", () => assert.ok(response.parsedBody.name));
}}
### Create a pet
POST {{host}}/pets
Content-Type: application/json
{"id": 7, "name": "{{petName}}"}
?? status == 201
apigenie test --spec openapi.yaml pets.http --report report.json --report-junit report.xml
PASS GET http://localhost:4010/v1/pets?limit=2 -> 200 (9 ms) listPets
PASS GET http://localhost:4010/v1/pets/746 -> 200 (4 ms) showPetById
PASS POST http://localhost:4010/v1/pets -> 201 (2 ms) createPets
Requests: 3 passed, 0 failed, 0 skipped
Contract: 3 requests (0 undocumented), 0 errors, 0 warnings
Coverage: 70.0% of 10 branches, 3 of 3 operations
Options
| Option | Description |
|---|---|
FILES |
.http and .rest files, or directories to search for them |
--spec SPEC |
The OpenAPI 3.0, 3.1 or 3.2 (or Swagger 2.0) document HTTP requests are validated against, the GraphQL schema (.graphql, .graphqls, .gql) of GraphQL requests, the .proto file of GRPC requests, or the AsyncAPI 3.x document WS, MQTT, KAFKA and AMQP requests and the requests of its HTTP channels are validated against |
--proto-path DIR |
A directory in which the imports of a .proto --spec are looked up; can be repeated |
--generate |
Also send one request per operation (OpenAPI), root field (GraphQL), method (.proto) or channel (AsyncAPI; one per operation of an HTTP, Kafka or AMQP channel) of --spec |
--base-url URL |
The server’s URL for --generate (the endpoint, required, for a GraphQL schema; HOST:PORT for a .proto file; ws:// or http:// for an AsyncAPI document’s WebSocket and HTTP channels), also put before request URLs that start with / |
--mqtt-url URL |
The MQTT broker (mqtt://host:port) of an AsyncAPI document’s channels for --generate |
--amqp-url URL |
The AMQP broker (amqp://host:port/vhost, amqps:// over TLS) of an AsyncAPI document’s AMQP channels for --generate |
--kafka-url URL |
A broker of the Kafka cluster (kafka://host:port, kafkas:// over TLS) of an AsyncAPI document’s Kafka channels for --generate |
--env NAME |
The environment of the environment files |
--env-file FILE |
An environment file to read after the others; can be repeated |
--var NAME=VALUE |
A variable above every other one; can be repeated |
--header NAME:VALUE |
A header for every request that doesn’t have it; can be repeated |
--timeout MS |
How long to wait for a connection and for a response (30000) |
--message-timeout MS |
How long WS, MQTT, KAFKA and AMQP requests, GraphQL subscriptions and gRPC client and bidi streams wait for messages (5000); # @timeout overrides it |
--bail |
Stop at the first failed request |
--ca-cert FILE |
A PEM file of certificate authorities trusted for https://, grpcs://, wss://, mqtts://, kafkas:// and amqps:// URLs, besides the system’s; can be repeated |
--insecure |
Don’t verify the certificates of https://, grpcs://, wss://, mqtts://, kafkas:// and amqps:// servers; the traffic stays encrypted |
--client-cert FILE |
A PEM client certificate (chain) presented to https://, grpcs://, wss://, mqtts://, kafkas:// and amqps:// servers that ask for one (mTLS); needs --client-key |
--client-key FILE |
The unencrypted PEM private key of --client-cert |
--report FILE |
Write the JSON report |
--report-junit FILE |
Write the JUnit XML report |
--fail-on none|errors|warnings |
The contract violations that fail the run (errors) |
--min-coverage PERCENT |
Fail the run below this coverage of --spec (a GraphQL schema: the share of its fields; a .proto file: the share of its methods; an AsyncAPI document: the share of its operations’ messages) |
--severity CHECK=LEVEL |
Change a contract check’s severity: error, warn, info or off (ignore); can be repeated. The checks are those of the OpenAPI, gRPC and AsyncAPI proxies or, with a GraphQL schema, of the GraphQL proxy; an unknown check is a bad option |
--seed N |
Seed of the values --generate makes up |
--formatter stylish|json |
A line per request and a summary (stylish), or a JSON object per line |
--quiet |
Show only failed requests and the summary |
--verbose |
Also show the warnings and notes of the contract checks, and the scripts’ log |
–fail-on errors|warnings|none
A failed request (an assertion, a script’s test or error, no response, an unknown @ref) always
fails the run. --fail-on says which contract violations do too: errors (the default),
warnings (errors and warnings) or none. A request with such violations is shown as FAIL.
–formatter json
Each request is a line {"event":"request",...} with the members of the JSON report’s
requests; warnings are {"event":"warning","file":...,"line":...,"message":...}, notes about a
Swagger 2.0 specification {"event":"info",...} with the same members; the last line is
{"event":"summary","formatVersion":1,"total":...,"passed":...,"failed":...,"skipped":...,"gate":{...}}
with the contract’s errors, warnings and coverage when there is a --spec. Its
formatVersion versions the whole stream, as formatVersion versions the report.
Reports
--report writes the contract report
of apigenie-server with "report": "apigenie-test", plus tests (the counts and the run’s gate)
and requests (each request’s file, line, URL, status, duration, operation, outcome, failures,
assertions, violations and log; grpc and grpcStatus for GRPC requests). With a .proto file
the contract members are those of the gRPC report ("specification": "grpc", methods,
services), with a GraphQL schema those of the GraphQL report ("specification": "graphql",
rootFields, fields, uncoveredFields, operations). Without --spec, only report, formatVersion, version, startedAt, endedAt, tests and requests are written.
formatVersion is the version of the format, which changes only when a member is removed or
changes meaning (see Compatibility); version is its
former name, kept for one release. The JSON Schema of the report is
apigenie-test-report.json. --report-junit writes a requests suite (a test case per request: failed, or
skipped for disabled requests) and, with --spec, the contract suite of apigenie-server.
With an AsyncAPI document, the report is the AsyncAPI proxy’s ("specification": "asyncapi",
channels, operations with their messages, violations); WS and MQTT requests also have channel,
address, messagesSent, messagesReceived and notes (what wasn’t checked, and why), and the
coverage --min-coverage checks is the share of the operations’ messages seen.
Exit status
| Code | Meaning |
|---|---|
| 0 | Every request passed and the contract gate held |
| 2 | Bad options: no request files without --generate, --generate without --spec, or with a GraphQL schema or a .proto file but no --base-url, --min-coverage without --spec, an unknown --severity check; a request file, an environment file or the specification (an OpenAPI or AsyncAPI document, a GraphQL schema, or a .proto file with its imports) that can’t be read or parsed; --generate without a URL it can use (no --base-url and no absolute server URL in the document, or a URL of the wrong scheme); a report that can’t be written, or an unexpected error |
| 5 | A request failed, or contract violations (--fail-on) or the coverage (--min-coverage) failed the run |
See also
apigenie-server, whose validating proxy checks traffic something else drives, with the same checks and reports- lint, which checks the document itself