apigenie test

Runs the requests of .http files and requests generated from an OpenAPI or AsyncAPI document, a GraphQL schema or a .proto file against a live server, checks their assertions and scripts, and validates every request, response and message against the document or the schema with the checks of the validating proxy. HTTP requests are checked against an OpenAPI document, GraphQL requests against a GraphQL schema, GRPC requests against a .proto file, and WebSocket and MQTT, Kafka and AMQP requests and the requests of HTTP channels against an AsyncAPI document.

Topics

Usage

apigenie test [OPTIONS] FILES...
apigenie test --spec SPEC --generate [--base-url URL] [OPTIONS] [FILES...]
apigenie test --spec schema.graphql [--generate --base-url URL] [OPTIONS] [FILES...]
apigenie test --spec service.proto [--proto-path DIR ...] FILES...
apigenie test --spec ASYNCAPI --generate [--base-url ws://HOST:PORT] [--mqtt-url mqtt://HOST:PORT] [OPTIONS]
apigenie test import COLLECTION -o DIR [--environment FILE ...] [--spec SPEC] [--force]

The request files are written in the httpYac flavour of the .http format (see Request files). Three requests, the second reading the response of the first, and the run that checks them against an OpenAPI document:

@host = http://localhost:4010/v1

### List pets
# @name list
GET {{host}}/pets?limit=2
Accept: application/json

?? status == 200
?? body $ isArray

### Read the first one
GET {{host}}/pets/{{list.response.body.$[0].id}}

{{
  exports.petName = response.parsedBody.name;
  test("has a name", () => assert.ok(response.parsedBody.name));
}}

### Create a pet
POST {{host}}/pets
Content-Type: application/json

{"id": 7, "name": "{{petName}}"}

?? status == 201
apigenie test --spec openapi.yaml pets.http --report report.json --report-junit report.xml
PASS GET http://localhost:4010/v1/pets?limit=2 -> 200 (9 ms) listPets
PASS GET http://localhost:4010/v1/pets/746 -> 200 (4 ms) showPetById
PASS POST http://localhost:4010/v1/pets -> 201 (2 ms) createPets
Requests: 3 passed, 0 failed, 0 skipped
Contract: 3 requests (0 undocumented), 0 errors, 0 warnings
Coverage: 70.0% of 10 branches, 3 of 3 operations

Options

Option Description
FILES .http and .rest files, or directories to search for them
--spec SPEC The OpenAPI 3.0, 3.1 or 3.2 (or Swagger 2.0) document HTTP requests are validated against, the GraphQL schema (.graphql, .graphqls, .gql) of GraphQL requests, the .proto file of GRPC requests, or the AsyncAPI 3.x document WS, MQTT, KAFKA and AMQP requests and the requests of its HTTP channels are validated against
--proto-path DIR A directory in which the imports of a .proto --spec are looked up; can be repeated
--generate Also send one request per operation (OpenAPI), root field (GraphQL), method (.proto) or channel (AsyncAPI; one per operation of an HTTP, Kafka or AMQP channel) of --spec
--base-url URL The server’s URL for --generate (the endpoint, required, for a GraphQL schema; HOST:PORT for a .proto file; ws:// or http:// for an AsyncAPI document’s WebSocket and HTTP channels), also put before request URLs that start with /
--mqtt-url URL The MQTT broker (mqtt://host:port) of an AsyncAPI document’s channels for --generate
--amqp-url URL The AMQP broker (amqp://host:port/vhost, amqps:// over TLS) of an AsyncAPI document’s AMQP channels for --generate
--kafka-url URL A broker of the Kafka cluster (kafka://host:port, kafkas:// over TLS) of an AsyncAPI document’s Kafka channels for --generate
--env NAME The environment of the environment files
--env-file FILE An environment file to read after the others; can be repeated
--var NAME=VALUE A variable above every other one; can be repeated
--header NAME:VALUE A header for every request that doesn’t have it; can be repeated
--timeout MS How long to wait for a connection and for a response (30000)
--message-timeout MS How long WS, MQTT, KAFKA and AMQP requests, GraphQL subscriptions and gRPC client and bidi streams wait for messages (5000); # @timeout overrides it
--bail Stop at the first failed request
--ca-cert FILE A PEM file of certificate authorities trusted for https://, grpcs://, wss://, mqtts://, kafkas:// and amqps:// URLs, besides the system’s; can be repeated
--insecure Don’t verify the certificates of https://, grpcs://, wss://, mqtts://, kafkas:// and amqps:// servers; the traffic stays encrypted
--client-cert FILE A PEM client certificate (chain) presented to https://, grpcs://, wss://, mqtts://, kafkas:// and amqps:// servers that ask for one (mTLS); needs --client-key
--client-key FILE The unencrypted PEM private key of --client-cert
--report FILE Write the JSON report
--report-junit FILE Write the JUnit XML report
--fail-on none|errors|warnings The contract violations that fail the run (errors)
--min-coverage PERCENT Fail the run below this coverage of --spec (a GraphQL schema: the share of its fields; a .proto file: the share of its methods; an AsyncAPI document: the share of its operations’ messages)
--severity CHECK=LEVEL Change a contract check’s severity: error, warn, info or off (ignore); can be repeated. The checks are those of the OpenAPI, gRPC and AsyncAPI proxies or, with a GraphQL schema, of the GraphQL proxy; an unknown check is a bad option
--seed N Seed of the values --generate makes up
--formatter stylish|json A line per request and a summary (stylish), or a JSON object per line
--quiet Show only failed requests and the summary
--verbose Also show the warnings and notes of the contract checks, and the scripts’ log

–fail-on errors|warnings|none

A failed request (an assertion, a script’s test or error, no response, an unknown @ref) always fails the run. --fail-on says which contract violations do too: errors (the default), warnings (errors and warnings) or none. A request with such violations is shown as FAIL.

–formatter json

Each request is a line {"event":"request",...} with the members of the JSON report’s requests; warnings are {"event":"warning","file":...,"line":...,"message":...}, notes about a Swagger 2.0 specification {"event":"info",...} with the same members; the last line is {"event":"summary","formatVersion":1,"total":...,"passed":...,"failed":...,"skipped":...,"gate":{...}} with the contract’s errors, warnings and coverage when there is a --spec. Its formatVersion versions the whole stream, as formatVersion versions the report.

Reports

--report writes the contract report of apigenie-server with "report": "apigenie-test", plus tests (the counts and the run’s gate) and requests (each request’s file, line, URL, status, duration, operation, outcome, failures, assertions, violations and log; grpc and grpcStatus for GRPC requests). With a .proto file the contract members are those of the gRPC report ("specification": "grpc", methods, services), with a GraphQL schema those of the GraphQL report ("specification": "graphql", rootFields, fields, uncoveredFields, operations). Without --spec, only report, formatVersion, version, startedAt, endedAt, tests and requests are written. formatVersion is the version of the format, which changes only when a member is removed or changes meaning (see Compatibility); version is its former name, kept for one release. The JSON Schema of the report is apigenie-test-report.json. --report-junit writes a requests suite (a test case per request: failed, or skipped for disabled requests) and, with --spec, the contract suite of apigenie-server. With an AsyncAPI document, the report is the AsyncAPI proxy’s ("specification": "asyncapi", channels, operations with their messages, violations); WS and MQTT requests also have channel, address, messagesSent, messagesReceived and notes (what wasn’t checked, and why), and the coverage --min-coverage checks is the share of the operations’ messages seen.

Exit status

Code Meaning
0 Every request passed and the contract gate held
2 Bad options: no request files without --generate, --generate without --spec, or with a GraphQL schema or a .proto file but no --base-url, --min-coverage without --spec, an unknown --severity check; a request file, an environment file or the specification (an OpenAPI or AsyncAPI document, a GraphQL schema, or a .proto file with its imports) that can’t be read or parsed; --generate without a URL it can use (no --base-url and no absolute server URL in the document, or a URL of the wrong scheme); a report that can’t be written, or an unexpected error
5 A request failed, or contract violations (--fail-on) or the coverage (--min-coverage) failed the run

See also

  • apigenie-server, whose validating proxy checks traffic something else drives, with the same checks and reports
  • lint, which checks the document itself
Was this helpful?