# apigenie test

Source: https://apigenie.pl/docs/reference/cli/test
Part of the reference of [apigenie](https://apigenie.pl/docs/reference/cli.md).
Its other pages: [bundle](https://apigenie.pl/docs/reference/cli/bundle.md), [changelog](https://apigenie.pl/docs/reference/cli/changelog.md), [config](https://apigenie.pl/docs/reference/cli/config.md), [convert](https://apigenie.pl/docs/reference/cli/convert.md), [credits](https://apigenie.pl/docs/reference/cli/credits.md), [diff](https://apigenie.pl/docs/reference/cli/diff.md), [evolve](https://apigenie.pl/docs/reference/cli/evolve.md), [format](https://apigenie.pl/docs/reference/cli/format.md), [i18n](https://apigenie.pl/docs/reference/cli/i18n.md), [init](https://apigenie.pl/docs/reference/cli/init.md), [license](https://apigenie.pl/docs/reference/cli/license.md), [lint](https://apigenie.pl/docs/reference/cli/lint.md), [overlay](https://apigenie.pl/docs/reference/cli/overlay.md).

Runs the requests of `.http` files and requests generated from an OpenAPI or AsyncAPI document, a
GraphQL schema or a `.proto` file against a live server, checks their assertions and scripts, and
validates every request, response and message against the document or the schema with the checks
of the [validating proxy](/docs/reference/server). HTTP requests are checked against an OpenAPI
document, [GraphQL requests](/docs/reference/cli/test/graphql) against a GraphQL schema,
[`GRPC` requests](/docs/reference/cli/test/grpc) against a `.proto` file, and
[WebSocket and MQTT](/docs/reference/cli/test/websocket-mqtt),
[Kafka](/docs/reference/cli/test/kafka) and [AMQP](/docs/reference/cli/test/amqp) requests and the
requests of [HTTP channels](/docs/reference/cli/test/validation#requests-of-http-channels) against
an AsyncAPI document.

## Topics

- [Request files](/docs/reference/cli/test/requests): Regions, URLs and TLS, and the directives a request takes.
- [Variables and OAuth 2](/docs/reference/cli/test/variables): Environment files, `{{name}}`, the responses of earlier requests, and tokens from a token endpoint.
- [Assertions and scripts](/docs/reference/cli/test/assertions): `??` lines, and the JavaScript that runs before and after a request.
- [Validation](/docs/reference/cli/test/validation): What `--spec` checks in requests, responses and messages, for each kind of document.
- [Generated requests](/docs/reference/cli/test/generate): `--generate`: one request per operation, root field, method or channel of `--spec`.
- [GraphQL requests](/docs/reference/cli/test/graphql): Queries, mutations and subscriptions over graphql-ws.
- [WebSocket and MQTT requests](/docs/reference/cli/test/websocket-mqtt): `WS` and `MQTT` regions, the broker settings and MQTT 5 properties.
- [Kafka requests](/docs/reference/cli/test/kafka): `KAFKA` regions: the records produced and read.
- [AMQP requests](/docs/reference/cli/test/amqp): `AMQP` regions: the messages published and consumed.
- [gRPC requests](/docs/reference/cli/test/grpc): `GRPC` requests to the methods of a `.proto` file, streams included.
- [Importing collections](/docs/reference/cli/test/import): `apigenie test import`: a Postman collection as request files.
- [httpYac compatibility](/docs/reference/cli/test/httpyac): What of the httpYac format is read, and what is not yet.

## Usage

```sh
apigenie test [OPTIONS] FILES...
apigenie test --spec SPEC --generate [--base-url URL] [OPTIONS] [FILES...]
apigenie test --spec schema.graphql [--generate --base-url URL] [OPTIONS] [FILES...]
apigenie test --spec service.proto [--proto-path DIR ...] FILES...
apigenie test --spec ASYNCAPI --generate [--base-url ws://HOST:PORT] [--mqtt-url mqtt://HOST:PORT] [OPTIONS]
apigenie test import COLLECTION -o DIR [--environment FILE ...] [--spec SPEC] [--force]
```

The request files are written in the httpYac flavour of the `.http` format (see
[Request files](/docs/reference/cli/test/requests)). Three requests, the second reading the
response of the first, and the run that checks them against an OpenAPI document:

```http
@host = http://localhost:4010/v1

### List pets
# @name list
GET {{host}}/pets?limit=2
Accept: application/json

?? status == 200
?? body $ isArray

### Read the first one
GET {{host}}/pets/{{list.response.body.$[0].id}}

{{
  exports.petName = response.parsedBody.name;
  test("has a name", () => assert.ok(response.parsedBody.name));
}}

### Create a pet
POST {{host}}/pets
Content-Type: application/json

{"id": 7, "name": "{{petName}}"}

?? status == 201
```

```sh
apigenie test --spec openapi.yaml pets.http --report report.json --report-junit report.xml
```

```text
PASS GET http://localhost:4010/v1/pets?limit=2 -> 200 (9 ms) listPets
PASS GET http://localhost:4010/v1/pets/746 -> 200 (4 ms) showPetById
PASS POST http://localhost:4010/v1/pets -> 201 (2 ms) createPets
Requests: 3 passed, 0 failed, 0 skipped
Contract: 3 requests (0 undocumented), 0 errors, 0 warnings
Coverage: 70.0% of 10 branches, 3 of 3 operations
```

## Options

| Option | Description |
| --- | --- |
| `FILES` | `.http` and `.rest` files, or directories to search for them |
| `--spec SPEC` | The OpenAPI 3.0, 3.1 or 3.2 (or Swagger 2.0) document HTTP requests are validated against, the GraphQL schema (`.graphql`, `.graphqls`, `.gql`) of GraphQL requests, the `.proto` file of `GRPC` requests, or the AsyncAPI 3.x document WS, MQTT, KAFKA and AMQP requests and the requests of its HTTP channels are validated against |
| `--proto-path DIR` | A directory in which the imports of a `.proto` `--spec` are looked up; can be repeated |
| `--generate` | Also send one request per operation (OpenAPI), root field (GraphQL), method (`.proto`) or channel (AsyncAPI; one per operation of an HTTP, Kafka or AMQP channel) of `--spec` |
| `--base-url URL` | The server's URL for `--generate` (the endpoint, required, for a GraphQL schema; `HOST:PORT` for a `.proto` file; `ws://` or `http://` for an AsyncAPI document's WebSocket and HTTP channels), also put before request URLs that start with `/` |
| `--mqtt-url URL` | The MQTT broker (`mqtt://host:port`) of an AsyncAPI document's channels for `--generate` |
| `--amqp-url URL` | The AMQP broker (`amqp://host:port/vhost`, `amqps://` over TLS) of an AsyncAPI document's AMQP channels for `--generate` |
| `--kafka-url URL` | A broker of the Kafka cluster (`kafka://host:port`, `kafkas://` over TLS) of an AsyncAPI document's Kafka channels for `--generate` |
| `--env NAME` | The environment of the environment files |
| `--env-file FILE` | An environment file to read after the others; can be repeated |
| `--var NAME=VALUE` | A variable above every other one; can be repeated |
| `--header NAME:VALUE` | A header for every request that doesn't have it; can be repeated |
| `--timeout MS` | How long to wait for a connection and for a response (30000) |
| `--message-timeout MS` | How long WS, MQTT, KAFKA and AMQP requests, GraphQL subscriptions and gRPC client and bidi streams wait for messages (5000); `# @timeout` overrides it |
| `--bail` | Stop at the first failed request |
| `--ca-cert FILE` | A PEM file of certificate authorities trusted for `https://`, `grpcs://`, `wss://`, `mqtts://`, `kafkas://` and `amqps://` URLs, besides the system's; can be repeated |
| `--insecure` | Don't verify the certificates of `https://`, `grpcs://`, `wss://`, `mqtts://`, `kafkas://` and `amqps://` servers; the traffic stays encrypted |
| `--client-cert FILE` | A PEM client certificate (chain) presented to `https://`, `grpcs://`, `wss://`, `mqtts://`, `kafkas://` and `amqps://` servers that ask for one (mTLS); needs `--client-key` |
| `--client-key FILE` | The unencrypted PEM private key of `--client-cert` |
| `--report FILE` | Write the JSON report |
| `--report-junit FILE` | Write the JUnit XML report |
| `--fail-on none\|errors\|warnings` | The contract violations that fail the run (`errors`) |
| `--min-coverage PERCENT` | Fail the run below this coverage of `--spec` (a GraphQL schema: the share of its fields; a `.proto` file: the share of its methods; an AsyncAPI document: the share of its operations' messages) |
| `--severity CHECK=LEVEL` | Change a contract check's severity: `error`, `warn`, `info` or `off` (`ignore`); can be repeated. The checks are those of the OpenAPI, gRPC and AsyncAPI proxies or, with a GraphQL schema, of the GraphQL proxy; an unknown check is a bad option |
| `--seed N` | Seed of the values `--generate` makes up |
| `--formatter stylish\|json` | A line per request and a summary (`stylish`), or a JSON object per line |
| `--quiet` | Show only failed requests and the summary |
| `--verbose` | Also show the warnings and notes of the contract checks, and the scripts' log |

### --fail-on errors|warnings|none

A failed request (an assertion, a script's test or error, no response, an unknown `@ref`) always
fails the run. `--fail-on` says which contract violations do too: `errors` (the default),
`warnings` (errors and warnings) or `none`. A request with such violations is shown as `FAIL`.

### --formatter json

Each request is a line `{"event":"request",...}` with the members of the JSON report's
`requests`; warnings are `{"event":"warning","file":...,"line":...,"message":...}`, notes about a
Swagger 2.0 specification `{"event":"info",...}` with the same members; the last line is
`{"event":"summary","formatVersion":1,"total":...,"passed":...,"failed":...,"skipped":...,"gate":{...}}`
with the contract's `errors`, `warnings` and `coverage` when there is a `--spec`. Its
`formatVersion` versions the whole stream, as `formatVersion` versions the report.

## Reports

`--report` writes the [contract report](/docs/reference/server/contract-testing)
of `apigenie-server` with `"report": "apigenie-test"`, plus `tests` (the counts and the run's gate)
and `requests` (each request's file, line, URL, status, duration, operation, outcome, failures,
assertions, violations and log; `grpc` and `grpcStatus` for `GRPC` requests). With a `.proto` file
the contract members are those of the gRPC report (`"specification": "grpc"`, `methods`,
`services`), with a GraphQL schema those of the GraphQL report (`"specification": "graphql"`,
`rootFields`, `fields`, `uncoveredFields`, `operations`). Without `--spec`, only `report`, `formatVersion`, `version`, `startedAt`, `endedAt`, `tests` and `requests` are written.
`formatVersion` is the version of the format, which changes only when a member is removed or
changes meaning (see [Compatibility](/docs/reference/cli/compatibility)); `version` is its
former name, kept for one release. The JSON Schema of the report is
[`apigenie-test-report.json`](https://apigenie.pl/schemas/apigenie-test-report.json). `--report-junit` writes a `requests` suite (a test case per request: failed, or
skipped for disabled requests) and, with `--spec`, the contract suite of `apigenie-server`.
With an AsyncAPI document, the report is the AsyncAPI proxy's (`"specification": "asyncapi"`,
channels, operations with their messages, violations); WS and MQTT requests also have `channel`,
`address`, `messagesSent`, `messagesReceived` and `notes` (what wasn't checked, and why), and the
coverage `--min-coverage` checks is the share of the operations' messages seen.

## Exit status

| Code | Meaning |
| --- | --- |
| 0 | Every request passed and the contract gate held |
| 2 | Bad options: no request files without `--generate`, `--generate` without `--spec`, or with a GraphQL schema or a `.proto` file but no `--base-url`, `--min-coverage` without `--spec`, an unknown `--severity` check; a request file, an environment file or the specification (an OpenAPI or AsyncAPI document, a GraphQL schema, or a `.proto` file with its imports) that can't be read or parsed; `--generate` without a URL it can use (no `--base-url` and no absolute server URL in the document, or a URL of the wrong scheme); a report that can't be written, or an unexpected error |
| 5 | A request failed, or contract violations (`--fail-on`) or the coverage (`--min-coverage`) failed the run |

## See also

- [`apigenie-server`](/docs/reference/server), whose validating proxy checks traffic
  something else drives, with the same checks and reports
- [lint](/docs/reference/cli/lint), which checks the document itself
