apigenie-generator server kafkajs

Tier 2 · Release 0.12.0: Level 1 ✓ · Level 2 ✓ · Level 3 ✓ · Verification

Generate a TypeScript (Node.js) Kafka application of an AsyncAPI document, on KafkaJS.

Usage

apigenie-generator server kafkajs [...OPTIONS] INPUT_FILE

Description

The input must be an AsyncAPI 3.x document. Documents referencing other files are bundled first.

The generator handles the Kafka channels: channels of a server whose protocol is kafka or kafka-secure, or with a kafka binding when the document has no servers. Other channels are skipped with an info message. The generated code runs on Node.js with KafkaJS 2.2.4, its only dependency.

The application is the one the document describes: it sends the messages of send operations and receives those of receive operations. Set the perspective option to counterpart to generate the other side.

import { KafkaApp } from "order-service-kafka-app";

const app = new KafkaApp({ brokers: ["localhost:9092"] }, {
  handleOrderCommands: {
    onPlaceOrderMessage: (message) => ({ payload: { requestId: message.payload.requestId, orderId: "o-1" } }),
    onCancelOrderMessage: () => undefined,
  },
});
await app.start();
await app.publishStockLevels.sendStockLevelMessage({ key: "ABC-0001", payload: { sku: "ABC-0001", quantity: 3 } });
await app.stop();

KafkaApp takes the handlers of all received operations. start() connects the producer and the consumers and resolves once every consumer joined its group; stop() commits, leaves the groups and disconnects. Each sent operation has a sender property.

Every sent operation gets a sender property with a send<Message>(message) method per message. It validates the message, sets the constant headers (e.g. ce_type) and resolves once the brokers have it. A message is { key?, headers, payload }, with the headers under their names on the wire (ce_id). The method of an operation with a reply sends the request with a new correlation id unless it has one, and resolves with the reply: the reply message, or { name, message } when there are several. It rejects with ReplyTimeoutError when no reply arrives in time ({ timeoutMs }, KafkaOptions.replyTimeoutMs, 30 seconds by default).

Every received operation gets a handler interface: on<Message>(message, record) for each of its messages and an optional onUnknown(record). Records are matched to messages by their ce_type (or other constant) header, by a constant payload property, or, failing both, by the first message whose payload is valid. The handlers of an operation with a reply return the reply (or undefined for none), which is sent with the request’s correlation id. Invalid records are reported to KafkaOptions.onError and skipped. A handler that throws is reported too and the record is retried (KafkaJS doesn’t commit it).

Payloads and headers are validated against their schemas by generated functions without dependencies; invalid messages throw MessageValidationError, whose issues point into /headers or /payload. KafkaOptions takes the brokers, the producer’s clientId, config (KafkaJS options for every client: ssl, sasl, retry, logLevel, …), groupIds replacing the generated consumer groups, fromBeginning (true by default), onError and schemaRegistry. The generated code creates no topics.

src/kafka.ts lists the SASL and X.509 schemes of each server in SECURITY, with the environment variables of their credentials (KAFKA_SCRAM_USERNAME, …), and securityConfig(server) turns them into the ssl and sasl options of KafkaOptions.config. Pass it as new KafkaApp({ brokers: [SERVERS[server]], config: securityConfig(server) }, handlers). It throws when the server has schemes and none of them has all its variables set. Strings with contentEncoding: base64 are Uint8Arrays in messages and base64 on the wire.

Avro payloads (a schemaFormat of application/vnd.apache.avro) get interfaces for their records and unions of string literals for their enums, in src/messages.ts, and are encoded in Avro’s binary encoding by src/avro_runtime.ts, without dependencies. Their schema id comes from KafkaOptions.schemaRegistry. It is either a Confluent-compatible registry, { url, username?, password?, timeoutMs? }, where the schema is registered the first time it is sent (a request times out after timeoutMs, 30 seconds by default; a registry that refuses the schema or can’t be reached in time rejects the send with an Error naming the subject), or a function (subject, schema) => Promise<number> returning the id. The message binding’s schemaIdLocation puts the id before the payload (payload, Confluent’s format, the default) or in the apicurio.value.globalId header (header), and its schemaLookupStrategy names the subject (TopicIdStrategy, the default, gives <topic>-value). Received records are decoded with the generated schema. Avro int and long are numbers; a received record with a long outside JavaScript’s safe integers is invalid and goes to onError.

The package contains src/types.ts (payload types), src/messages.ts (message and header interfaces), src/kafka.ts (servers, topics, ids, senders, handlers and the KafkaApp), src/codecs.ts (validators and the descriptions of the messages the runtime works with), src/runtime.ts, src/validation_runtime.ts, src/avro_runtime.ts (with Avro payloads only) and src/index.ts, with package.json and tsconfig.json.

Not supported, and reported as warnings: topics with parameters, replies at runtime-expression addresses, replies without correlation ids, and security schemes other than plain, scramSha256, scramSha512, userPassword and X509 (such as gssapi).

apigenie-generator client kafkajs generates the other side on the same runtime. The design is ADR-120.

Extensions

Extension Effect
x-apigenie.codegen.name name of the type of a component schema
x-apigenie.codegen.disabled true on a channel or an operation leaves it out (with the operations of a channel left out)

The extensions can live in an overlay for this target instead of in the document, e.g. kafkajs.overlay.yaml:

overlay: 1.0.0
info:
  title: TypeScript names
  version: 1.0.0
actions:
  - target: $.components.schemas.Order
    update:
      x-apigenie:
        codegen:
          name: PurchaseOrder
  - target: $.operations.publishAudit
    update:
      x-apigenie:
        codegen:
          disabled: true
apigenie-generator server kafkajs asyncapi.yaml --overlay kafkajs.overlay.yaml -o out

Options

-c, –config

Path to the configuration file to use. See the “Configuration” section for more information.

–overlay OVERLAY

Path to an OpenAPI Overlay (1.0 to 1.2) applied to the input before generating, e.g. one setting the extensions of the “Extensions” section. Can be repeated: the overlays apply in order, after those listed under commands.generate.server_kafkajs.overlay in the configuration file (with paths relative to it). Warnings and errors about a value an overlay set point to the overlay. The overlays apply to the AsyncAPI document bundled with the files it references.

–strict

Makes an overlay action whose target selects nothing an error (exit status 2, nothing is generated) instead of a warning (OVERLAY:LINE:COL: action #N (TARGET) selects no nodes.).

–check

Writes nothing: compares what the generator would write with the output directory, prints the files that would be created or changed, and exits with 1 when there are any (since 0.12.0). See Check that generated code is up to date.

-o, –output

Directory where the package is generated.

–package-name

Name of the generated package. Defaults to the title of the API followed by -kafka-app. It is also the base of the generated consumer group and client ids.

–package-version

Version of the generated package. Defaults to the version of the API.

Configuration

version: '1.0'
commands:
  generate:
    server_kafkajs:
      layout: package
      import_extension: .js
      perspective: application
      validation: true
Option Description
layout package (default): an npm package with sources in src/; sources: the sources only
import_extension Extension of relative imports: .js (default, Node.js) or an empty string (bundlers)
perspective application (default): the described application; counterpart: its counterpart
validation true (default): validate payloads against their schemas; false: check the headers only

Unknown options are errors.

Verification

The codegen conformance suite runs this target on the Carpet Express AsyncAPI fixture. Level 1 compares the output with its snapshots and checks each feature, Level 2 runs the language's type checker and linter, and Level 3 runs the generated code in contract tests against the fixture. All targets.

This target's support tier is Tier 2. It goes through all three levels before every release, like Tier 1. It doesn't aim to support every feature: a feature of the fixture can stay unsupported, with the reason its manifest gives.

ResultsLevel 1Level 2Level 3
Release 0.12.0, server kafkajs✓ snapshots of 3.0; 31 supported, 0 not yet✓ pass (Version 6.0.3, ESLint 10 + typescript-eslint 8, KafkaJS 2.2.4)✓ 7 of 7 operations pass (Version 6.0.3, ESLint 10 + typescript-eslint 8, KafkaJS 2.2.4)
Release 0.12.0, server kafkajs (counterpart)✓ snapshots of 3.0; 31 supported, 0 not yet✓ pass (Version 6.0.3, ESLint 10 + typescript-eslint 8, KafkaJS 2.2.4)✓ 7 of 7 operations pass (Version 6.0.3, ESLint 10 + typescript-eslint 8, KafkaJS 2.2.4)
Development version, server kafkajs✓ snapshots of 3.0; 31 supported, 0 not yetruns before a releaseruns before a release
Development version, server kafkajs (counterpart)✓ snapshots of 3.0; 31 supported, 0 not yetruns before a releaseruns before a release
Was this helpful?