apigenie-generator server phoenix (AsyncAPI)
Tier 2 · Release 0.12.0: Level 1 ✓ · Level 2 ✓ · Level 3 ✓ · Verification
Generate a Phoenix application of the WebSocket and Kafka channels of an AsyncAPI document.
Usage
apigenie-generator server phoenix [...OPTIONS] INPUT_FILE
Description
apigenie-generator server phoenix generates from OpenAPI documents (see
apigenie-generator server phoenix) and from
AsyncAPI 3.x documents, telling them apart by their openapi or asyncapi field. Documents
referencing other files are bundled first.
From an AsyncAPI document it generates a Mix project (Elixir 1.18 or later): the WebSocket
channels on Phoenix 1.7 with Bandit, and the Kafka channels on
brod and
Broadway. By default the application is the one the
document describes: it sends the messages of send operations and receives the messages of
receive operations. Set the perspective option to counterpart to swap them; the
counterpart has the Kafka channels only, since Phoenix serves WebSocket channels as the
application. The project depends on snappyer, a NIF that
needs a C++ compiler, so that brod reads Snappy-compressed records (the default of Go’s franz-go
producers); LZ4 and zstd records need lz4b or ezstd added by hand.
The generator writes the channels of servers with the ws/wss and kafka/kafka-secure
protocols, and channels with a WebSocket or Kafka binding in documents without servers. Channels
of other protocols (MQTT, AMQP, HTTP) are skipped and reported as info.
defmodule MyApp.PickupDesk do
@behaviour CarpetExpressEventsWeb.Channels.PickupDesk.Handler
alias CarpetExpressEvents.Models
@impl true
def request_pickup(%Models.PickupRequest{} = request, _session) do
# the reply gets the request's requestId when it has none
{:reply, %Models.PickupConfirmed{request_id: nil, pickup: book(request)}}
end
end
defmodule MyApp.LabelFarm do
@behaviour CarpetExpressEvents.Kafka.RequestLabelRender
@impl true
def render_label(message, _metadata) do
# sent to carpetexpress.label-render.results with the request's correlation id
{:reply, %CarpetExpressEvents.Messages.LabelRendered{key: message.key, payload: render(message.payload)}}
end
end
# config/config.exs
config :carpet_express_events, CarpetExpressEventsWeb.Channels.PickupDesk.Handler,
impl: MyApp.PickupDesk
The project, for the application carpet_express_events (named by app, --package-name or
the document’s title):
| File | Contents |
|---|---|
lib/<app>/models/*.ex |
the structs of the schemas the payloads use, decoded and encoded by decode.ex and encode.ex (as the OpenAPI models of server phoenix) |
lib/<app>/messages/*.ex |
per Kafka message, a struct (payload, headers, key) and the description of its record |
lib/<app>/avro/*.ex |
the structs of the Avro records and the enums of the payloads, with their schemas |
lib/<app>/kafka/security.ex |
the security schemes of the Kafka servers and brod’s client configuration of each |
lib/<app>/kafka.ex |
the supervisor of the Kafka processes and a function per topic |
lib/<app>/kafka/<operation>.ex |
per sent operation, its send (and request) functions; per received operation, the behaviour of its callbacks |
lib/<app>/kafka/<operation>_consumer.ex |
per received operation, its Broadway pipeline |
lib/<app>_web/user_socket.ex, channels/*.ex |
the socket of the Phoenix clients, a Phoenix channel per WebSocket channel and the behaviour of its callbacks (channels/<channel>/handler.ex) |
lib/<app>_web/security.ex |
the behaviour checking the credentials of the WebSocket channels with security |
lib/<app>_web/endpoint.ex |
the endpoint: the socket, then the plain WebSockets of the channels |
lib/<app>/runtime/, lib/<app>_web/runtime/ |
the runtime of the Kafka processes and of the channels |
config/*.exs, mix.exs, .formatter.exs, .credo.exs |
the project; config/runtime.exs reads PORT, KAFKA_BROKERS, KAFKA_SECURITY and, when Avro payloads are sent, KAFKA_SCHEMA_REGISTRY_URL |
- WebSocket channels, two transports. Each channel is a Phoenix channel: Phoenix clients join
its topic on
/socket/websocket, the address with its parameters as segments (/tracking/{shipmentId}istracking:<shipmentId>), with the ws binding’s query as the join payload. The same channel is served as a plain WebSocket at its address, where the frames are the messages as the AsyncAPI document describes them (theprefixof<Root>Web.Runtime.PlainSocketPlugmounts the addresses under a path). Both transports decode the address parameters and the query, call the optionaljoin/2callback, decode received messages into their structs before the callbacks get them, and send what the channel module’spush_<message>/2(one connection) andbroadcast_<message>(every connection to the topic) functions send. A request’s callback returns{:reply, message}: the reply of the Phoenix protocol, or a frame on a plain WebSocket, with the request’s correlation id copied when it has none. - Received Kafka operations. A Broadway pipeline per operation reads its topic in its
consumer group and hands each record to a callback of the operation’s behaviour
(
CarpetExpressEvents.Kafka.PublishShipmentEvents), named after the message. Several messages of a channel are told apart by a header or a payload property with a distinct constant value (ce_type), otherwise by the first message whose payload and headers decode. Records matching no message go to the optionalon_unknown/1, records that don’t decode to the optionalon_invalid/2(and fail). Batching:batch_sizeandbatch_timeoutfromx-apigenie-broadwayof the operation’s (or channel’s) Kafka binding, else thebroadwayoption, else Broadway’s 100 records and 1000 ms; the pipeline module’s configuration overrides them, and the consumer group and offset reset policy too. - Sent Kafka operations.
send_<message>/1checks the headers of a message, encodes it and returns once the brokers have the record, keyed bykey. - Request/reply on Kafka, with a reply channel:
request_<message>/2sends the request with its correlation id (generated when it has none) and waits for the reply (:timeout, 30 seconds by default), read by a consumer of the reply topic without consumer group; a replier’s callback returns the reply, sent to the reply topic with the request’s correlation id.
Group ids come from the groupId schemas of the Kafka operation bindings (their const, single
enum value, default or first example), else <app in kebab case>-<operation>. Client ids come
from their clientId schemas the same way: the brod client of the producers has the
application’s, <app in kebab case> by default, and the Broadway pipelines and the consumers of
replies the operation’s, else <app in kebab case>-<operation>. The partitions, replicas and
topic configuration of channel bindings are documented on the topic functions; the generated code
creates no topics.
Not supported, and reported as warnings: payloads of non-JSON schema formats other than Avro (the
messages are skipped, and operations left without a message too), reply addresses given by runtime
expressions, Kafka topics with parameters, WebSocket addresses whose segments mix text and
parameters, and security schemes a WebSocket handshake can’t carry (userPassword, X509, the
SASL schemes). Channels of other protocols (MQTT, AMQP, HTTP) are out of scope. The models check
the schemas’ constraints, if/then/else, dependentRequired, dependentSchemas and not
when they decode a payload.
Kafka security
Each SASL or client certificate scheme of the Kafka servers has a function of
<Root>.Kafka.Security, named after the scheme key, returning brod’s client configuration of the
scheme: kafka_scram(username, password) for the SASL scheme kafkaScram (over TLS on
kafka-secure servers), kafka_mtls(cert_file, key_file, ca_file) for the client certificate of
kafkaMtls. client_config(scheme) takes the credentials from environment
variables named after the scheme key, e.g. KAFKA_SCRAM_USERNAME and KAFKA_SCRAM_PASSWORD, or
KAFKA_MTLS_CERT, KAFKA_MTLS_KEY and KAFKA_MTLS_CA. The Kafka supervisor connects with the
scheme of its :security option, which config/runtime.exs reads from KAFKA_SECURITY;
:client_config adds to brod’s configuration.
config :carpet_express_events, :kafka, brokers: "kafka:9093", security: "kafkaScram"
WebSocket security
A channel’s security is the schemes of its operations, else of its servers, that a WebSocket
handshake carries: API keys in the query, a header or a cookie (httpApiKey), basic credentials
and bearer tokens (http with the basic or bearer scheme, oauth2, openIdConnect). A
connection to a channel with security is accepted when the implementation of <Root>Web.Security
accepts its credential of one of the schemes, and refused otherwise: the join of a Phoenix client
with {:error, %{reason: "unauthorized"}}, the handshake of a plain WebSocket with 401
Unauthorized. authenticate/2 returns {:ok, principal} to accept the connection,
and principal becomes the session’s principal; it returns {:error, reason} to refuse it.
defmodule MyApp.Security do
@behaviour CarpetExpressEventsWeb.Security
# each returns {:ok, merchant} or {:error, :invalid}
@impl true
def authenticate("streamToken", {:api_key, token}), do: MyApp.Tokens.verify(token)
def authenticate("oauth2", {:bearer, token, scopes}), do: MyApp.OAuth.verify(token, scopes)
end
config :carpet_express_events, CarpetExpressEventsWeb.Security, impl: MyApp.Security
Without an implementation, connections to channels with security are refused. The headers schema
of a channel’s ws binding is decoded into session.params.headers. Phoenix clients send query
parameters as the parameters of the socket or of the join, and only x- headers reach Phoenix
channels (the endpoint’s connect_info: [:x_headers]); plain WebSockets get every header.
Avro payloads
A message whose payload is an Avro schema (a schemaFormat of application/vnd.apache.avro)
carries a struct of its record, lib/<app>/avro/<record>.ex, with its schema, its fields and
their defaults; Avro enums are atoms. The record is written and read by
<Root>.Runtime.Kafka.AvroCodec, a dependency-free encoder and decoder of the Avro binary
encoding with the logical types date, time-*, timestamp-* and local-timestamp-* (as
Date, Time, DateTime and NaiveDateTime).
The id of the record’s schema goes where the Kafka binding’s schemaIdLocation says: in the
apicurio.value.globalId header, or in the payload after a zero byte (the Confluent wire format).
The sent schemas are registered in a Confluent-compatible registry, the schemaRegistryUrl of the
Kafka server by default, under the subject of the binding’s schemaLookupStrategy
(<topic>-value by default), the first time each is sent; the :schema_registry option of the
Kafka supervisor (url, username, password, or a register function replacing the requests)
and KAFKA_SCHEMA_REGISTRY_URL change it. The registry is not asked when reading: the reader’s
schema is the generated one. Unions are supported when they make a type nullable (["null", T]);
messages with other unions, with decimals or whose payload is not a record are skipped with a
warning, as are request/reply operations with Avro messages.
Options
-c, –config
Path to the configuration file to use. See the “Configuration” section for more information.
–check
Writes nothing: compares what the generator would write with the output directory, prints the files that would be created or changed, and exits with 1 when there are any (since 0.12.0). See Check that generated code is up to date.
-o, –output
Directory where the project is generated.
–package-name
Name of the Mix application, converted to snake_case (carpet-express-events gives
carpet_express_events, root module CarpetExpressEvents); defaults to the title of the API
(events when it gives no name). Its kebab case is the base of default group ids.
–package-version
Version of the Mix project; 0.1.0 by default.
-p, –property
Applies to OpenAPI documents only; using it with an AsyncAPI document is an error.
Configuration
The options are read from the asyncapi section of the configuration of apigenie-generator server phoenix; its other keys configure the generation from OpenAPI documents.
version: '1.0'
commands:
generate:
server_phoenix:
asyncapi:
app: carpet_express_events
namespace: CarpetExpressEvents
perspective: application
broadway: { batch_size: 100, batch_timeout: 1000, concurrency: 10 }
| Option | Type | Default | Description |
|---|---|---|---|
app |
string | from --package-name or the title |
Name of the Mix application (snake_case) |
namespace |
string | the application in PascalCase | Root module of the generated modules; the web modules are in <Root>Web |
perspective |
application | counterpart |
application |
The described application, or its counterpart, receiving what it sends |
field_prefix |
string | field_ |
Prefix of struct fields whose property names give no valid name |
module_overrides |
map | Module names of payload schemas, e.g. ShipmentId: ShipmentRef |
|
dev_deps |
boolean | true |
credo and dialyxir as dependencies of the dev and test environments |
broadway |
map | batch_size, batch_timeout (ms) and concurrency of the pipelines |
Verification
The codegen conformance suite runs this target on the Carpet Express AsyncAPI fixture. Level 1 compares the output with its snapshots and checks each feature, Level 2 runs the language's type checker and linter, and Level 3 runs the generated code in contract tests against the fixture. All targets.
This target's support tier is Tier 2. It goes through all three levels before every release, like Tier 1. It doesn't aim to support every feature: a feature of the fixture can stay unsupported, with the reason its manifest gives.
| Results | Level 1 | Level 2 | Level 3 |
|---|---|---|---|
Release 0.12.0, server phoenix | ✓ snapshots of 3.0; 41 supported, 0 not yet | ✓ pass (Erlang/OTP 29 [erts-17.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns]; Mix 1.20.4 (compiled with Erlang/OTP 29)) | ✓ 11 of 11 operations pass (Erlang/OTP 29 [erts-17.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns]; Mix 1.20.4 (compiled with Erlang/OTP 29)) |
Release 0.12.0, server phoenix (counterpart) | ✓ snapshots of 3.0; 30 supported, 0 not yet | ✓ pass (Erlang/OTP 29 [erts-17.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns]; Mix 1.20.4 (compiled with Erlang/OTP 29)) | ✓ 11 of 11 operations pass (Erlang/OTP 29 [erts-17.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns]; Mix 1.20.4 (compiled with Erlang/OTP 29)) |
Development version, server phoenix | ✓ snapshots of 3.0; 41 supported, 0 not yet | runs before a release | runs before a release |
Development version, server phoenix (counterpart) | ✓ snapshots of 3.0; 30 supported, 0 not yet | runs before a release | runs before a release |