Using Aiven for Apache Kafka

Kalix connects to Aiven's Kafka service via TLS, using a CA certificate provided by Aiven for the service, authenticating using SASL (Simple Authentication and Security Layer) SCRAM.

In this guide we use the default avnadmin account, but you may want to create a specific service user to use for your Kalix service connection.

Steps to connect to an Aiven Kafka service

Log in to the Aiven web console and select the Aiven Kafka service Kalix should connect to.
Enable SASL for your Aiven Kafka (See Aiven’s Use SASL Authentication with Apache Kafka)
1. Scroll down the Service overview page to the Advanced configuration section.
2. Turn on the setting labelled kafka_authentication_methods.sasl, and click Save advanced configuration.
3. The connection information at the top of the Service overview page will now offer the ability to connect via SASL or via client certificate. Select SASL in "Authentication Method" to show the right connection details:
4. Download the CA Certificate via the link in the connection information.
Ensure you are on the correct Kalix project
```
kalix config get-project
```
Create a Kalix TLS CA secret with the CA certificate for the service (e.g. called kafka-ca-cert)
```
kalix secret create tls-ca kafka-ca-cert --cert ./ca.pem
```
Copy the CA password from the "Connection Information" and store it in a Kalix secret (e.g. called kafka-secret)
```
kalix secret create generic kafka-secret --literal pwd=<the password>
```

Use kalix projects config to set the broker details. Set the Aiven username and service URI according to the Aiven connection information page.

kalix projects config set broker \
  --broker-service kafka \
  --broker-auth scram-sha-256  \
  --broker-user avnadmin \
  --broker-password-secret kafka-secret/pwd \
  --broker-bootstrap-servers <kafka...aivencloud.com:12976> \
  --broker-ca-cert-secret kafka-ca-cert

The broker-password-secret and broker-ca-cert-secret refer to the names of the Kalix secrets created earlier rather than the actual secret values.

An optional description can be added with the parameter --description to provide additional notes about the broker.

The broker config can be inspected using:

kalix projects config get broker

Create a topic

To create a topic, you can either use the Aiven console, or the Aiven CLI.

Browser

Instructions from Aiven’s Creating an Apache Kafka topic

Open the Aiven Console.
In the Services page, click on the Aiven for Apache Kafka® service where you want to crate the topic.
Select the Topics tab:
1. In the Add new topic section, enter a name for your topic.
2. In the Advanced configuration you can set the replication factor, number of partitions and other advanced settings. These can be modified later.
Click Add Topic on the right hand side of the console.

You can now use the topic to connect with Kalix.

Aiven CLI

See Aiven’s Manage Aiven for Apache Kafka topics

avn service topic-create \
  <service name> \
  <topic name> \
  --partitions 3 \
  --replication 2

You can now use the topic to connect with Kalix.

Delivery characteristics

When your application consumes messages from Kafka, it will try to deliver messages to your service in 'at-least-once' fashion while preserving order.

Kafka partitions are consumed independently. When passing messages to a certain entity or using them to update a view row by specifying the id as the Cloud Event ce-subject attribute on the message, the same id must be used to partition the topic to guarantee that the messages are processed in order in the entity or view. Ordering is not guaranteed for messages arriving on different Kafka partitions.

Correct partitioning is especially important for topics that stream directly into views and transform the updates: when messages for the same subject id are spread over different transactions, they may read stale data and lose updates.

To achieve at-least-once delivery, messages that are not acknowledged will be redelivered. This means redeliveries of 'older' messages may arrive behind fresh deliveries of 'newer' messages. The first delivery of each message is always in-order, though.

When publishing messages to Kafka from Kalix, the ce-subject attribute, if present, is used as the Kafka partition key for the message.

Testing Kalix eventing

See Testing Kalix eventing