Kafka ¶

Nais offers Kafka as a managed service through Aiven.

Start using Kafka by creating a Topic resource in one of our Kubernetes clusters.

A Topic belongs to one of the Kafka pools. A pool is a highly available, replicated Kafka cluster running at Aiven. After the topic is created, relevant users are added to the topic's access control list (ACL).

To get started with Kafka in your application, see accessing topics from an application.

What happens on deploy? ¶

When you deploy an application that requests access to Kafka, Naiserator will create an AivenApplication resource in the cluster. The AivenApplication has a name that matches the deployed application, and within its spec.kafka, the name of the secret to generate. Naiserator will request that a secret with this name used in the deployment.

When an AivenApplication resource is created or updated, Aivenator will create a new service user and generate credentials. These credentials are then inserted into the requested secret and used in the deployment.

If there is a problem generating the secret, this might fail your deployment. In this case, Aivenator will update the status part of the resource, with further information about the problem.

Application design guidelines ¶

Authentication and authorization ¶

The Nais platform will generate new credentials when your applications is deployed. Kafka requires TLS client certificates for authentication. Make sure your Kafka and/or TLS library can do client certificate authentication, and that you can specify a custom CA certificate for server validation.

Readiness and liveness ¶

Making proper use of liveness and readiness probes can help with many situations. If producing or consuming Kafka messages are a vital part of your application, you should consider failing one or both probes if you have trouble with Kafka connectivity. Depending on your application, failing liveness might be the proper course of action. This will make sure your application is restarted when it is experiencing problems, which might help.

In other cases, failing just the readiness probe will allow your application to continue running, attempting to move forward without being killed. Failing readiness will be most helpful during deployment, where the old instances will keep running until the new are ready. If the new instances are not able to connect to Kafka, keeping the old ones until the problem is resolved will allow your application to continue working.

FAQ/Troubleshooting ¶

Why do I have to specify a pool name if there is only one per environment? ¶

I can't produce/consume on my topic, with an error message like "topic not found". What's wrong? ¶

I can't produce/consume on my topic, with an error message like "not authorized". What's wrong? ¶

I get the error MountVolume.SetUp failed for volume "kafka-credentials" : secret ... not found ¶

Are Schemas backed up? ¶

See the Aiven documentation for more details.