Running Kafka in Kubernetes with KRaft mode and SSL

Learn how to launch an Apache Kafka with the Apache Kafka Raft (KRaft) consensus protocol and SSL encryption. This article is a continuation of my previous article Running Kafka in Kubernetes with KRaft mode.

Prerequisites

An understanding of Apache Kafka, Kubernetes, and Minikube.

The following steps were initially taken on a MacBook Pro with 32GB memory running MacOS Ventura v13.4.

Make sure to have the following applications installed:

• Docker v23.0.5

• Minikube v1.29.0 (running K8s v1.26.1 internally)

It's possible the steps below will work with different versions of the above tools, but if you run into unexpected issues, you'll want to ensure you have identical versions. Minikube was chosen for this exercise due to its focus on local development.

Deployment Components

The first step to enable SSL encryption is to a create public/private key pair for every server. You can find the detailed instructions to create the keys here.

The second step is to create two ConfigMaps that will be mounted as volumes for the Broker and the client with truststore, keystore, and password. You can find the detailed instructions to create the ConfigMaps here.

The deployment we will create will have the following components:

• Namespace: kafka This is the namespace within which all components will be scoped.

• Service Account: kafka Service accounts are used to control permissions and access to resources within the cluster.

• Headless Service: kafka-headless It exposes ports 9092 (for PLAINTEXT communication) and 9093 (for SSL traffic).

• StatefulSet: kafka It manages Kafka pods and ensures they have stable hostnames and storage.

The source code for this deployment can be found in this GitHub repository.

Specifically for the SSL configurations, the next parameters were configured in the StatefulSet:

• Configure the truststore, keystore, and password:

KAFKA_SSL_KEY_CREDENTIALS
KAFKA_SSL_KEYSTORE_CREDENTIALS
KAFKA_SSL_TRUSTSTORE_CREDENTIALS

• Configure the ports for the Kafka brokers to listen for SSL

KAFKA_ADVERTISED_LISTENERS 
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
KAFKA_LISTENERS

Creating the deployment

Clone the repo:

git clone https://github.com/rafaelmnatali/kafka-k8s.git
cd ssl

deploy Kafka using the following commands:

kubectl apply -f 00-namespace.yaml
kubectl apply -f 01-kafka-local.yaml

Verify communication across brokers

There should now be three Kafka brokers each running on separate pods within your cluster. Name resolution for the headless service and the three pods within the StatefulSet is automatically configured by Kubernetes as they are created, allowing for communication across brokers. See the related documentation for more details on this feature.

You can check the first pod's logs with the following command:

kubectl logs kafka-0

The name resolution of the three pods can take more time to work than it takes the pods to start, so you may see UnknownHostException warnings in the pod logs initially:

WARN [RaftManager nodeId=2] Error connecting to node kafka-1.kafka-headless.kafka.svc.cluster.local:29093 (id: 1 rack: null) (org.apache.kafka.clients.NetworkClient) java.net.UnknownHostException: kafka-1.kafka-headless.kafka.svc.cluster.local         ... 

But eventually each pod will successfully resolve pod hostnames and end with a message stating the broker has been unfenced:

INFO [Controller 0] Unfenced broker: UnfenceBrokerRecord(id=1, epoch=176) (org.apache.kafka.controller.ClusterControlManager)

Create a topic using the SSL endpoint

The Kafka StatefulSet should now be up and running successfully. Now we can create a topic using the SSL endpoint.

You can deploy Kafka Client using the following command:

kubectl apply -f 02-kafka-client.yaml

Check if the Pod is Running:

kubectl get pods 

Output:

NAME        READY   STATUS    RESTARTS   AGE 
kafka-cli   1/1     Running   0          12m

Connect to the pod kafka-cli:

kubectl exec -it kafka-cli -- bash

Create a topic named test-ssl with three partitions and a replication factor of 3.

kafka-topics --create --topic test-ssl --partitions 3 --replication-factor 3 --bootstrap-server ${BOOTSTRAP_SERVER} --command-config /etc/kafka/secrets/client_security.properties  Created topic test-ssl.

The environment variable BOOTSTRAP_SERVER contains the list of the brokers, therefore, we save time in typing.

List all the topics in Kafka:

kafka-topics --bootstrap-server kafka-0.kafka-headless.kafka.svc.cluster.local:9093 --list --command-config /etc/kafka/secrets/client_security.properties  
test 
test-ssl 
test-test

Summary and next steps

This tutorial showed you how to get Kafka running in KRaft mode on a Kubernetes cluster with SSL encryption. This is a step to secure communication between clients and brokers. I invite you to keep studying and investigating how to improve security in your environment.

In the next post, I'll be showing how to configure authentication between clients and Kafka!

References

• https://hub.docker.com/r/confluentinc/cp-kafka

• https://docs.confluent.io/platform/current/installation/docker/config-reference.html

• https://docs.confluent.io/platform/current/kafka/authentication_ssl.html#optional-settings

• https://github.com/confluentinc/kafka-images/blob/master/examples/kafka-cluster-ssl/docker-compose.yml

• https://rafaelnatali.wixsite.com/rmn-technology/blog/categories/kafka

• https://github.com/rafaelmnatali/kafka-k8s