This article will describe how to use Kubernetes Probes to monitor the health of your Kafka pods.
Container Health
K8s provides a number of features that allow you to build robust solutions, such as the ability to automatically restart unhealthy containers. The kubelet uses liveness probes to know when to restart a container. For example, liveness probes could catch a deadlock, where an application is running, but unable to make progress. Restarting a container in such a state can help to make the application more available despite bugs.
Liveness Probes
Allow to automatically determine whether or not a container application is in a healthy state. Liveness probes allow you to customise this detection mechanism.
In the Kafka StatefulSet configuration you can add:
livenessProbe:
failureThreshold: 6
initialDelaySeconds: 60
periodSeconds: 60
successThreshold: 1
tcpSocket:
port: 9092
timeoutSeconds: 5Liveness probes run constantly, the periodSeconds field specifies every 60 seconds. The initialDelaySeconds field tells the kubelet that it should wait 5 seconds before performing the first probe. timeoutSeconds is the amount of time that the probe will wait for a response from the port 9092 and determine success or failure. If the probe fails 6 times the pod will be restarted.
Readiness Probes
Are used to determine when a container is ready to accept requests. Therefore, user traffic will not be sent to a particular pod until its containers have all passed the readiness checks.
The configuration is similar to the Liveness probe:
readinessProbe:
failureThreshold: 6
initialDelaySeconds: 120
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 9092
timeoutSeconds: 5In this configuration, the Readiness probe will wait 2 minutes and then every 10 seconds will try to connect with the port 9092. In the first successful connection, the pod will be ready to receive requests.
Summary
Liveness and Readiness probes can a powerful way to monitor and automatically recover your Kafka pods from failures. You can see a detailed list of the parameters for probes in the Kubernetes documentation. At the beginning of the implementation closely monitor these configurations to make sure they are working as you expect. You don't want your pods restating every 10 seconds due to a bad configuration!
Comments