This page describes the steps to quickly implement basic monitoring for Kafka Brokers using Prometheus and Grafana.
Enabling Prometheus for Kafka Brokers
Create a ConfigMap with the Kafka metrics to be scraped by Prometheus. A basic example can be found in this repository: https://github.com/confluentinc/jmx-monitoring-stacks/blob/6.0.1-post/jmxexporter-prometheus-grafana/assets/prometheus/jmx-exporter/kafka_broker.yml
Download the JMX exporter JAR file:
wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.15.0/jmx_prometheus_javaagent-0.15.0.jar Create a Secret from the JAR file:
kubectl create secret generic sys-jmx-prometheus-jar --from-file jmx_prometheus_javaagent-0.15.0.jarThe Secret is used because I didn't want to customise the Kafka Broker Docker image.
Add the ConfigMap and Secret to the Statefulset manifest file:
...
volumeMounts:
- mountPath: /etc/prometheus
name: jmx-config
readOnly: true
- mountPath: /etc/prometheus/jar
name: jmx-prometheus-jar
readOnly: true
...
volumes:
...
- configMap:
defaultMode: 420
name: sys-kafka-jmx-configmap
name: jmx-config
- name: jmx-prometheus-jar
secret:
defaultMode: 420
secretName: sys-jmx-prometheus-jar
...Add the Prometheus configuration files to the KAFKA_OPTS variable at startup:
-javaagent:/etc/prometheus/jar/jmx_prometheus_javaagent-0.15.0.jar=9404:/etc/prometheus/jmx-kafka-prometheus.ymlThe default port used by JMX exporters is 9404 (if you do not add them explicitly).
Prometheus
The next yaml file install Prometheus version 2.44.0. It also creates a PersistentVolume to store the collected metrics, and a ConfigMap with the Kafka Broker addresses to scrape (scrape_configs).
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: prometheus-vol
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
storageClassName: px-sc-general
---
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-server-conf
labels:
name: prometheus-server-conf
data:
prometheus.yml: |-
global:
scrape_interval: 60s
evaluation_interval: 60s
scrape_timeout: 60s
rule_files:
- /etc/prometheus/prometheus.rules
scrape_configs:
- job_name: 'kafka'
static_configs:
- targets:
- "kafka-0.dev-kafka-headless.kafka.svc.cluster.local:9404"
- "kafka-1.dev-kafka-headless.kafka.svc.cluster.local:9404"
- "kafka-2.dev-kafka-headless.kafka.svc.cluster.local:9404"
labels:
env: "dev"
---
apiVersion: v1
kind: Service
metadata:
name: prometheus-service
annotations:
prometheus.io/scrape: 'true'
prometheus.io/port: '9090'
spec:
selector:
app: prometheus-server
type: NodePort
ports:
- port: 8080
targetPort: 9090
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus-deployment
labels:
app: prometheus-server
spec:
replicas: 1
selector:
matchLabels:
app: prometheus-server
template:
metadata:
labels:
app: prometheus-server
spec:
containers:
- name: prometheus
image: prom/prometheus
args:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus/"
- "--storage.tsdb.retention.time=10d"
- "--storage.tsdb.retention.size=50GiB"
ports:
- containerPort: 9090
resources:
limits:
cpu: "1"
memory: 1400Mi
requests:
cpu: 250m
memory: 512Mi
volumeMounts:
- name: prometheus-config-volume
mountPath: /etc/prometheus/
- name: prometheus-storage-volume
mountPath: /prometheus/
securityContext:
fsGroup: 65534
runAsGroup: 65534
runAsNonRoot: true
runAsUser: 65534
volumes:
- name: prometheus-config-volume
configMap:
defaultMode: 420
name: prometheus-server-conf
- name: prometheus-storage-volume
persistentVolumeClaim:
claimName: prometheus-volGrafana
The next yaml file install Grafana 9.5.3 and creates a ConfigMap that creates a datasource for the Prometheus we installed before.
---
apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-datasources
data:
prometheus.yaml: |-
{
"apiVersion": 1,
"datasources": [
{
"access":"proxy",
"editable": true,
"name": "prometheus",
"orgId": 1,
"type": "prometheus",
"url": "http://prometheus-service.kafka.svc:8080",
"version": 1
}
]
}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: grafana
spec:
replicas: 1
selector:
matchLabels:
app: grafana
template:
metadata:
name: grafana
labels:
app: grafana
spec:
containers:
- name: grafana
image: grafana/grafana:9.5.3
ports:
- name: grafana
containerPort: 3000
resources:
limits:
memory: "1Gi"
cpu: "500m"
requests:
memory: 500M
cpu: "500m"
volumeMounts:
- mountPath: /var/lib/grafana
name: grafana-storage
- mountPath: /etc/grafana/provisioning/datasources
name: grafana-datasources
readOnly: false
volumes:
- name: grafana-storage
emptyDir: {}
- name: grafana-datasources
configMap:
defaultMode: 420
name: grafana-datasources
---
apiVersion: v1
kind: Service
metadata:
name: grafana
annotations:
prometheus.io/scrape: 'true'
prometheus.io/port: '3000'
spec:
selector:
app: grafana
type: NodePort
ports:
- port: 3000
targetPort: 3000
Visualising Metrics
In the newly installed Grafana, create a dashboard to show Kafka Broker metrics.
A basic example can be found in this repository:
This is an example of how the dashboard looks like:
That's it! With a few steps you're already monitoring your Kafka Broker!
Reference
Comments