Monitorando Kafka com Prometheus e Grafana

Esta página descreve os passos para implementar rapidamente monitoramento básico para Brokers do Kafka usando Prometheus e Grafana.

Habilitando o Prometheus para Brokers do Kafka

Crie um ConfigMap com as métricas do Kafka a serem coletadas pelo Prometheus. Um exemplo básico pode ser encontrado neste repositório: https://github.com/confluentinc/jmx-monitoring-stacks/blob/6.0.1-post/jmxexporter-prometheus-grafana/assets/prometheus/jmx-exporter/kafka_broker.yml

Download o arquivo JAR do JMX exporter:

wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.15.0/jmx_prometheus_javaagent-0.15.0.jar

Crie um Secret a partir do JAR file:

kubectl create secret generic sys-jmx-prometheus-jar --from-file jmx_prometheus_javaagent-0.15.0.jar

O Secret é utilizado porque eu não queria personalizar a imagem Docker do Kafka Broker.

Adicione o ConfigMap e Secret no Statefulset:

...
volumeMounts:
- mountPath: /etc/prometheus
  name: jmx-config
  readOnly: true
- mountPath: /etc/prometheus/jar
  name: jmx-prometheus-jar
  readOnly: true
...
volumes:
...
- configMap:
    defaultMode: 420
    name: sys-kafka-jmx-configmap
  name: jmx-config
- name: jmx-prometheus-jar
  secret:
    defaultMode: 420
    secretName: sys-jmx-prometheus-jar
...

Adicione os arquivos de configuração do Prometheus na inicialização da variável KAFKA_OPTS:

-javaagent:/etc/prometheus/jar/jmx_prometheus_javaagent-0.15.0.jar=9404:/etc/prometheus/jmx-kafka-prometheus.yml

A porta padrão usada pelos exportadores JMX é 9404 (se não forem adicionados explicitamente).

Prometheus

O arquivo yaml a seguir instala o Prometheus versão 2.44.0. Além disso, cria um PersistentVolume para armazenar as métricas, e um ConfigMap com o endereço dos Kafka Brokers (scrape_configs).

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: prometheus-vol
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  storageClassName: px-sc-general
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-server-conf
  labels:
    name: prometheus-server-conf
data:
  prometheus.yml: |-
    global:
      scrape_interval: 60s
      evaluation_interval: 60s
      scrape_timeout: 60s
    rule_files:
      - /etc/prometheus/prometheus.rules
    scrape_configs:
      - job_name: 'kafka'
        static_configs:
          - targets:
              - "kafka-0.dev-kafka-headless.kafka.svc.cluster.local:9404"
              - "kafka-1.dev-kafka-headless.kafka.svc.cluster.local:9404"
              - "kafka-2.dev-kafka-headless.kafka.svc.cluster.local:9404"
            labels:
              env: "dev"
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus-service
  annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/port:   '9090'
  
spec:
  selector: 
    app: prometheus-server
  type: NodePort  
  ports:
    - port: 8080
      targetPort: 9090 
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-deployment
  labels:
    app: prometheus-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-server
  template:
    metadata:
      labels:
        app: prometheus-server
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus
          args:
            - "--config.file=/etc/prometheus/prometheus.yml"
            - "--storage.tsdb.path=/prometheus/"
            - "--storage.tsdb.retention.time=10d"
            - "--storage.tsdb.retention.size=50GiB"
          ports:
            - containerPort: 9090
          resources:
            limits:
              cpu: "1"
              memory: 1400Mi
            requests:
              cpu: 250m
              memory: 512Mi
          volumeMounts:
            - name: prometheus-config-volume
              mountPath: /etc/prometheus/
            - name: prometheus-storage-volume
              mountPath: /prometheus/
      securityContext:
        fsGroup: 65534
        runAsGroup: 65534
        runAsNonRoot: true
        runAsUser: 65534
      volumes:
        - name: prometheus-config-volume
          configMap:
            defaultMode: 420
            name: prometheus-server-conf
        - name: prometheus-storage-volume
          persistentVolumeClaim:
            claimName: prometheus-vol

Grafana

O arquivo yaml a seguir instala o Grafana 9.5.3 e configure um ConfigMap que cria um datasource para o Prometheus instalado anteriormente.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources
data:
  prometheus.yaml: |-
    {
        "apiVersion": 1,
        "datasources": [
            {
               "access":"proxy",
                "editable": true,
                "name": "prometheus",
                "orgId": 1,
                "type": "prometheus",
                "url": "http://prometheus-service.kafka.svc:8080",
                "version": 1
            }
        ]
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      name: grafana
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:9.5.3
        ports:
        - name: grafana
          containerPort: 3000
        resources:
          limits:
            memory: "1Gi"
            cpu: "500m"
          requests: 
            memory: 500M
            cpu: "500m"
        volumeMounts:
          - mountPath: /var/lib/grafana
            name: grafana-storage
          - mountPath: /etc/grafana/provisioning/datasources
            name: grafana-datasources
            readOnly: false
      volumes:
        - name: grafana-storage
          emptyDir: {}
        - name: grafana-datasources
          configMap:
              defaultMode: 420
              name: grafana-datasources
---
apiVersion: v1
kind: Service
metadata:
  name: grafana
  annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/port:   '3000'
spec:
  selector: 
    app: grafana
   type: NodePort 
   ports:
     - port: 3000
       targetPort: 3000