Kafka

Introduction

Apache Kafka is a distributed event streaming platform designed for high-throughput, fault-tolerant messaging. It is widely used for building real-time data pipelines and event-driven applications. Monitoring Kafka is essential to ensure optimal performance, detect potential bottlenecks, and maintain system reliability.

To monitor Kafka efficiently, we can use the OpenTelemetry Kafka Receiver, which collects key Kafka performance metrics. These include broker health, topic throughput, partition activity, consumer lag, and more, helping to gain deep visibility into Kafka’s performance and health.

Kubernetes 2.0 ConfigMap

Users should update or append the existing ConfigMap named opsramp-workload-metric-user-config by adding application-specific fields. These fields may include authentication credentials, collection frequency, and other relevant configurations. To run OpenTelemetry Redis Receiver with Kubernetes, you need to create a ConfigMap and deploy it.

apiVersion: v1
kind: ConfigMap
metadata:
  name: opsramp-workload-metric-user-config
  namespace: opsramp-agent
data:
  workloads: |
    kafka:
      - name: kafka
        collectionInterval: 60s
        protocolVersion: "2.0.0"
        scrapers:
          - brokers
          - topics
          - consumers
        auth: none
        port: 9092
        targetPodSelector:
          matchLabels:
            - key: app.kubernetes.io/instance
              operator: ==
              value:
                - my-kafka-plain
            - key: app.kubernetes.io/name
              operator: ==
              value:
                - kafka

      - name: kafka-sasl-plaintext
        collectionInterval: 60s
        protocolVersion: "2.0.0"
        scrapers:
          - brokers
          - topics
          - consumers
        auth: sasl_plaintext
        userName:
          value: "user1"
        password:
          value: "password1"
        port: 9092
        targetPodSelector:
          matchLabels:
            - key: app.kubernetes.io/instance
              operator: ==
              value:
                - my-kafka-plain
            - key: app.kubernetes.io/name
              operator: ==
              value:
                - kafka

      - name: kafka-tls
        collectionInterval: 60s
        protocolVersion: "2.0.0"
        scrapers:
          - brokers
          - topics
          - consumers
        auth: tls
        certFile:
          source: secret
          value: default/kafka-client-tls-secret/client.pem
        keyFile:
          source: secret
          value: default/kafka-client-tls-secret/client-key.pem
        insecureSkipVerify: true
        port: 9093
        targetPodSelector:
          matchLabels:
            - key: app.kubernetes.io/instance
              operator: ==
              value:
                - my-kafka-plain
            - key: app.kubernetes.io/name
              operator: ==
              value:
                - kafka

Supported Metrics

Supported metrics for this workload as provided by the Kubernetes 2.0 Agent.