Apache Kafka is an open-source stream-processing software platform developed by the Apache Software Foundation, written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.

Prerequisite

  1. Install Kafka and configure the JMX Port using the environment variable.
export JMX_PORT=5199 && ./bin/kafka-server-start.sh -daemon config/server.properties
  1. For Virtual Machines, install the Linux Agent.

Configuring the credentials

Configure the credentials in the directory /opt/opsramp/agent/conf/app.d/creds.yaml

kafka:
- name: kafka
  user: <username>
  pwd: <Password>
  encoding-type: plain
  labels:
    key1: val1
    key2: val2

Configuring the application

Virtual machine

Configure the application in the directory /opt/opsramp/agent/conf/app/discovery/auto-detection.yaml

- name: kafka
  instance-checks:
    service-check:
      - kafka
    process-check:
      - kafka
    port-check:
      - 9092
  mon-type: "jmx"
  misc:
    jmx-port: "5199"

Docker environment

Configure the application in the directory /opt/opsramp/agent/conf/app/discovery/auto-container-detection.yaml

- name: kafka
  container-checks:
    image-check:
      - kafka
    port-check:
      - 9092
  mon-type: "jmx"
  misc:
    jmx-port: "5199"

Kubernetes environment

Configure the application in config.yaml

- name: kafka
  container-checks:
    image-check:
      - kafka
    port-check:
      - 9092
  mon-type: "jmx"
  misc:
    jmx-port: "5199"

Validate

Go to Resources under the Infrastructure tab to check if your resources are onboarded and the metrics are collected.

Metrics

OpsRamp MetricMetric Display NameUnit
kafka_channel_request_queue_size

Number of queued requests
Request Queue SizeRequests
kafka_channel_response_queue_size

Number of queued responses
Response Queue SizeResponses
kafka_controller_active_controller_count

Number of active controller on broker
Active Controller CountControllers
kafka_fetch_requests_delayed

Requests delayed in the fetch purgatory
Fetch Delayed RequestsRequests
kafka_fetch_requests_purgatory_delayed

Requests delayed in the fetch purgatory
Fetch Delayed RequestsRequests
kafka_fetch_requests_purgatory_size

Requests waiting in the fetch purgatory
Fetch Purgatory SizeRequests
kafka_fetch_requests_size

Requests waiting in the fetch purgatory
Fetch Purgatory SizeRequests
kafka_jvm_gc_collection_count

Number of garbage objects collected
Kafka JVM GC collection_countObjects collected
kafka_jvm_gc_collection_time

Time taken for collection of the garbage objects
Kafka JVM GC collection_timeSeconds
kafka_jvm_mem_heap_committed

Heap memory committed for the server
Kafka JVM Mem heap_committedmegabytes
kafka_jvm_mem_heap_used

Heap memory usage of the server
Kafka JVM Mem heap_usedmegabytes
kafka_jvm_mem_non_heap_committed

Non-heap memory committed for the server
Kafka JVM Mem non_heap_committedmegabytes
kafka_jvm_mem_non_heap_used

Non-heap memory usage of the server
Kafka JVM Mem non_heap_usedmegabytes
kafka_jvm_open_fds

Number of Open file descriptors of the server
Kafka JVM OpenFDsOpen FDs
kafka_jvm_threads

Number of threads
Kafka JVM ThreadsThreads
kafka_jvm_uptime

Uptime of the server
UptimeMinutes
kafka_log_flush_rate

Log flush rate and time
LogFlush Rate And TimeLog flushes
kafka_metrics_controlled_shutdown_req_queue_time

Refers to the time for which the request is waiting in queue
Controlled Shutdown Request Queue TimeRequest time,ms
kafka_metrics_controlled_shutdown_requests

Request rate
Controlled Shutdown RequestsRequests / sec
kafka_metrics_controlled_shutdown_resp_send_time

Time to send the response
Controlled Shutdown Response Send TimeRequest time,ms
kafka_metrics_controlled_shutdown_total_time

Request total time
Controlled Shutdown Total TimeTotal time,ms
kafka_metrics_fetch_consumer_req_queue_time

Refers to the time for which the request is waiting in the request queue
Fetch Consumer Request Queue Timemilliseconds
kafka_metrics_fetch_consumer_resp_send_time

Time to send the response
Fetch Consumer Response Send Timemilliseconds
kafka_metrics_fetch_consumer_total_time

Request total time
Fetch Consumer Total Timemilliseconds
kafka_metrics_fetch_follower_local_time

Time taken to process the request at the leader
Fetch Follower Local Timemilliseconds
kafka_metrics_fetch_follower_resp_queue_time

Time taken by the request to wait in the response queue
Fetch Follower Response Queue Timemilliseconds
kafka_metrics_fetch_requests

Request rate
Fetch RequestsRequests / second
kafka_metrics_leader_isr_local_time

Time taken to process the request at the leader
Leader And Isr Local Timemilliseconds
kafka_metrics_leader_isr_remote_time

Time taken by the request to wait for the follower
Leader And Isr Remote Timemilliseconds
kafka_metrics_metadata_req_queue_time

Time taken by the request to wait in the request queue
Metadata Request Queue Timemilliseconds
kafka_metrics_offset_commit_remote_time

Time taken by the request to wait for the follower
Offset Commit Remote Timemilliseconds
kafka_metrics_offset_commit_resp_queue_time

Time taken by the request to wait in the response queue
Offset Commit Response Queue Timemilliseconds
kafka_metrics_offsets_req_queue_time

Time taken by the request to wait in the request queue
Offsets Request Queue Timemilliseconds
kafka_metrics_offsets_resp_queue_time

Time taken by the request to wait in the response queue
Offsets Response Queue Timemilliseconds
kafka_metrics_produce_remote_time

Time taken by the request to wait for the follower
Producer Remote Timemilliseconds
kafka_metrics_stop_replica_total_time

Request total time
Stop Replica Total Timemilliseconds
kafka_metrics_update_metadata_remote_time

Time taken by the request to wait for the follower
Update Metadata Remote Timemilliseconds
kafka_metrics_update_metadata_requests

Request rate
Update Metadata RequestsRequests / second
kafka_net_bytes_in

Bytes in rate
Bytes Inbytes / second
kafka_net_bytes_out

Bytes out rate
Bytes Outbytes / second
kafka_net_bytes_rejected

Bytes Rejected
Bytes Rejectedbytes / second
kafka_net_messages_in

Messages in rate
Messages InMessages / second
kafka_producer_requests_delayed

Requests delayed in the producer purgatory
Producer Delayed RequestsRequests
kafka_producer_requests_purgatory_delayed

Requests delayed in the producer purgatory
Producer Delayed RequestsRequests
kafka_producer_requests_purgatory_size

Requests waiting in the producer purgatory
Producer Purgatory SizeRequests
kafka_producer_requests_size

Requests waiting in the producer purgatory
Producer Purgatory SizeRequests
kafka_replication_isr_expands

ISR expansion rate
ISR ExpandsExapands / second
kafka_replication_isr_shrinks

ISR shrink rate
ISR ShrinksShrinks / second
kafka_replication_leader_count

Leader replica counts
Leader CountTotal Leaders
kafka_replication_leader_elections

Leader election rate
Leader Election Rate And TimeLeader elections
kafka_replication_max_lag

Maximum lag in messages between follower and leader replicas
Replication Max LagMessages
kafka_replication_partitions

Partition counts
Partition CountTotal Partitions
kafka_replication_unclean_leader_elections

Unclean leader election rate
Unclean Leader ElectionsElections / second
kafka_replication_under_replicated_partitions

Number of under replicated partitions(|ISR| < |all replicas|)
Under Replicated PartitionsUnder Replicated Partitions
kafka_request_fetch_failed

Failed fetch requests rate
Failed Fetch RequestsRequests / second
kafka_request_fetch_time_99percentile

Time for produce requests for 99th percentile
Fetch Total Time 99percentileRequests / second
kafka_request_fetch_time_avg

Request total time
Fetch Total Timemilliseconds
kafka_request_handler_avg_idle_pct

Average fraction of time when the request handler threads are idle
Request Handler Threads Idle TimeFraction
kafka_request_metadata_time_99percentile

Time for metadata requests for 99th percentile
Metadata 99percentile Timemilliseconds
kafka_request_metadata_time_avg

Request total time
Metadata Total Timemilliseconds
kafka_request_offsets_time_99percentile

Time for offset requests for 99th percentile
Offsets Request Time 99percentilemilliseconds
kafka_request_offsets_time_avg

Average time for an offset request
Offsets Request Timemilliseconds
kafka_request_produce_failed

Failed producer requests rate
Failed Produce RequestsRequests / second
kafka_request_produce_time_99percentile

Time for produce requests for 99th percentile
Produce Request Time 99percentileRequest / second
kafka_request_produce_time_avg

Average time for a produce request
Produce Request TimeRequest / second
kafka_request_update_metadata_time_99percentile

Time for update metadata requests for 99th percentile
Update Metadata 99percentile Timemilliseconds
kafka_request_update_metadata_time_avg

Request total time
Update Metadata Total Timemilliseconds