Collector Type: Agent
Category: Application Monitors
Application Name: Kafka
Global Template Name: LINUX Kafka Monitor Template
Introduction
Apache Kafka is an open-source stream-processing software platform developed by the Apache Software Foundation, written in Scala and Java, and monitored through JMX. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.
Prerequisite: Refer to JMX Monitoring to enable JMX in the java application.
Parameters
| Name | Default Value | Description |
|---|---|---|
| IPAddress | 127.0.0.1 | IP Address on which the JMX listens. |
| Port | 7199 | JMX Listener port. |
| Username | NA | Username, if enabled in JMX. |
| Password | NA | Password, if enabled in JMX. |
| Java Bin Path | java | Path of the java binary. |
Note: All field attributes are mandatory, use default values wherever applicable.
Collected Metrics
| Metric Name | Display Name | Description |
|---|---|---|
| kafka.net.bytes_out | Kafka Bytes Out | Bytes out rate. |
| kafka.net.bytes_in | Kafka Bytes In | Bytes in rate. |
| kafka.net.messages_in | Kafka Messages In | Messages in rate. |
| kafka.net.bytes_rejected | Kafka Bytes Rejected | Bytes Rejected. |
| kafka.request.produce.failed | Kafka Failed Produce Requests | Failed producer requests rate. |
| kafka.request.fetch.failed | HDFS Datanode Cache Used | Failed fetch requests rate. |
| kafka.replication.isr_shrinks | Kafka ISR Shrinks | ISR shrink rate. |
| kafka.replication.isr_expands | Kafka ISR Expands | ISR expansion rate. |
| kafka.replication.under_replicated_partitions | Kafka Under Replicated Partitions | Number of under replicated partitions (|ISR| < |all replicas|). |
| kafka.replication.partitions | Kafka Partition Count | Partition counts. |
| kafka.replication.leader_count | Kafka Leader Count | Leader replica counts. |
| kafka.fetch.requests_waiting | Kafka Fetch Purgatory Size | Requests waiting in the fetch purgatory. |
| kafka.producer.requests_waiting | Kafka Producer Purgatory Size | Requests waiting in the producer purgatory. |
| kafka.fetch.requests_delayed | Kafka Fetch Delayed Requests | Requests delayed in the fetch purgatory. |
| kafka.producer.requests_delayed | Kafka Producer Delayed Requests | Requests delayed in the producer purgatory. |
| kafka.replication.max_lag | Kafka Replication Max Lag | Max lag, in messages, between follower and leader replicas. |
| kafka.metrics.produce_remote_time | Kafka Producer Remote Time | Time the request waits for the follower. |
| kafka.metrics.stop_replica_total_time | Kafka Stop Replica Total Time | Request total time. |
| kafka.metrics.offsets.resp_queue_time | Kafka Offsets Response Queue Time | Time the request waiting in the response queue. |
| kafka.metrics.offsets.req_queue_time | Kafka Offsets Request Queue Time | Time the request waiting in the request queue. |
| kafka.metrics.offset_commit.remote_time | Kafka Offset Commit Remote Time | Time the request waits for the follower. |
| kafka.metrics.offset_commit_resp_queue_time | Kafka Offset Commit Response Queue Time | Time the request waiting in the response queue. |
| kafka.metrics.leader_isr.local_time | Kafka Leader And Isr Local Time | Time the request being processed at the leader. |
| kafka.metrics.leader_isr.remote_time | Kafka Leader And Isr Remote Time | Time the request waits for the follower. |
| kafka.request.fetch.time_avg | Kafka Fetch Total Time | Request total time. |
| kafka.request.fetch.time_99percentile | Kafka Fetch Total Time 99percentile | Time to produce requests for the 99th percentile. |
| kafka.metrics.fetch.requests | Kafka Fetch Requests | Request rate. |
| kafka.metrics.fetch_follower.local_time | Kafka Fetch Follower Local Time | Time the request being processed at the leader. |
| kafka.metrics.fetch_follower.resp_queue_time | Kafka Fetch Follower Response Queue Time | Time the request waiting in the response queue. |
| kafka.metrics.fetch_consumer.req_queue_time | Kafka Fetch Consumer Request Queue Time | Time the request waiting in the request queue. |
| kafka.metrics.fetch_consumer.resp_send_time | Kafka Fetch Consumer Response Send Time | Time to send the response. |
| kafka.metrics.fetch_consumer.total_time | Kafka Fetch Consumer Total Time | Request total time. |
| kafka.metrics.controlled_shutdown.req_queue_time | Kafka Controlled Shutdown Request Queue Time | Time the request waiting in the request queue. |
| kafka.metrics.controlled_shutdown.resp_send_time | Kafka Controlled Shutdown Response Send Time | Time to send the response. |
| kafka.metrics.controlled_shutdown.total_time | Kafka Controlled Shutdown Total Time | Request total time. |
| kafka.metrics.controlled_shutdown.requests | Kafka Controlled Shutdown Requests | Request rate. |
| kafka.metrics.metadata.req_queue_time | Kafka Metadata Request Queue Time | Time the request waiting in the request queue. |
| kafka.request.metadata.time.avg | Kafka Metadata Total Time | Request total time. |
| kafka.request.metadata.time.99percentile | Kafka Metadata 99percentile Time | Time to metadata requests for the 99th percentile. |
| kafka.request.update_metadata.time.avg | Kafka Update Metadata Total Time | Request total time. |
| kafka.request.update_metadata.time.99percentile | Kafka Update Metadata 99percentile Time | Time for updating metadata requests for the 99th percentile. |
| kafka.metrics.update_metadata.remote_time | Kafka Update Metadata Remote Time | Time the request waits for the follower. |
| kafka.metrics.update_metadata.requests | Kafka Update Metadata Requests | Request rate. |
| kafka.channel.queue_size_request | Kafka Request Queue Size | |
| kafka.channel.queue_size_response | Kafka Response Queue Size | |
| kafka.replication.leader_elections | Kafka Leader Election Rate And Time | Leader election rate. |
| kafka.replication.unclean_leader_elections | Kafka Unclean Leader Elections | Unclean leader election rate. |
| kafka.controller.active_controller_count | Kafka Active Controller Count | Is the controller active on the broker. |
| kafka.log.flush_rate | Kafka LogFlush Rate And Time | Log flush rate and time. |
| kafka.jvm.uptime | Kafka Uptime | Uptime of the server. |
| kafka.jvm.gc_collection_count | Kafka JVM GC collection_count | Number of garbage objects collected. |
| kafka.jvm.gc_collection_time | Kafka JVM GC collection_time | Time taken for collection of the garbage objects. |
| kafka.jvm.mem_heap_committed | Kafka JVM Mem heap_committed | Heap memory committed (in MB) for the server. |
| kafka.jvm.mem_heap_used | Kafka JVM Mem heap_used | Heap memory usage (in MB) of the server. |
| kafka.jvm.mem_non_heap_committed | Kafka JVM Mem non_heap_committed | Non-heap memory committed (in MB) for the server. |
| kafka.jvm.mem_non_heap_used | Kafka JVM Mem non_heap_used | Non-heap memory usage (in MB) of the server. |
| kafka.jvm.threads | Kafka JVM Threads | Number of threads. |
| kafka.jvm.open_fds | Kafka JVM OpenFDs | Number of Open file descriptors of the server. |
| kafka.request.produce.time.avg | Kafka Produce Request Time | Average time for a produce request. |
| kafka.request.produce.time.99percentile | Kafka Produce Request Time 99percentile | Time to produce requests for the 99th percentile. |
| kafka.request.offsets.time.avg | Kafka Offsets Request Time | Average time for an offset request. |
| kafka.request.offsets.time.99percentile | Kafka Offsets Request Time 99 percentile | Time for offset requests for the 99th percentile. |
| kafka.request.handler_avg_idle_pct | Kafka Request Handler Threads Idle Time | Average fraction of time the request handler threads are idle. |