Collector Type: Agent
Category: Application Monitors
Application Name: Kafka
Global Template Name: LINUX Kafka Monitor Template
Introduction
Apache Kafka is an open-source stream-processing software platform developed by the Apache Software Foundation, written in Scala and Java, and monitored through JMX. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.
Prerequisite: Refer to JMX Monitoring to enable JMX in the java application.
Parameters
| Name | Default Value |
|---|---|
| IPAddress IP Address on which the JMX listens. | 127.0.0.1 |
| Port JMX Listener port. | 7199 |
| Username Username, if enabled in JMX. | NA |
| Password Password, if enabled in JMX. | NA |
| Java Bin Path Path of the java binary. | java |
Note: All field attributes are mandatory, use default values wherever applicable.
Collected Metrics
| Metric Name | Display Name |
|---|---|
| kafka.net.bytes_out Bytes out rate. | Kafka Bytes Out |
| kafka.net.bytes_in Bytes in rate. | Kafka Bytes In |
| kafka.net.messages_in Messages in rate. | Kafka Messages In |
| kafka.net.bytes_rejected Bytes Rejected. | Kafka Bytes Rejected |
| kafka.request.produce.failed Failed producer requests rate. | Kafka Failed Produce Requests |
| kafka.request.fetch.failed Failed fetch requests rate. | HDFS Datanode Cache Used |
| kafka.replication.isr_shrinks ISR shrink rate. | Kafka ISR Shrinks |
| kafka.replication.isr_expands ISR expansion rate. | Kafka ISR Expands |
| kafka.replication.under_replicated_partitions Number of under replicated partitions (|ISR| < |all replicas|). | Kafka Under Replicated Partitions |
| kafka.replication.partitions Partition counts. | Kafka Partition Count |
| kafka.replication.leader_count Leader replica counts. | Kafka Leader Count |
| kafka.fetch.requests_waiting Requests waiting in the fetch purgatory. | Kafka Fetch Purgatory Size |
| kafka.producer.requests_waiting Requests waiting in the producer purgatory. | Kafka Producer Purgatory Size |
| kafka.fetch.requests_delayed Requests delayed in the fetch purgatory. | Kafka Fetch Delayed Requests |
| kafka.producer.requests_delayed Requests delayed in the producer purgatory. | Kafka Producer Delayed Requests |
| kafka.replication.max_lag Max lag, in messages, between follower and leader replicas. | Kafka Replication Max Lag |
| kafka.metrics.produce_remote_time Time the request waits for the follower. | Kafka Producer Remote Time |
| kafka.metrics.stop_replica_total_time Request total time. | Kafka Stop Replica Total Time |
| kafka.metrics.offsets.resp_queue_time Time the request waiting in the response queue. | Kafka Offsets Response Queue Time |
| kafka.metrics.offsets.req_queue_time Time the request waiting in the request queue. | Kafka Offsets Request Queue Time |
| kafka.metrics.offset_commit.remote_time Time the request waits for the follower. | Kafka Offset Commit Remote Time |
| kafka.metrics.offset_commit_resp_queue_time Time the request waiting in the response queue. | Kafka Offset Commit Response Queue Time |
| kafka.metrics.leader_isr.local_time Time the request being processed at the leader. | Kafka Leader And Isr Local Time |
| kafka.metrics.leader_isr.remote_time Time the request waits for the follower. | Kafka Leader And Isr Remote Time |
| kafka.request.fetch.time_avg Request total time. | Kafka Fetch Total Time |
| kafka.request.fetch.time_99percentile Time to produce requests for the 99th percentile. | Kafka Fetch Total Time 99percentile |
| kafka.metrics.fetch.requests Request rate. | Kafka Fetch Requests |
| kafka.metrics.fetch_follower.local_time Time the request being processed at the leader. | Kafka Fetch Follower Local Time |
| kafka.metrics.fetch_follower.resp_queue_time Time the request waiting in the response queue. | Kafka Fetch Follower Response Queue Time |
| kafka.metrics.fetch_consumer.req_queue_time Time the request waiting in the request queue. | Kafka Fetch Consumer Request Queue Time |
| kafka.metrics.fetch_consumer.resp_send_time Time to send the response. | Kafka Fetch Consumer Response Send Time |
| kafka.metrics.fetch_consumer.total_time Request total time. | Kafka Fetch Consumer Total Time |
| kafka.metrics.controlled_shutdown.req_queue_time Time the request waiting in the request queue. | Kafka Controlled Shutdown Request Queue Time |
| kafka.metrics.controlled_shutdown.resp_send_time Time to send the response. | Kafka Controlled Shutdown Response Send Time |
| kafka.metrics.controlled_shutdown.total_time Request total time. | Kafka Controlled Shutdown Total Time |
| kafka.metrics.controlled_shutdown.requests Request rate. | Kafka Controlled Shutdown Requests |
| kafka.metrics.metadata.req_queue_time Time the request waiting in the request queue. | Kafka Metadata Request Queue Time |
| kafka.request.metadata.time.avg Request total time. | Kafka Metadata Total Time |
| kafka.request.metadata.time.99percentile Time to metadata requests for the 99th percentile. | Kafka Metadata 99percentile Time |
| kafka.request.update_metadata.time.avg Request total time. | Kafka Update Metadata Total Time |
| kafka.request.update_metadata.time.99percentile Time for updating metadata requests for the 99th percentile. | Kafka Update Metadata 99percentile Time |
| kafka.metrics.update_metadata.remote_time Time the request waits for the follower. | Kafka Update Metadata Remote Time |
| kafka.metrics.update_metadata.requests Request rate. | Kafka Update Metadata Requests |
| kafka.channel.queue_size_request | Kafka Request Queue Size |
| kafka.channel.queue_size_response | Kafka Response Queue Size |
| kafka.replication.leader_elections Leader election rate. | Kafka Leader Election Rate And Time |
| kafka.replication.unclean_leader_elections Unclean leader election rate. | Kafka Unclean Leader Elections |
| kafka.controller.active_controller_count Is the controller active on the broker. | Kafka Active Controller Count |
| kafka.log.flush_rate Log flush rate and time. | Kafka LogFlush Rate And Time |
| kafka.jvm.uptime Uptime of the server. | Kafka Uptime |
| kafka.jvm.gc_collection_count Number of garbage objects collected. | Kafka JVM GC collection_count |
| kafka.jvm.gc_collection_time Time taken for collection of the garbage objects. | Kafka JVM GC collection_time |
| kafka.jvm.mem_heap_committed Heap memory committed (in MB) for the server. | Kafka JVM Mem heap_committed |
| kafka.jvm.mem_heap_used Heap memory usage (in MB) of the server. | Kafka JVM Mem heap_used |
| kafka.jvm.mem_non_heap_committed Non-heap memory committed (in MB) for the server. | Kafka JVM Mem non_heap_committed |
| kafka.jvm.mem_non_heap_used Non-heap memory usage (in MB) of the server. | Kafka JVM Mem non_heap_used |
| kafka.jvm.threads Number of threads. | Kafka JVM Threads |
| kafka.jvm.open_fds Number of Open file descriptors of the server. | Kafka JVM OpenFDs |
| kafka.request.produce.time.avg Average time for a produce request. | Kafka Produce Request Time |
| kafka.request.produce.time.99percentile Time to produce requests for the 99th percentile. | Kafka Produce Request Time 99percentile |
| kafka.request.offsets.time.avg Average time for an offset request. | Kafka Offsets Request Time |
| kafka.request.offsets.time.99percentile Time for offset requests for the 99th percentile. | Kafka Offsets Request Time 99 percentile |
| kafka.request.handler_avg_idle_pct Average fraction of time the request handler threads are idle. | Kafka Request Handler Threads Idle Time |