Collector Type: Agent

Category: Application Monitors

Application Name: Kafka

Global Template Name: LINUX Kafka Monitor Template

Introduction

Apache Kafka is an open-source stream-processing software platform developed by the Apache Software Foundation, written in Scala and Java, and monitored through JMX. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.

Prerequisite: Refer to JMX Monitoring to enable JMX in the java application.

Parameters

NameDefault ValueDescription
IPAddress127.0.0.1IP Address on which the JMX listens.
Port7199JMX Listener port.
UsernameNAUsername, if enabled in JMX.
PasswordNAPassword, if enabled in JMX.
Java Bin PathjavaPath of the java binary.

Note: All field attributes are mandatory, use default values wherever applicable.

Collected Metrics

Metric NameDisplay NameDescription
kafka.net.bytes_outKafka Bytes OutBytes out rate.
kafka.net.bytes_inKafka Bytes InBytes in rate.
kafka.net.messages_inKafka Messages InMessages in rate.
kafka.net.bytes_rejectedKafka Bytes RejectedBytes Rejected.
kafka.request.produce.failedKafka Failed Produce RequestsFailed producer requests rate.
kafka.request.fetch.failedHDFS Datanode Cache UsedFailed fetch requests rate.
kafka.replication.isr_shrinksKafka ISR ShrinksISR shrink rate.
kafka.replication.isr_expandsKafka ISR ExpandsISR expansion rate.
kafka.replication.under_replicated_partitionsKafka Under Replicated PartitionsNumber of under replicated partitions (|ISR| < |all replicas|).
kafka.replication.partitionsKafka Partition CountPartition counts.
kafka.replication.leader_countKafka Leader CountLeader replica counts.
kafka.fetch.requests_waitingKafka Fetch Purgatory SizeRequests waiting in the fetch purgatory.
kafka.producer.requests_waitingKafka Producer Purgatory SizeRequests waiting in the producer purgatory.
kafka.fetch.requests_delayedKafka Fetch Delayed RequestsRequests delayed in the fetch purgatory.
kafka.producer.requests_delayedKafka Producer Delayed RequestsRequests delayed in the producer purgatory.
kafka.replication.max_lagKafka Replication Max LagMax lag, in messages, between follower and leader replicas.
kafka.metrics.produce_remote_timeKafka Producer Remote TimeTime the request waits for the follower.
kafka.metrics.stop_replica_total_timeKafka Stop Replica Total TimeRequest total time.
kafka.metrics.offsets.resp_queue_timeKafka Offsets Response Queue TimeTime the request waiting in the response queue.
kafka.metrics.offsets.req_queue_timeKafka Offsets Request Queue TimeTime the request waiting in the request queue.
kafka.metrics.offset_commit.remote_timeKafka Offset Commit Remote TimeTime the request waits for the follower.
kafka.metrics.offset_commit_resp_queue_timeKafka Offset Commit Response Queue TimeTime the request waiting in the response queue.
kafka.metrics.leader_isr.local_timeKafka Leader And Isr Local TimeTime the request being processed at the leader.
kafka.metrics.leader_isr.remote_timeKafka Leader And Isr Remote TimeTime the request waits for the follower.
kafka.request.fetch.time_avgKafka Fetch Total TimeRequest total time.
kafka.request.fetch.time_99percentileKafka Fetch Total Time 99percentileTime to produce requests for the 99th percentile.
kafka.metrics.fetch.requestsKafka Fetch RequestsRequest rate.
kafka.metrics.fetch_follower.local_timeKafka Fetch Follower Local TimeTime the request being processed at the leader.
kafka.metrics.fetch_follower.resp_queue_timeKafka Fetch Follower Response Queue TimeTime the request waiting in the response queue.
kafka.metrics.fetch_consumer.req_queue_timeKafka Fetch Consumer Request Queue TimeTime the request waiting in the request queue.
kafka.metrics.fetch_consumer.resp_send_timeKafka Fetch Consumer Response Send TimeTime to send the response.
kafka.metrics.fetch_consumer.total_timeKafka Fetch Consumer Total TimeRequest total time.
kafka.metrics.controlled_shutdown.req_queue_timeKafka Controlled Shutdown Request Queue TimeTime the request waiting in the request queue.
kafka.metrics.controlled_shutdown.resp_send_timeKafka Controlled Shutdown Response Send TimeTime to send the response.
kafka.metrics.controlled_shutdown.total_timeKafka Controlled Shutdown Total TimeRequest total time.
kafka.metrics.controlled_shutdown.requestsKafka Controlled Shutdown RequestsRequest rate.
kafka.metrics.metadata.req_queue_timeKafka Metadata Request Queue TimeTime the request waiting in the request queue.
kafka.request.metadata.time.avgKafka Metadata Total TimeRequest total time.
kafka.request.metadata.time.99percentileKafka Metadata 99percentile TimeTime to metadata requests for the 99th percentile.
kafka.request.update_metadata.time.avgKafka Update Metadata Total TimeRequest total time.
kafka.request.update_metadata.time.99percentileKafka Update Metadata 99percentile TimeTime for updating metadata requests for the 99th percentile.
kafka.metrics.update_metadata.remote_timeKafka Update Metadata Remote TimeTime the request waits for the follower.
kafka.metrics.update_metadata.requestsKafka Update Metadata RequestsRequest rate.
kafka.channel.queue_size_requestKafka Request Queue Size
kafka.channel.queue_size_responseKafka Response Queue Size
kafka.replication.leader_electionsKafka Leader Election Rate And TimeLeader election rate.
kafka.replication.unclean_leader_electionsKafka Unclean Leader ElectionsUnclean leader election rate.
kafka.controller.active_controller_countKafka Active Controller CountIs the controller active on the broker.
kafka.log.flush_rateKafka LogFlush Rate And TimeLog flush rate and time.
kafka.jvm.uptimeKafka UptimeUptime of the server.
kafka.jvm.gc_collection_countKafka JVM GC collection_countNumber of garbage objects collected.
kafka.jvm.gc_collection_timeKafka JVM GC collection_timeTime taken for collection of the garbage objects.
kafka.jvm.mem_heap_committedKafka JVM Mem heap_committedHeap memory committed (in MB) for the server.
kafka.jvm.mem_heap_usedKafka JVM Mem heap_usedHeap memory usage (in MB) of the server.
kafka.jvm.mem_non_heap_committedKafka JVM Mem non_heap_committedNon-heap memory committed (in MB) for the server.
kafka.jvm.mem_non_heap_usedKafka JVM Mem non_heap_usedNon-heap memory usage (in MB) of the server.
kafka.jvm.threadsKafka JVM ThreadsNumber of threads.
kafka.jvm.open_fdsKafka JVM OpenFDsNumber of Open file descriptors of the server.
kafka.request.produce.time.avgKafka Produce Request TimeAverage time for a produce request.
kafka.request.produce.time.99percentileKafka Produce Request Time 99percentileTime to produce requests for the 99th percentile.
kafka.request.offsets.time.avgKafka Offsets Request TimeAverage time for an offset request.
kafka.request.offsets.time.99percentileKafka Offsets Request Time 99 percentileTime for offset requests for the 99th percentile.
kafka.request.handler_avg_idle_pctKafka Request Handler Threads Idle TimeAverage fraction of time the request handler threads are idle.