Collector Type: Agent

Category: Application Monitors

Application Name: Kafka

Global Template Name: LINUX Kafka Monitor Template

Introduction

Apache Kafka is an open-source stream-processing software platform developed by the Apache Software Foundation, written in Scala and Java, and monitored through JMX. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.

Prerequisite: Refer to JMX Monitoring to enable JMX in the java application.

Parameters

NameDefault Value
IPAddress

IP Address on which the JMX listens.
127.0.0.1
Port

JMX Listener port.
7199
Username

Username, if enabled in JMX.
NA
Password

Password, if enabled in JMX.
NA
Java Bin Path

Path of the java binary.
java

Note: All field attributes are mandatory, use default values wherever applicable.

Collected Metrics

Metric NameDisplay Name
kafka.net.bytes_out

Bytes out rate.
Kafka Bytes Out
kafka.net.bytes_in

Bytes in rate.
Kafka Bytes In
kafka.net.messages_in

Messages in rate.
Kafka Messages In
kafka.net.bytes_rejected

Bytes Rejected.
Kafka Bytes Rejected
kafka.request.produce.failed

Failed producer requests rate.
Kafka Failed Produce Requests
kafka.request.fetch.failed

Failed fetch requests rate.
HDFS Datanode Cache Used
kafka.replication.isr_shrinks

ISR shrink rate.
Kafka ISR Shrinks
kafka.replication.isr_expands

ISR expansion rate.
Kafka ISR Expands
kafka.replication.under_replicated_partitions

Number of under replicated partitions (|ISR| < |all replicas|).
Kafka Under Replicated Partitions
kafka.replication.partitions

Partition counts.
Kafka Partition Count
kafka.replication.leader_count

Leader replica counts.
Kafka Leader Count
kafka.fetch.requests_waiting

Requests waiting in the fetch purgatory.
Kafka Fetch Purgatory Size
kafka.producer.requests_waiting

Requests waiting in the producer purgatory.
Kafka Producer Purgatory Size
kafka.fetch.requests_delayed

Requests delayed in the fetch purgatory.
Kafka Fetch Delayed Requests
kafka.producer.requests_delayed

Requests delayed in the producer purgatory.
Kafka Producer Delayed Requests
kafka.replication.max_lag

Max lag, in messages, between follower and leader replicas.
Kafka Replication Max Lag
kafka.metrics.produce_remote_time

Time the request waits for the follower.
Kafka Producer Remote Time
kafka.metrics.stop_replica_total_time

Request total time.
Kafka Stop Replica Total Time
kafka.metrics.offsets.resp_queue_time

Time the request waiting in the response queue.
Kafka Offsets Response Queue Time
kafka.metrics.offsets.req_queue_time

Time the request waiting in the request queue.
Kafka Offsets Request Queue Time
kafka.metrics.offset_commit.remote_time

Time the request waits for the follower.
Kafka Offset Commit Remote Time
kafka.metrics.offset_commit_resp_queue_time

Time the request waiting in the response queue.
Kafka Offset Commit Response Queue Time
kafka.metrics.leader_isr.local_time

Time the request being processed at the leader.
Kafka Leader And Isr Local Time
kafka.metrics.leader_isr.remote_time

Time the request waits for the follower.
Kafka Leader And Isr Remote Time
kafka.request.fetch.time_avg

Request total time.
Kafka Fetch Total Time
kafka.request.fetch.time_99percentile

Time to produce requests for the 99th percentile.
Kafka Fetch Total Time 99percentile
kafka.metrics.fetch.requests

Request rate.
Kafka Fetch Requests
kafka.metrics.fetch_follower.local_time

Time the request being processed at the leader.
Kafka Fetch Follower Local Time
kafka.metrics.fetch_follower.resp_queue_time

Time the request waiting in the response queue.
Kafka Fetch Follower Response Queue Time
kafka.metrics.fetch_consumer.req_queue_time

Time the request waiting in the request queue.
Kafka Fetch Consumer Request Queue Time
kafka.metrics.fetch_consumer.resp_send_time

Time to send the response.
Kafka Fetch Consumer Response Send Time
kafka.metrics.fetch_consumer.total_time

Request total time.
Kafka Fetch Consumer Total Time
kafka.metrics.controlled_shutdown.req_queue_time

Time the request waiting in the request queue.
Kafka Controlled Shutdown Request Queue Time
kafka.metrics.controlled_shutdown.resp_send_time

Time to send the response.
Kafka Controlled Shutdown Response Send Time
kafka.metrics.controlled_shutdown.total_time

Request total time.
Kafka Controlled Shutdown Total Time
kafka.metrics.controlled_shutdown.requests

Request rate.
Kafka Controlled Shutdown Requests
kafka.metrics.metadata.req_queue_time

Time the request waiting in the request queue.
Kafka Metadata Request Queue Time
kafka.request.metadata.time.avg

Request total time.
Kafka Metadata Total Time
kafka.request.metadata.time.99percentile

Time to metadata requests for the 99th percentile.
Kafka Metadata 99percentile Time
kafka.request.update_metadata.time.avg

Request total time.
Kafka Update Metadata Total Time
kafka.request.update_metadata.time.99percentile

Time for updating metadata requests for the 99th percentile.
Kafka Update Metadata 99percentile Time
kafka.metrics.update_metadata.remote_time

Time the request waits for the follower.
Kafka Update Metadata Remote Time
kafka.metrics.update_metadata.requests

Request rate.
Kafka Update Metadata Requests
kafka.channel.queue_size_requestKafka Request Queue Size
kafka.channel.queue_size_responseKafka Response Queue Size
kafka.replication.leader_elections

Leader election rate.
Kafka Leader Election Rate And Time
kafka.replication.unclean_leader_elections

Unclean leader election rate.
Kafka Unclean Leader Elections
kafka.controller.active_controller_count

Is the controller active on the broker.
Kafka Active Controller Count
kafka.log.flush_rate

Log flush rate and time.
Kafka LogFlush Rate And Time
kafka.jvm.uptime

Uptime of the server.
Kafka Uptime
kafka.jvm.gc_collection_count

Number of garbage objects collected.
Kafka JVM GC collection_count
kafka.jvm.gc_collection_time

Time taken for collection of the garbage objects.
Kafka JVM GC collection_time
kafka.jvm.mem_heap_committed

Heap memory committed (in MB) for the server.
Kafka JVM Mem heap_committed
kafka.jvm.mem_heap_used

Heap memory usage (in MB) of the server.
Kafka JVM Mem heap_used
kafka.jvm.mem_non_heap_committed

Non-heap memory committed (in MB) for the server.
Kafka JVM Mem non_heap_committed
kafka.jvm.mem_non_heap_used

Non-heap memory usage (in MB) of the server.
Kafka JVM Mem non_heap_used
kafka.jvm.threads

Number of threads.
Kafka JVM Threads
kafka.jvm.open_fds

Number of Open file descriptors of the server.
Kafka JVM OpenFDs
kafka.request.produce.time.avg

Average time for a produce request.
Kafka Produce Request Time
kafka.request.produce.time.99percentile

Time to produce requests for the 99th percentile.
Kafka Produce Request Time 99percentile
kafka.request.offsets.time.avg

Average time for an offset request.
Kafka Offsets Request Time
kafka.request.offsets.time.99percentile

Time for offset requests for the 99th percentile.
Kafka Offsets Request Time 99 percentile
kafka.request.handler_avg_idle_pct

Average fraction of time the request handler threads are idle.
Kafka Request Handler Threads Idle Time