Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a fully managed service that enables you to build and run applications that use Apache Kafka to process streaming data.
Amazon MSK provides the control-plane operations, such as those for creating, updating, and deleting clusters.
- You can use Apache Kafka data-plane operations, such as those for producing and consuming data.
- It runs open-source versions of Apache Kafka.
This means existing applications, tooling, and plugins from partners and the Apache Kafka community are supported without requiring changes to application code.
Use the AWS public cloud integration to discover and collect metrics against the AWS service.
External reference
Monitoring an Amazon MSK Cluster
Setup
To set up the AWS integration and discover the AWS service, go to AWS Integration Discovery Profile and select MSK Cluster.
Event support
CloudTrail event support
- Not Supported
CloudWatch alarm support
- Supported
- Configurable in OpsRamp AWS Integration Discovery Profile.
Supported metrics
| OpsRamp Metric | AWS Metric | Metric Display Name | Unit | Aggregation Type | Description |
|---|---|---|---|---|---|
| aws_kafka_broker_node_BurstBalance | BurstBalance | Burst Balance | Percent | Average | Remaining balance of input/output burst credits for EBS volumes |
| aws_kafka_broker_node_BytesInPerSec | BytesInPerSec | Bytes In Per Sec | Bytes/Second | Average | Bytes per second received from clients |
| aws_kafka_broker_node_BytesOutPerSec | BytesOutPerSec | Bytes Out Per Sec | Bytes/Second | Average | Bytes per second sent to clients |
| aws_kafka_broker_node_ClientConnectionCount | ClientConnectionCount | Client Connection Count | Count | Average | Number of active authenticated client connections |
| aws_kafka_broker_node_ConnectionCount | ConnectionCount | Connection Count | Count | Average | Number of active authenticated, unauthenticated, and inter-broker connections |
| aws_kafka_broker_node_CPUCreditBalance | CPUCreditBalance | CPU Credit Balance | Count | Average | Earned CPU credits a broker has accrued since launch |
| aws_kafka_broker_node_CpuIdle | CpuIdle | Cpu Idle | Percent | Average | Percentage of CPU idle time |
| aws_kafka_broker_node_CpuIoWait | CpuIoWait | Cpu Io Wait | Percent | Average | Percentage of CPU idle time during a pending disk operation |
| aws_kafka_broker_node_CpuSystem | CpuSystem | Cpu System | Percent | Average | Percentage of CPU in kernel space |
| aws_kafka_broker_node_CpuUser | CpuUser | Cpu User | Percent | Average | Percentage of CPU in user space |
| aws_kafka_broker_node_KafkaAppLogsDiskUsed | KafkaAppLogsDiskUsed | Kafka App Logs Disk Used | Percent | Average | Percentage of disk space used for application logs |
| aws_kafka_broker_node_KafkaDataLogsDiskUsed | KafkaDataLogsDiskUsed | Kafka Data Logs Disk Used | Percent | Average | Percentage of disk space used for data logs |
| aws_kafka_broker_node_LeaderCount | LeaderCount | Leader Count | Count | Average | Total number of leaders of partitions per broker, not including replicas |
| aws_kafka_broker_node_MemoryBuffered | MemoryBuffered | Memory Buffered | Bytes | Average | Size in bytes of buffered memory for the broker |
| aws_kafka_broker_node_MemoryCached | MemoryCached | Memory Cached | Bytes | Average | Size in bytes of cached memory for the broker |
| aws_kafka_broker_node_MemoryFree | MemoryFree | Memory Free | Bytes | Average | Size in bytes of free and available memory for the broker |
| aws_kafka_broker_node_HeapMemoryAfterGC | HeapMemoryAfterGC | Heap Memory After GC | Percent | Average | Percentage of total heap memory in use after garbage collection |
| aws_kafka_broker_node_MemoryUsed | MemoryUsed | Memory Used | Bytes | Average | Size in bytes of memory in use for the broker |
| aws_kafka_broker_node_MessagesInPerSec | MessagesInPerSec | Messages In Per Sec | Count/Second | Average | Number of incoming messages per second for the broker |
| aws_kafka_broker_node_NetworkRxDropped | NetworkRxDropped | Network Rx Dropped | Count | Average | Number of dropped receive packages |
| aws_kafka_broker_node_NetworkRxErrors | NetworkRxErrors | Network Rx Errors | Count | Average | Number of network receive errors for the broker |
| aws_kafka_broker_node_NetworkRxPackets | NetworkRxPackets | Network Rx Packets | Count | Average | Number of packets received by the broker |
| aws_kafka_broker_node_NetworkTxDropped | NetworkTxDropped | Network Tx Dropped | Count | Average | Number of dropped transmit packages |
| aws_kafka_broker_node_NetworkTxErrors | NetworkTxErrors | Network Tx Errors | Count | Average | Number of network transmit errors for the broker |
| aws_kafka_broker_node_NetworkTxPackets | NetworkTxPackets | Network Tx Packets | Count | Average | Number of packets transmitted by the broker |
| aws_kafka_broker_node_PartitionCount | PartitionCount | Partition Count | Count | Average | The total number of topic partitions per broker, including replicas. |
| aws_kafka_broker_node_ProduceTotalTimeMsMean | ProduceTotalTimeMsMean | Produce Total Time Ms Mean | Milliseconds | Average | The mean produce time in milliseconds. |
| aws_kafka_broker_node_RequestBytesMean | RequestBytesMean | Request Bytes Mean | Bytes | Average | The mean number of request bytes for the broker. |
| aws_kafka_broker_node_RequestTime | RequestTime | Request Time | Milliseconds | Average | The average time in milliseconds spent in broker network and I/O threads to process requests. |
| aws_kafka_broker_node_RootDiskUsed | RootDiskUsed | Root Disk Used | Percent | Average | The percentage of the root disk used by the broker. |
| aws_kafka_broker_node_SwapFree | SwapFree | Swap Free | Bytes | Average | The size in bytes of swap memory that is available for the broker. |
| aws_kafka_broker_node_SwapUsed | SwapUsed | Swap Used | Bytes | Average | The size in bytes of swap memory that is in use for the broker. |
| aws_kafka_broker_node_TrafficShaping | TrafficShaping | Traffic Shaping | Count | Average | High-level metrics indicating the number of packets shaped (dropped or queued) due to exceeding network allocations. Finer detail is available with PER_BROKER metrics. |
| aws_kafka_broker_node_UnderMinIsrPartitionCount | UnderMinIsrPartitionCount | Under Min Isr Partition Count | Count | Average | The number of under minIsr partitions for the broker. |
| aws_kafka_broker_node_UnderReplicatedPartitions | UnderReplicatedPartitions | Under Replicated Partitions | Count | Average | The number of under-replicated partitions for the broker. |
| aws_kafka_broker_node_UserPartitionExists | UserPartitionExists | User Partition Exists | Count | Average | A Boolean metric that indicates the presence of a user-owned partition on a broker. A value of 1 indicates the presence of partitions on the broker. |
| aws_kafka_broker_node_ZooKeeperRequestLatencyMsMean | ZooKeeperRequestLatencyMsMean | Zoo Keeper Request Latency Ms Mean | Milliseconds | Average | For ZooKeeper-based cluster. The mean latency in milliseconds for Apache ZooKeeper requests from broker. |
| aws_kafka_broker_node_ZooKeeperSessionState | ZooKeeperSessionState | Zoo Keeper Session State | Count | Average | For ZooKeeper-based cluster. Connection status of broker's ZooKeeper session which may be one of the following: NOT_CONNECTED: '0.0', ASSOCIATING: '0.1', CONNECTING: '0.5', CONNECTEDREADONLY: '0.8', CONNECTED: '1.0', CLOSED: '5.0', AUTH_FAILED: '10.0'. |