Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a fully managed service that enables you to build and run applications that use Apache Kafka to process streaming data.
Amazon MSK provides the control-plane operations, such as those for creating, updating, and deleting clusters.
- You can use Apache Kafka data-plane operations, such as those for producing and consuming data.
- It runs open-source versions of Apache Kafka. This means existing applications, tooling, and plugins from partners and the Apache Kafka community are supported without requiring changes to application code.
Use the AWS public cloud integration to discover and collect metrics against the AWS service.
External reference
Monitoring an Amazon MSK Cluster
Setup
To set up the AWS integration and discover the AWS service, go to AWS Integration Discovery Profile and select MSK Cluster.
Event support
CloudTrail event support
- Supported
- Configurable in OpsRamp AWS Integration Discovery Profile.
CloudWatch alarm support
- Supported
- Configurable in OpsRamp AWS Integration Discovery Profile.
Supported metrics
| OpsRamp Metric | AWS Metric | Metric Display Name | Unit | Aggregation Type | Description |
|---|---|---|---|---|---|
| aws_kafka_cluster_ActiveControllerCount | ActiveControllerCount | Active Controller Count | Count | Average | Only one controller per cluster should be active at any given time |
| aws_kafka_cluster_GlobalPartitionCount | GlobalPartitionCount | Global Partition Count | Count | Average | Number of partitions across all topics in the cluster, excluding replicas |
| aws_kafka_cluster_GlobalTopicCount | GlobalTopicCount | Global Topic Count | Count | Average | Total number of topics across all brokers in the cluster |
| aws_kafka_cluster_EstimatedMaxTimeLag | EstimatedMaxTimeLag | Estimated Max Time Lag | Seconds | Average | Time estimate in seconds to drain MaxOffsetLag |
| aws_kafka_cluster_MaxOffsetLag | MaxOffsetLag | Max Offset Lag | Count | Average | Maximum offset lag across all partitions in a topic |
| aws_kafka_cluster_OfflinePartitionsCount | OfflinePartitionsCount | Offline Partitions Count | Count | Average | Total number of partitions that are offline in the cluster |
| aws_kafka_cluster_RollingEstimatedTimeLagMax | RollingEstimatedTimeLagMax | Rolling Estimated Time Lag Max | Seconds | Average | Rolling maximum time estimate in seconds to drain partition offset lag across all partitions in a topic |
| aws_kafka_cluster_SumOffsetLag | SumOffsetLag | Sum Offset Lag | Count | Average | Aggregated offset lag for all partitions in a topic |