Introduction

Amazon Kinesis Data Firehose is a fully managed service for delivering real-time streaming data to destinations such as Amazon Simple Storage Service (Amazon S3), Amazon Redshift, Amazon Elasticsearch Service (Amazon ES), and Splunk. Kinesis Data Firehose is part of the Kinesis streaming data platform, along with Kinesis Data Streams, Kinesis Video Streams, and Amazon Kinesis Data Analytics. With Kinesis Data Firehose, you don’t need to write applications or manage resources. You configure your data producers to send data to Kinesis Data Firehose, and it automatically delivers the data to the destination that you specified. You can also configure Kinesis Data Firehose to transform your data before delivering it.

Setup

To set up the OpsRamp AWS integration and discover the AWS service, go to AWS Integration Discovery Profile and select Firehose.

Metrics

OpsRamp MetricMetric Display NameUnitAggregation TypeDescription
aws_firehose_PutRecord_BytesPutRecord.BytesBytesAverageNumber of bytes put to the Amazon Kinesis stream using PutRecord over the specified time period.
aws_firehose_PutRecord_LatencyPutRecord.LatencymsAverageTime taken per PutRecord operation, measured over the specified time period.
aws_firehose_IncomingBytesIncomingBytesBytesAverageNumber of bytes successfully put to the Amazon Kinesis stream over the specified time period. This metric includes bytes from PutRecord and PutRecords.
aws_firehose_IncomingRecordsIncomingRecordsCountAverageNumber of records sucessfully put to the Amazon Kinesis stream over the specified time period. This metric includes record counts from PutRecord and PutRecords.
aws_firehose_DeliveryToElasticsearch_BytesDeliveryToElasticsearch.Bytes.firehoseBytesAverageNumber of bytes indexed to Amazon ES over the specified time period.
aws_firehose_DeliveryToElasticsearch_RecordsDeliveryToElasticsearch.Records.firehoseCountAverageNumber of records indexed to Amazon ES over the specified time period.
aws_firehose_DeliveryToElasticsearch_SuccessDeliveryToElasticsearch.Success.firehoseCountAverageSum of the successfully indexed records over the sum of records that were attempted.
aws_firehose_DeliveryToRedshift_BytesDeliveryToRedshift.Bytes.firehoseBytesAverageNumber of bytes copied to Amazon Redshift over the specified time period.
aws_firehose_DeliveryToRedshift_RecordsDeliveryToRedshift.Records.firehoseCountAverageNumber of records copied to Amazon Redshift over the specified time period.
aws_firehose_DeliveryToRedshift_SuccessDeliveryToRedshift.Success.firehoseCountAverageSum of successful Amazon Redshift COPY commands over the sum of all Amazon Redshift COPY commands.
aws_firehose_DeliveryToRedshift_SuccessDeliveryToS3.Bytes.firehoseBytesAverageNumber of bytes delivered to Amazon S3 over the specified time period.
aws_firehose_DeliveryToS3_DataFreshnessDeliveryToS3.DataFreshness.firehoseSecondsAverageAge (from getting into Kinesis Firehose to now) of the oldest record in Kinesis Firehose. Any record older than this age has been delivered to the S3 bucket.
aws_firehose_DeliveryToS3_RecordsDeliveryToS3.Records.firehoseCountAverageNumber of records delivered to Amazon S3 over the specified time period.
aws_firehose_DeliveryToS3_SuccessDeliveryToS3.Success.firehoseCountAverageSum of successful Amazon S3 put commands over the sum of all Amazon S3 put commands.
aws_firehose_DescribeDeliveryStream_LatencyDescribeDeliveryStream.Latency.firehoseMillisecondsAverageTime taken per DescribeDeliveryStream operation, measured over the specified time period.
aws_firehose_DescribeDeliveryStream_RequestsDescribeDeliveryStream.Requests.firehoseCountAverageTotal number of DescribeDeliveryStream requests.
aws_firehose_ListDeliveryStreams_LatencyListDeliveryStreams.Latency.firehoseMillisecondsAverageTime taken per ListDeliveryStream operation, measured over the specified time period.
aws_firehose_ListDeliveryStreams_RequestsListDeliveryStreams.Requests.firehoseCountAverageTotal number of ListFirehose requests.
aws_firehose_PutRecord_RequestsPutRecord.Requests.firehoseCountAverageTotal number of PutRecord requests, which is equal to total number of records from PutRecord operations.
aws_firehose_PutRecordBatch_BytesPutRecordBatch.Bytes.firehoseBytesAverageNumber of bytes put to the Kinesis Firehose delivery stream using PutRecordBatch over the specified time period.
aws_firehose_PutRecordBatch_LatencyPutRecordBatch.Latency.firehoseMillisecondsAverageTime taken per PutRecordBatch operation, measured over the specified time period.
aws_firehose_PutRecordBatch_RecordsPutRecordBatch.Records.firehoseCountAverageTotal number of records from PutRecordBatch operations.
aws_firehose_PutRecordBatch_RequestsPutRecordBatch.Requests.firehoseCountAverageTotal number of PutRecordBatch requests.
aws_firehose_UpdateDeliveryStream_RequestsUpdateDeliveryStream.Requests.firehoseCountAverageTotal number of UpdateDeliveryStream requests.
aws_firehose_UpdateDeliveryStream_LatencyUpdateDeliveryStream.Latency.firehoseMillisecondsAverageTime taken per UpdateDeliveryStream operation, measured over the specified time period.
aws_firehose_BackupToS3_BytesBackupToS3 BytesBytesAverageNumber of bytes delivered to Amazon S3 for backup over the specified time period.
aws_firehose_BackupToS3_DataFreshnessBackupToS3 DataFreshnessSecondsAverageAge (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to the Amazon S3 bucket for backup. Kinesis Data Firehose emits this metric when data transformation is enabled for Amazon S3 or Amazon Redshift destinations.
aws_firehose_BackupToS3_RecordsBackupToS3 RecordsCountAverageNumber of records delivered to Amazon S3 for backup over the specified time period.
aws_firehose_BytesPerSecondLimitBytes Per Second LimitBytesAverageThe current maximum number of bytes per second that a delivery stream can ingest before throttling.
aws_firehose_DataReadFromKinesisStream_BytesData Read From Kinesis Stream BytesBytesAverageWhen the data source is a Kinesis data stream, this metric indicates the number of bytes read from that data stream.
aws_firehose_DataReadFromKinesisStream_RecordsData Read From Kinesis Stream RecordsCountAverageWhen the data source is a Kinesis data stream, this metric indicates the number of records read from that data stream.
aws_firehose_DeliveryToElasticsearch_DataFreshnessDelivery To Elasticsearch Data FreshnessSecondsAverageAge (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose.
aws_firehose_DeliveryToSplunk_BytesDelivery To Splunk BytesBytesAverageNumber of bytes delivered to Splunk over the specified time period.
aws_firehose_DeliveryToSplunk_DataAckLatencyDelivery To Splunk Data Ack LatencySecondsAverageThe approximate duration it takes to receive an acknowledgement from Splunk after Kinesis Data Firehose sends it data.
aws_firehose_DeliveryToSplunk_DataFreshnessDelivery To Splunk Data FreshnessSecondsAverageAge (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to Splunk.
aws_firehose_KinesisMillisBehindLatestKinesis Millis Behind LatestMillisecondAverageWhen the data source is a Kinesis data stream, this metric indicates the number of milliseconds that the last read record is behind the newest record in the Kinesis data stream.
aws_firehose_RecordsPerSecondLimitRecords Per Second LimitBytesAverageCurrent maximum number of records per second that a delivery stream can ingest before throttling.
aws_firehose_ThrottledRecordsThrottled RecordsCountAverageNumber of records that were throttled because data ingestion exceeded one of the delivery stream limits.
aws_firehose_IncomingPutRequestsIncoming Put RequestsCountAverageNumber of PutRecord and PutRecordBatch requests over the specified period of time.
aws_firehose_PutRequestsPerSecondLimitPut Requests Per Second LimitCountAverageMaximum number of put requests per second that a delivery stream can handle before throttling.
aws_firehose_ThrottledDescribeStreamThrottled Describe StreamCountAverageTotal number of times the DescribeStream operation is throttled when the data source is a Kinesis data stream.
aws_firehose_ThrottledGetRecordsThrottled Get RecordsCountAverageTotal number of times the GetRecords operation is throttled when the data source is a Kinesis data stream.
aws_firehose_ThrottledGetShardIteratorThrottled Get Shard IteratorCountAverageTotal number of times the GetShardIterator operation is throttled when the data source is a Kinesis data stream.
aws_firehose_KMSKeyAccessDeniedKMS Key Access DeniedCountAverageNumber of times the service encounters a KMSAccessDeniedException for the delivery stream.
aws_firehose_KMSKeyDisabledKMS Key DisabledCountAverageNumber of times the service encounters a KMSDisabledException for the delivery stream.
aws_firehose_KMSKeyInvalidStateKMS Key Invalid StateCountAverageNumber of times the service encounters a KMSInvalidStateException for the delivery stream.
aws_firehose_KMSKeyNotFoundKMS Key Not FoundCountAverageNumber of times the service encounters a KMSNotFoundException for the delivery stream.

Event support

CloudTrail event support

  • Supported (CreateDeliveryStream, DeleteDeliveryStream)
  • Configurable in OpsRamp AWS Integration Discovery Profile.

CloudWatch alarm support

  • Supported
  • Configurable in OpsRamp AWS Integration Discovery Profile.

External reference