Amazon Kinesis Data Firehose is a fully managed service for delivering real-time streaming data to destinations such as Amazon Simple Storage Service (Amazon S3), Amazon Redshift, Amazon Elasticsearch Service (Amazon ES), and Splunk. Kinesis Data Firehose is part of the Kinesis streaming data platform, including Kinesis Data Streams, Kinesis Video Streams, and Amazon Kinesis Data Analytics. With Kinesis Data Firehose, you do not need to write applications or manage resources. You configure your data producers to send data to Kinesis Data Firehose, and it automatically delivers the data to the destination that you specified. You can also configure Kinesis Data Firehose to transform your data before delivering it.
Use the AWS public cloud integration to discover and collect metrics against the AWS service.
External reference
What Is Amazon Kinesis Data Firehose?
Setup
To set up the AWS integration and discover the AWS service, go to AWS Integration Discovery Profile and select Firehose.
Event support
CloudTrail event support
- Supported (CreateDeliveryStream, DeleteDeliveryStream)
- Configurable in OpsRamp AWS Integration Discovery Profile.
CloudWatch alarm support
- Supported
- Configurable in OpsRamp AWS Integration Discovery Profile.
Supported metrics
| OpsRamp Metric | AWS Metric | Metric Display Name | Unit | Aggregation Type | Description |
|---|---|---|---|---|---|
| aws_firehose_ActivePartitionsLimit | ActivePartitionsLimit | Active Partitions Limit | Count | Average | The maximum number of active partitions that a Firehose stream processes before sending data to the error bucket. |
| aws_firehose_PartitionCount | PartitionCount | Partition Count | Count | Average | The number of partitions that are being processed, in other words, the active partition count. This number varies between 1 and the partition count limit of 500 (default). |
| aws_firehose_PartitionCountExceeded | PartitionCountExceeded | Partition Count Exceeded | None | Average | This metric indicates if you are exceeding the partition count limit. It emits 1 or 0 based on whether limit is breached or not. |
| aws_firehose_JQProcessing_Duration | JQProcessing.Duration | JQ Processing Duration | Milliseconds | Average | Returns the amount of time it took to execute JQ expression in the JQ Lambda function. |
| aws_firehose_PerPartitionThroughput | PerPartitionThroughput | Per Partition Throughput | Bytes/Second | Average | Indicates the throughput that is being processed per partition. This metric enables you to monitor the per partition throughput. |
| aws_firehose_DeliveryToS3_ObjectCount | DeliveryToS3.ObjectCount | Delivery To S3 Object Count | Count | Average | Indicates the number of objects that are being delivered to your S3 bucket. |
| aws_firehose_DeliveryToAmazonOpenSearchService_Bytes | DeliveryToAmazonOpenSearchService.Bytes | Delivery To Amazon Open Search Service Bytes | Bytes | Average | The number of bytes indexed to OpenSearch Service over the specified time period. |
| aws_firehose_DeliveryToAmazonOpenSearchService_DataFreshness | DeliveryToAmazonOpenSearchService.DataFreshness | Delivery To Amazon Open Search Service Data Freshness | Seconds | Average | The age (from getting into Amazon Data Firehose to now) of the oldest record in Amazon Data Firehose. Any record older than this age has been delivered to OpenSearch Service. |
| aws_firehose_DeliveryToAmazonOpenSearchService_Records | DeliveryToAmazonOpenSearchService.Records | Delivery To Amazon Open Search Service Records | Count | Average | The number of records indexed to OpenSearch Service over the specified time period. |
| aws_firehose_DeliveryToAmazonOpenSearchService_Success | DeliveryToAmazonOpenSearchService.Success | Delivery To Amazon Open Search Service Success | Count | Sum | The sum of the successfully indexed records. |
| aws_firehose_DeliveryToAmazonOpenSearchService_AuthFailure | DeliveryToAmazonOpenSearchService.AuthFailure | Delivery To Amazon Open Search Service Auth Failure | None | Average | Authentication/authorization error. Verify the OS/ES cluster policy and role permissions. 0 indicates that there is no issue. 1 indicates authentication failure. |
| aws_firehose_DeliveryToAmazonOpenSearchService_DeliveryRejected | DeliveryToAmazonOpenSearchService.DeliveryRejected | Delivery To Amazon Open Search Service Delivery Rejected | None | Average | Delivery rejected error. Verify the OS/ES cluster policy and role permissions. 0 indicates that there is no issue. 1 indicates that there''s a delivery failure. |
| aws_firehose_DeliveryToAmazonOpenSearchServerless_Bytes | DeliveryToAmazonOpenSearchServerless.Bytes | Delivery To Amazon Open Search Serverless Bytes | Bytes | Average | The number of bytes indexed to OpenSearch Serverless over the specified time period. |
| aws_firehose_DeliveryToAmazonOpenSearchServerless_DataFreshness | DeliveryToAmazonOpenSearchServerless.DataFreshness | Delivery To Amazon Open Search Serverless Data Freshness | Seconds | Average | The age (from getting into Amazon Data Firehose to now) of the oldest record in Amazon Data Firehose. Any record older than this age has been delivered to OpenSearch Serverless. |
| aws_firehose_DeliveryToAmazonOpenSearchServerless_Records | DeliveryToAmazonOpenSearchServerless.Records | Delivery To Amazon Open Search Serverless Records | Count | Average | The number of records indexed to OpenSearch Serverless over the specified time period. |
| aws_firehose_DeliveryToAmazonOpenSearchServerless_Success | DeliveryToAmazonOpenSearchServerless.Success | Delivery To Amazon Open Search Serverless Success | Count | Sum | The sum of the successfully indexed records. |
| aws_firehose_DeliveryToAmazonOpenSearchServerless_AuthFailure | DeliveryToAmazonOpenSearchServerless.AuthFailure | Delivery To Amazon Open Search Serverless Auth Failure | None | Average | Authentication/authorization error. Verify the OS/ES cluster policy and role permissions. 0 indicates that there is no issue. 1 indicates that there is an authentication failure. |
| aws_firehose_DeliveryToAmazonOpenSearchServerless_DeliveryRejected | DeliveryToAmazonOpenSearchServerless.DeliveryRejected | Delivery To Amazon Open Search Serverless Delivery Rejected | None | Average | Delivery rejected error. Verify the OS/ES cluster policy and role permissions. 0 indicates that there is no issue. 1 indicates that there is a delivery failure. |
| aws_firehose_DeliveryToRedshift_Bytes | DeliveryToRedshift.Bytes | Delivery To Redshift Bytes | Count | Average | The number of bytes copied to Amazon Redshift over the specified time period. |
| aws_firehose_DeliveryToRedshift_Records | DeliveryToRedshift.Records | Delivery To Redshift Records | Count | Average | The number of records copied to Amazon Redshift over the specified time period. |
| aws_firehose_DeliveryToRedshift_Success | DeliveryToRedshift.Success | Delivery To Redshift Success | Count | Sum | The sum of successful Amazon Redshift COPY commands. |
| aws_firehose_DeliveryToRedshift_DataFreshness | DeliveryToRedshift.DataFreshness | Delivery To Redshift Data Freshness | Seconds | Average | The age (from getting into Amazon Data Firehose to now) of the oldest record in Amazon Data Firehose. Any record that is older than this age is delivered to the Amazon Redshift cluster. |
| aws_firehose_BackupToS3_Bytes | BackupToS3.Bytes | Backup To S3 Bytes | Count | Average | The number of bytes delivered to Amazon S3 for backup over the specified time period. Amazon Data Firehose emits this metric when backup to Amazon S3 is enabled. |
| aws_firehose_BackupToS3_DataFreshness | BackupToS3.DataFreshness | Backup To S3 Data Freshness | Seconds | Average | The age (from getting into Amazon Data Firehose to now) of the oldest record in Amazon Data Firehose. Any record older than this age has been delivered to the Amazon S3 bucket for backup. Amazon Data Firehose emits this metric when backup to Amazon S3 is enabled. |
| aws_firehose_BackupToS3_Records | BackupToS3.Records | Backup To S3 Records | Count | Average | The number of records delivered to Amazon S3 for backup over the specified time period. Amazon Data Firehose emits this metric when backup to Amazon S3 is enabled. |
| aws_firehose_BackupToS3_Success | BackupToS3.Success | Backup To S3 Success | Count | Sum | The sum of successful Amazon S3 put commands for backup. Amazon Data Firehose emits this metric when backup to Amazon S3 is enabled. |
| aws_firehose_DeliveryToS3_Bytes | DeliveryToS3.Bytes | Delivery To S3 Bytes | Bytes | Average | The number of bytes delivered to Amazon S3 over the specified time period. When data transformation is enabled, this metric reflects the pre-processed byte size before transformation. |
| aws_firehose_DeliveryToS3_DataFreshness | DeliveryToS3.DataFreshness | Delivery To S3 Data Freshness | Seconds | Average | The age (from getting into Amazon Data Firehose to now) of the oldest record in Amazon Data Firehose. Any record older than this age has been delivered to the S3 bucket. |
| aws_firehose_DeliveryToS3_Records | DeliveryToS3.Records | Delivery To S3 Records | Count | Average | The number of records delivered to Amazon S3 over the specified time period. |
| aws_firehose_DeliveryToS3_Success | DeliveryToS3.Success | Delivery To S3 Success | Count | Sum | The sum of successful Amazon S3 put commands. |
| aws_firehose_DeliveryToSnowflake_Bytes | DeliveryToSnowflake.Bytes | Delivery To Snowflake Bytes | Bytes | Average | The number of bytes delivered to Snowflake over the specified time period. |
| aws_firehose_DeliveryToSnowflake_DataFreshness | DeliveryToSnowflake.DataFreshness | Delivery To Snowflake Data Freshness | Seconds | Average | Age (from getting into Firehose to now) of the oldest record in Firehose. Any record older than this age has been delivered to Snowflake. Note that it can take a few seconds to commit data to Snowflake after Firehose insert call is successful. |
| aws_firehose_DeliveryToSnowflake_DataCommitLatency | DeliveryToSnowflake.DataCommitLatency | Delivery To Snowflake Data Commit Latency | Seconds | Average | The time it takes for the data to be committed to Snowflake after Firehose inserted records successfully. |
| aws_firehose_DeliveryToSnowflake_Records | DeliveryToSnowflake.Records | Delivery To Snowflake Records | Count | Average | The number of records delivered to Snowflake over the specified time period. |
| aws_firehose_DeliveryToSnowflake_Success | DeliveryToSnowflake.Success | Delivery To Snowflake Success | Count | Sum | The sum of successful insert calls made to Snowflake. |
| aws_firehose_DeliveryToSplunk_Bytes | DeliveryToSplunk.Bytes | Delivery To Splunk Bytes | Bytes | Average | The number of bytes delivered to Splunk over the specified time period. |
| aws_firehose_DeliveryToSplunk_DataAckLatency | DeliveryToSplunk.DataAckLatency | Delivery To Splunk Data Ack Latency | Seconds | Average | The approximate duration it takes to receive an acknowledgement from Splunk after Amazon Data Firehose sends it data. The increasing or decreasing trend for this metric is more useful than the absolute approximate value. Increasing trends can indicate slower indexing and acknowledgement rates from Splunk indexers. |
| aws_firehose_DeliveryToSplunk_DataFreshness | DeliveryToSplunk.DataFreshness | Delivery To Splunk Data Freshness | Seconds | Average | Age (from getting into Amazon Data Firehose to now) of the oldest record in Amazon Data Firehose. Any record older than this age has been delivered to Splunk. |
| aws_firehose_DeliveryToSplunk_Records | DeliveryToSplunk.Records | Delivery To Splunk Records | Count | Average | The number of records delivered to Splunk over the specified time period. |
| aws_firehose_DeliveryToSplunk_Success | DeliveryToSplunk.Success | Delivery To Splunk Success | Count | Sum | The sum of the successfully indexed records. |
| aws_firehose_DeliveryToHttpEndpoint_Bytes | DeliveryToHttpEndpoint.Bytes | Delivery To Http Endpoint Bytes | Bytes | Average | The number of bytes delivered successfully to the HTTP endpoint. |
| aws_firehose_DeliveryToHttpEndpoint_Records | DeliveryToHttpEndpoint.Records | Delivery To Http Endpoint Records | Count | Average | The number of records delivered successfully to the HTTP endpoint. This metric is only emitted for successful delivery attempts and is not emitted when a delivery attempt fails. |
| aws_firehose_DeliveryToHttpEndpoint_DataFreshness | DeliveryToHttpEndpoint.DataFreshness | Delivery To Http Endpoint Data Freshness | Seconds | Average | Age of the oldest record in Amazon Data Firehose. |
| aws_firehose_DeliveryToHttpEndpoint_Success | DeliveryToHttpEndpoint.Success | Delivery To Http Endpoint Success | Count | Average | The number of records successfully delivered to the HTTP endpoint per delivery attempt. Unlike DeliveryToHttpEndpoint.Records, this metric is emitted for every delivery attempt. On success, the value equals the number of records in the delivery attempt. On failure of all records in the delivery attempt, the value is 0. Use the Minimum statistic to monitor for delivery failures. |
| aws_firehose_DeliveryToHttpEndpoint_ProcessedBytes | DeliveryToHttpEndpoint.ProcessedBytes | Delivery To Http Endpoint Processed Bytes | Bytes | Sum | The number of attempted processed bytes, including retries. |
| aws_firehose_DeliveryToHttpEndpoint_ProcessedRecords | DeliveryToHttpEndpoint.ProcessedRecords | Delivery To Http Endpoint Processed Records | Count | Sum | The number of attempted records including retries. |
| aws_firehose_DataReadFromKinesisStream_Bytes | DataReadFromKinesisStream.Bytes | Data Read From Kinesis Stream Bytes | Bytes | Average | When the data source is a Kinesis data stream, this metric indicates the number of bytes read from that data stream. This number includes rereads due to failovers. |
| aws_firehose_DataReadFromKinesisStream_Records | DataReadFromKinesisStream.Records | Data Read From Kinesis Stream Records | Count | Average | When the data source is a Kinesis data stream, this metric indicates the number of records read from that data stream. This number includes rereads due to failovers. |
| aws_firehose_ThrottledDescribeStream | ThrottledDescribeStream | Throttled Describe Stream | Count | Average | The total number of times the DescribeStream operation is throttled when the data source is a Kinesis data stream. |
| aws_firehose_ThrottledGetRecords | ThrottledGetRecords | Throttled Get Records | Count | Average | The total number of times the GetRecords operation is throttled when the data source is a Kinesis data stream. |
| aws_firehose_ThrottledGetShardIterator | ThrottledGetShardIterator | Throttled Get Shard Iterator | Count | Average | The total number of times the GetShardIterator operation is throttled when the data source is a Kinesis data stream. |
| aws_firehose_KinesisMillisBehindLatest | KinesisMillisBehindLatest | Kinesis Millis Behind Latest | Milliseconds | Average | When the data source is a Kinesis data stream, this metric indicates the number of milliseconds that the last read record is behind the newest record in the Kinesis data stream. |
| aws_firehose_IncomingBytes | IncomingBytes | Incoming Bytes | Bytes | Average | The number of bytes ingested successfully into the Firehose stream over the specified time period. Data ingestion could be throttled when it exceeds one of the Firehose stream limits. Throttled data will not be counted for IncomingBytes. |
| aws_firehose_IncomingRecords | IncomingRecords | Incoming Records | Count | Average | The number of records ingested successfully into the Firehose stream over the specified time period. Data ingestion could be throttled when it exceeds one of the Firehose stream limits. Throttled data will not be counted for IncomingRecords. |
| aws_firehose_IncomingPutRequests | IncomingPutRequests | Incoming Put Requests | Count | Average | The number of successful PutRecord and PutRecordBatch requests over a specified period of time. |
| aws_firehose_RecordsPerSecondLimit | RecordsPerSecondLimit | Records Per Second Limit | Count | Average | The current maximum number of records per second that a Firehose stream can ingest before throttling. |
| aws_firehose_BytesPerSecondLimit | BytesPerSecondLimit | Bytes Per Second Limit | Bytes | Average | The current maximum number of bytes per second that a Firehose stream can ingest before throttling. To request an increase to this limit, go to the AWS Support Center and choose Create case, then choose Service limit increase. |
| aws_firehose_ThrottledRecords | ThrottledRecords | Throttled Records | Count | Average | The number of records that were throttled because data ingestion exceeded one of the Firehose stream limits. |
| aws_firehose_DataReadFromSource_Records | DataReadFromSource.Records | Data Read From Source Records | Count | Average | The number of records read from the source Kafka Topic. |
| aws_firehose_DataReadFromSource_Bytes | DataReadFromSource.Bytes | Data Read From Source Bytes | Bytes | Average | The number of bytes read from the source Kafka Topic. |
| aws_firehose_SourceThrottled_Delay | SourceThrottled.Delay | Source Throttled Delay | Milliseconds | Average | The amount of time that the source Kafka cluster is delayed in returning the records from the source Kafka Topic. |
| aws_firehose_KafkaOffsetLag | KafkaOffsetLag | Kafka Offset Lag | Count | Average | The difference between the largest offset of the record that Firehose has read from the source Kafka Topic and the largest offset of the record available from the source Kafka Topic. |
| aws_firehose_FailedValidation_Records | FailedValidation.Records | Failed Validation Records | Count | Average | The number of records that failed record validation. |
| aws_firehose_FailedValidation_Bytes | FailedValidation.Bytes | Failed Validation Bytes | Bytes | Average | The number of bytes that failed record validation. |
| aws_firehose_DataReadFromSource_Backpressured | DataReadFromSource.Backpressured | Data Read From Source Backpressured | None | Average | Indicates that a Firehose stream is delayed in reading records from the source partition either because BytesPerSecondLimit per partition has exceeded or that the normal flow of delivery is slow or has stopped. |
| aws_firehose_DescribeDeliveryStream_Latency | DescribeDeliveryStream.Latency | Describe Delivery Stream Latency | Milliseconds | Average | The time taken per DescribeDeliveryStream operation, measured over the specified time period. |
| aws_firehose_DescribeDeliveryStream_Requests | DescribeDeliveryStream.Requests | Describe Delivery Stream Requests | Count | Average | The total number of DescribeDeliveryStream requests. |
| aws_firehose_ListDeliveryStreams_Latency | ListDeliveryStreams.Latency | List Delivery Streams Latency | Milliseconds | Average | The time taken per ListDeliveryStream operation, measured over the specified time period. |
| aws_firehose_ListDeliveryStreams_Requests | ListDeliveryStreams.Requests | List Delivery Streams Requests | Count | Average | The total number of ListFirehose requests. |
| aws_firehose_PutRecord_Bytes | PutRecord.Bytes | Put Record Bytes | Bytes | Average | The number of bytes put to the Firehose stream using PutRecord over the specified time period. |
| aws_firehose_PutRecord_Latency | PutRecord.Latency | Put Record Latency | Milliseconds | Average | The time taken per PutRecord operation, measured over the specified time period. |
| aws_firehose_PutRecord_Requests | PutRecord.Requests | Put Record Requests | Count | Average | The total number of PutRecord requests, which is equal to total number of records from PutRecord operations. |
| aws_firehose_PutRecordBatch_Bytes | PutRecordBatch.Bytes | Put Record Batch Bytes | Bytes | Average | The number of bytes put to the Firehose stream using PutRecordBatch over the specified time period. |
| aws_firehose_PutRecordBatch_Latency | PutRecordBatch.Latency | Put Record Batch Latency | Milliseconds | Average | The time taken per PutRecordBatch operation, measured over the specified time period. |
| aws_firehose_PutRecordBatch_Records | PutRecordBatch.Records | Put Record Batch Records | Count | Average | The total number of records from PutRecordBatch operations. |
| aws_firehose_PutRecordBatch_Requests | PutRecordBatch.Requests | Put Record Batch Requests | Count | Average | The total number of PutRecordBatch requests. |
| aws_firehose_UpdateDeliveryStream_Latency | UpdateDeliveryStream.Latency | Update Delivery Stream Latency | Milliseconds | Average | The time taken per UpdateDeliveryStream operation, measured over the specified time period. |
| aws_firehose_UpdateDeliveryStream_Requests | UpdateDeliveryStream.Requests | Update Delivery Stream Requests | Count | Average | The total number of UpdateDeliveryStream requests. |
| aws_firehose_ExecuteProcessing_Duration | ExecuteProcessing.Duration | Execute Processing Duration | Milliseconds | Average | The time it takes for each Lambda function invocation performed by Firehose. |
| aws_firehose_ExecuteProcessing_Success | ExecuteProcessing.Success | Execute Processing Success | Percent | Average | The sum of the successful Lambda function invocations over the sum of the total Lambda function invocations. |
| aws_firehose_SucceedProcessing_Records | SucceedProcessing.Records | Succeed Processing Records | Count | Sum | The number of successfully processed records over the specified time period. |
| aws_firehose_SucceedProcessing_Bytes | SucceedProcessing.Bytes | Succeed Processing Bytes | Bytes | Sum | The number of successfully processed bytes over the specified time period. |
| aws_firehose_OutputDecompressedBytes_Success | OutputDecompressedBytes.Success | Output Decompressed Bytes Success | Bytes | Average | Successful decompressed data in bytes. |
| aws_firehose_OutputDecompressedBytes_Failed | OutputDecompressedBytes.Failed | Output Decompressed Bytes Failed | Bytes | Average | Failed decompressed data in bytes. |
| aws_firehose_OutputDecompressedRecords_Success | OutputDecompressedRecords.Success | Output Decompressed Records Success | Count | Average | Number of successful decompressed records. |
| aws_firehose_OutputDecompressedRecords_Failed | OutputDecompressedRecords.Failed | Output Decompressed Records Failed | Count | Average | Number of failed decompressed records. |
| aws_firehose_SucceedConversion_Records | SucceedConversion.Records | Succeed Conversion Records | Count | Sum | The number of successfully converted records. |
| aws_firehose_SucceedConversion_Bytes | SucceedConversion.Bytes | Succeed Conversion Bytes | Bytes | Sum | The size of the successfully converted records. |
| aws_firehose_FailedConversion_Records | FailedConversion.Records | Failed Conversion Records | Count | Sum | The number of records that could not be converted. |
| aws_firehose_FailedConversion_Bytes | FailedConversion.Bytes | Failed Conversion Bytes | Bytes | Sum | The size of the records that could not be converted. |
| aws_firehose_KMSKeyAccessDenied | KMSKeyAccessDenied | KMS Key Access Denied | Count | Average | The number of times the service encounters a KMSAccessDeniedException for the Firehose stream. |
| aws_firehose_KMSKeyDisabled | KMSKeyDisabled | KMS Key Disabled | Count | Average | The number of times the service encounters a KMSDisabledException for the Firehose stream. |
| aws_firehose_KMSKeyInvalidState | KMSKeyInvalidState | KMS Key Invalid State | Count | Average | The number of times the service encounters a KMSInvalidStateException for the Firehose stream. |
| aws_firehose_KMSKeyNotFound | KMSKeyNotFound | KMS Key Not Found | Count | Average | The number of times the service encounters a KMSNotFoundException for the Firehose stream. |