Documentation is now available for the Fall 2020 Update release!

AWS Elastic MapReduce

Leave Feedback

Introduction

Amazon EMR is a managed cluster platform that simplifies running big data frameworks (such as Apache Hadoop and Apache Spark) on AWS to process and analyze vast amounts of data.

By using these frameworks and related open-source projects (such as Apache Hive and Apache Pig), you can:

  • Process data for analytics purposes and business intelligence workloads.
  • Use Amazon EMR to transform and move large amounts of data into and out of other AWS data stores and databases. For example, Amazon Simple Storage Service (Amazon S3) and Amazon DynamoDB.

Setup

To set up the OpsRamp AWS integration and discover the AWS service, go to AWS Integration Discovery Profile and select EMR.

Metrics

OpsRamp MetricMetric Display NameUnitAggregation TypeDescription
aws_elasticmapreduce_IsIdleIsIdleCountAVERAGEIndicates that a cluster is no longer performing work, but is still alive and accruing charges. Set to 1 if no tasks and jobs are running; set to 0 otherwise.
aws_elasticmapreduce_ContainerAllocatedContainerAllocatedCountAVERAGENumber of resource containers allocated by the ResourceManager.
aws_elasticmapreduce_ContainerReservedContainerReservedCountAVERAGENumber of containers reserved.
aws_elasticmapreduce_ContainerPendingContainerPendingCountAVERAGENumber of containers in the queue that have not yet been allocated.
aws_elasticmapreduce_AppsCompletedAppsCompletedCountAVERAGENumber of applications submitted to YARN (Hadoop generation)) that have completed.
aws_elasticmapreduce_AppsKilledAppsKilledCountAVERAGENumber of applications submitted to YARN (Hadoop generation)) that have been killed.
aws_elasticmapreduce_AppsPendingAppsPendingCountAVERAGENumber of applications submitted to YARN (Hadoop generation) that are in a pending state.
aws_elasticmapreduce_AppsRunningAppsRunningCountAVERAGENumber of applications submitted to YARN (Hadoop generation) that are running.
aws_elasticmapreduce_AppsSubmittedAppsSubmittedCountAVERAGENumber of applications submitted to YARN (Hadoop generation).
aws_elasticmapreduce_CapacityRemainingGBCapacityRemainingGBBytesAVERAGEAmount of remaining HDFS disk capacity.
aws_elasticmapreduce_CoreNodesRunningCoreNodesRunningCountAVERAGENumber of core nodes working. Data points for this metric are reported only when a corresponding instance group exists.
aws_elasticmapreduce_CoreNodesPendingCoreNodesPendingCountAVERAGENumber of core nodes waiting to be assigned. All of the core nodes requested may not be immediately available; this metric reports the pending requests.
aws_elasticmapreduce_CorruptBlocksCorruptBlocksCountAVERAGEGives the big picture about what is going on with cluster and can provide insight into what is causing the slow down in processing.
aws_elasticmapreduce_HDFSUtilizationHDFSUtilizationPercentAVERAGEPercentage of HDFS storage currently used.
aws_elasticmapreduce_HDFSBytesReadHDFSBytesReadBytes ReadAVERAGENumber of bytes read from HDFS.
aws_elasticmapreduce_HDFSBytesWrittenHDFSBytesWrittenBytes WrittenAVERAGENumber of bytes written to HDFS.
aws_elasticmapreduce_LiveDataNodesLiveDataNodesPercentAVERAGEPercentage of data nodes that are receiving work from Hadoop.
aws_elasticmapreduce_MRTotalNodesMRTotalNodesCountAVERAGENumber of nodes presently available to MapReduce jobs.
aws_elasticmapreduce_MRActiveNodesMRActiveNodesCountAVERAGENumber of nodes presently running MapReduce tasks or jobs.
aws_elasticmapreduce_MRLostNodesMRLostNodesCountAVERAGENumber of nodes allocated to MapReduce that have been marked in a LOST state.
aws_elasticmapreduce_MRUnhealthyNodesMRUnhealthyNodesAVERAGENumber of nodes available to MapReduce jobs marked in an UNHEALTHY state.
aws_elasticmapreduce_MRDecommissionedNodesMRDecommissionedNodesCountAVERAGENumber of nodes allocated to MapReduce applications that have been marked in a DECOMMISSIONED state.
aws_elasticmapreduce_MRRebootedNodesMRRebootedNodesCountAVERAGENumber of nodes available to MapReduce that have been rebooted and marked in a REBOOTED state.
aws_elasticmapreduce_S3BytesWrittenS3BytesWrittenBytes WrittenAVERAGENumber of bytes written to Amazon S3.
aws_elasticmapreduce_S3BytesReadS3BytesReadBytes ReadAVERAGENumber of bytes read from Amazon S3.
aws_elasticmapreduce_MissingBlocksMissingBlocksCountAVERAGENumber of blocks in which HDFS has no replicas. These might be corrupt blocks.
aws_elasticmapreduce_TotalLoadTotalLoadCountAVERAGETotal number of concurrent data transfers.
aws_elasticmapreduce_MemoryTotalMBMemoryTotalMBBytesAVERAGETotal amount of memory in the cluster.
aws_elasticmapreduce_MemoryReservedMBMemoryReservedMBBytesAVERAGEAmount of memory reserved.
aws_elasticmapreduce_MemoryAvailableMBMemoryAvailableMBBytesAVERAGEAmount of memory available to be allocated.
aws_elasticmapreduce_MemoryAllocatedMBMemoryAllocatedMBBytesAVERAGEAmount of memory allocated to the cluster.
aws_elasticmapreduce_PendingDeletionBlocksPendingDeletionBlocksCountAVERAGENumber of blocks marked for deletion.
aws_elasticmapreduce_UnderReplicatedBlocksUnderReplicatedBlocksCountAVERAGENumber of blocks that need to be replicated one or more times.
aws_elasticmapreduce_dfs_FSNamesystem_PendingReplicationBlocksdfs.FSNamesystem.PendingReplicationBlocksCountAVERAGEStatus of block replication: blocks being replicated, age of replication requests, and unsuccessful replication requests.
aws_elasticmapreduce_ContainerPendingRatioContainer Pending RatioCountAverageRatio of pending containers to containers allocated (ContainerPendingRatio = ContainerPending / ContainerAllocated). If ContainerAllocated = 0, then ContainerPendingRatio = ContainerPending. The value of ContainerPendingRatio represents a number, not a percentage. This value is useful for scaling cluster resources based on container allocation behavior.
aws_elasticmapreduce_AppsFailedApps FailedCountAverageNumber of applications submitted to YARN that have failed to complete.
aws_elasticmapreduce_YARNMemoryAvailablePercentageYARN Memory Available PercentagePercentAveragePercentage of remaining memory available to YARN (YARNMemoryAvailablePercentage = MemoryAvailableMB / MemoryTotalMB). This value is useful for scaling cluster resources based on YARN memory usage.
cloud.instance.stateStatus/Staten/an/an/a

Event support

CloudTrail event support

  • Supported
  • Configurable in OpsRamp AWS Integration Discovery Profile.

CloudWatch alarm support

  • Supported
  • Configurable in OpsRamp AWS Integration Discovery Profile.

External reference