Amazon EMR is a managed cluster platform that simplifies running big data frameworks (such as Apache Hadoop and Apache Spark) on AWS to process and analyze vast amounts of data.
By using these frameworks and related open-source projects (such as Apache Hive and Apache Pig), you can:
- Process data for analytics purposes and business intelligence workloads.
- Use Amazon EMR to transform and move large amounts of data into and out of other AWS data stores and databases. For example, Amazon Simple Storage Service (Amazon S3) and Amazon DynamoDB.
Use the AWS public cloud integration to discover and collect metrics against the AWS service.
External reference
Setup
To set up the AWS integration and discover the AWS service, go to AWS Integration Discovery Profile and select EMR.
Event support
CloudTrail event support
- Supported
- Configurable in OpsRamp AWS Integration Discovery Profile.
CloudWatch alarm support
- Supported
- Configurable in OpsRamp AWS Integration Discovery Profile.
Supported metrics
| OpsRamp Metric | AWS Metric | Metric Display Name | Unit | Aggregation Type | Description |
|---|---|---|---|---|---|
| aws_elasticmapreduce_IsIdle | IsIdle | Is Idle | None | Average | Indicates that a cluster is no longer performing work, but is still alive and accruing charges. |
| aws_elasticmapreduce_ContainerAllocated | ContainerAllocated | Container Allocated | Count | Average | The number of resource containers allocated by the ResourceManager. |
| aws_elasticmapreduce_ContainerReserved | ContainerReserved | Container Reserved | Count | Average | The number of containers reserved. |
| aws_elasticmapreduce_ContainerPending | ContainerPending | Container Pending | Count | Average | The number of containers in the queue that have not yet been allocated. |
| aws_elasticmapreduce_ContainerPendingRatio | ContainerPendingRatio | Container Pending Ratio | Count | Average | The ratio of pending containers to containers allocated |
| aws_elasticmapreduce_AppsCompleted | AppsCompleted | Apps Completed | Count | Average | The number of applications submitted to YARN that have completed. |
| aws_elasticmapreduce_AppsFailed | AppsFailed | Apps Failed | Count | Average | The number of applications submitted to YARN that have failed to complete. |
| aws_elasticmapreduce_AppsKilled | AppsKilled | Apps Killed | Count | Average | The number of applications submitted to YARN that have been killed. |
| aws_elasticmapreduce_AppsPending | AppsPending | Apps Pending | Count | Average | The number of applications submitted to YARN that are in a pending state. |
| aws_elasticmapreduce_AppsRunning | AppsRunning | Apps Running | Count | Average | The number of applications submitted to YARN that are running. |
| aws_elasticmapreduce_AppsSubmitted | AppsSubmitted | Apps Submitted | Count | Average | The number of applications submitted to YARN. |
| aws_elasticmapreduce_CoreNodesRunning | CoreNodesRunning | Core Nodes Running | Count | Average | The number of core nodes working. Data points for this metric are reported only when a corresponding instance group exists. |
| aws_elasticmapreduce_CoreNodesPending | CoreNodesPending | Core Nodes Pending | Count | Average | The number of core nodes waiting to be assigned. |
| aws_elasticmapreduce_LiveDataNodes | LiveDataNodes | Live Data Nodes | Percent | Average | The percentage of data nodes that are receiving work from Hadoop. |
| aws_elasticmapreduce_MRTotalNodes | MRTotalNodes | MR Total Nodes | Count | Average | The number of nodes presently available to MapReduce jobs. |
| aws_elasticmapreduce_MRActiveNodes | MRActiveNodes | MR Active Nodes | Count | Average | The number of nodes presently running MapReduce tasks or jobs. |
| aws_elasticmapreduce_MRLostNodes | MRLostNodes | MR Lost Nodes | Count | Average | The number of nodes allocated to MapReduce that have been marked in a LOST state. |
| aws_elasticmapreduce_MRUnhealthyNodes | MRUnhealthyNodes | MR Unhealthy Nodes | Count | Average | The number of nodes available to MapReduce jobs marked in an UNHEALTHY state. |
| aws_elasticmapreduce_MRDecommissionedNodes | MRDecommissionedNodes | MR Decommissioned Nodes | Count | Average | The number of nodes allocated to MapReduce applications that have been marked in a DECOMMISSIONED state. |
| aws_elasticmapreduce_MRRebootedNodes | MRRebootedNodes | MR Rebooted Nodes | Count | Average | The number of nodes available to MapReduce that have been rebooted and marked in a REBOOTED state. |
| aws_elasticmapreduce_MultiMasterInstanceGroupNodesRunning | MultiMasterInstanceGroupNodesRunning | Multi Master Instance Group Nodes Running | Count | Average | The number of running master nodes. |
| aws_elasticmapreduce_MultiMasterInstanceGroupNodesRunningPercentage | MultiMasterInstanceGroupNodesRunningPercentage | Multi Master Instance Group Nodes Running Percentage | Count | Average | The percentage of master nodes that are running over the requested master node instance count. |
| aws_elasticmapreduce_MultiMasterInstanceGroupNodesRequested | MultiMasterInstanceGroupNodesRequested | Multi Master Instance Group Nodes Requested | Count | Average | The number of requested master nodes. |
| aws_elasticmapreduce_S3BytesWritten | S3BytesWritten | S3Bytes Written | Count | Average | The number of bytes written to Amazon S3. This metric aggregates MapReduce jobs only, and does not apply for other workloads on Amazon EMR. |
| aws_elasticmapreduce_S3BytesRead | S3BytesRead | S3Bytes Read | Count | Average | The number of bytes read from Amazon S3. This metric aggregates MapReduce jobs only, and does not apply for other workloads on Amazon EMR. |
| aws_elasticmapreduce_HDFSUtilization | HDFSUtilization | HDFS Utilization | Percent | Average | The percentage of HDFS storage currently used. |
| aws_elasticmapreduce_HDFSBytesRead | HDFSBytesRead | HDFS Bytes Read | Count | Average | The number of bytes read from HDFS. This metric aggregates MapReduce jobs only, and does not apply for other workloads on Amazon EMR. |
| aws_elasticmapreduce_HDFSBytesWritten | HDFSBytesWritten | HDFS Bytes Written | Count | Average | The number of bytes written to HDFS. This metric aggregates MapReduce jobs only, and does not apply for other workloads on Amazon EMR. |
| aws_elasticmapreduce_MissingBlocks | MissingBlocks | Missing Blocks | Count | Average | The number of blocks in which HDFS has no replicas. These might be corrupt blocks. |
| aws_elasticmapreduce_CorruptBlocks | CorruptBlocks | Corrupt Blocks | Count | Average | The number of blocks that HDFS reports as corrupted. |
| aws_elasticmapreduce_TotalLoad | TotalLoad | Total Load | Count | Average | The total number of concurrent data transfers. |
| aws_elasticmapreduce_MemoryTotalMB | MemoryTotalMB | Memory Total MB | Count | Average | The total amount of memory in the cluster. |
| aws_elasticmapreduce_MemoryReservedMB | MemoryReservedMB | Memory Reserved MB | Count | Average | The amount of memory reserved. |
| aws_elasticmapreduce_MemoryAvailableMB | MemoryAvailableMB | Memory Available MB | Count | Average | The amount of memory available to be allocated. |
| aws_elasticmapreduce_YARNMemoryAvailablePercentage | YARNMemoryAvailablePercentage | YARN Memory Available Percentage | Percent | Average | The percentage of remaining memory available to YARN |
| aws_elasticmapreduce_MemoryAllocatedMB | MemoryAllocatedMB | Memory Allocated MB | Count | Average | The amount of memory allocated to the cluster. |
| aws_elasticmapreduce_PendingDeletionBlocks | PendingDeletionBlocks | Pending Deletion Blocks | Count | Average | The number of blocks marked for deletion. |
| aws_elasticmapreduce_UnderReplicatedBlocks | UnderReplicatedBlocks | Under Replicated Blocks | Count | Average | The number of blocks that need to be replicated one or more times. |
| aws_elasticmapreduce_dfs_FSNamesystem_PendingReplicationBlocks | DfsPendingReplicationBlocks | Dfs Pending Replication Blocks | Count | Average | The status of block replication: blocks being replicated, age of replication requests, and unsuccessful replication requests. |
| aws_elasticmapreduce_CapacityRemainingGB | CapacityRemainingGB | Capacity Remaining GB | Count | Average | The amount of remaining HDFS disk capacity. |