AWS Glue is a fully managed ETL (extract, transform, and load) service that makes it simple and cost-effective to categorize your data, clean it, enrich it, and move it reliably between various data stores.
Glue Jobs Resource Information:
- Type of Resource: Generic monitorable resource
- Discovery type: AWS SDK discovery type
- Discovery profile selection: Resource can be discovered by “Glue Jobs” in the profiler
- ResourceTag: AWS_GLUE_JOBS
- Resource Unique Identification: JobName+"/"+region (job.name() + “/” + region)
- Dependencies: Glue Jobs may depend on Glue databases, tables, and connections for ETL operations
Glue ML Transform Resource Information:
- Type of Resource: Generic non-monitorable resource
- Discovery type: AWS SDK discovery type
- Discovery profile selection: Resource can be discovered by “Glue ML Transform” in the profiler
- ResourceTag: AWS_GLUE_ML_TRANSFORM
- Resource Unique Identification: TransformId (ml.transformId())
- Dependencies: Glue ML Transform may depend on Glue databases and tables for machine learning operations
Glue Crawlers Resource Information:
- Type of Resource: Generic non-monitorable resource
- Discovery type: AWS SDK discovery type
- Discovery profile selection: Resource can be discovered by “Glue Crawlers” in the profiler
- ResourceTag: AWS_GLUE_CRAWLERS
- Resource Unique Identification: crawlerName+"/"+region (crawler.name() + “/” + region)
- Dependencies: Glue Crawlers may depend on data stores like S3, databases, and Glue databases
Glue Tables Resource Information:
- Type of Resource: Generic non-monitorable resource
- Discovery type: AWS SDK discovery type
- Discovery profile selection: Resource can be discovered by “Glue Tables” in the profiler
- ResourceTag: AWS_GLUE_TABLES
- Resource Unique Identification: tableDBName+"/"+tableName+"/"+region (table.databaseName() + “/” + table.name() + “/” + region)
- Dependencies: Glue Tables run on Glue databases and may depend on data sources
Glue Databases Resource Information:
- Type of Resource: Generic non-monitorable resource
- Discovery type: AWS SDK discovery type
- Discovery profile selection: Resource can be discovered by “Glue Databases” in the profiler
- ResourceTag: AWS_GLUE_DB
- Resource Unique Identification: DBName+"/"+region (db.name() + “/” + region)
- Dependencies: Glue Databases may contain tables and be accessed by Glue crawlers and jobs
Glue Dev Endpoints Resource Information:
- Type of Resource: Generic non-monitorable resource
- Discovery type: AWS SDK discovery type
- Discovery profile selection: Resource can be discovered by “Glue Dev Endpoints” in the profiler
- ResourceTag: AWS_GLUE_DEV_ENDPOINTS
- Resource Unique Identification: endpointName+"/"+region (endpoint.endpointName() + “/” + region)
- Dependencies: Glue Dev Endpoints may depend on VPC configurations and security groups
AWS Glue consists of a central metadata repository known as the AWS Glue Data Catalog, an ETL engine that automatically generates Python or Scala code, and a flexible scheduler that handles dependency resolution, job monitoring, and retries. AWS Glue is serverless, so there is no infrastructure to set up or manage.
AWS Glue is designed to work with semi-structured data. It introduces a component called a dynamic frame, which you can use in your ETL scripts. A dynamic frame is similar to an Apache Spark dataframe, except that each record is self-describing, so no schema is required initially. With dynamic frames, you get schema flexibility and a set of advanced transformations specifically designed for dynamic frames.
You can convert between dynamic frames and Spark dataframes to take advantage of both AWS Glue and Spark transformations to do the kinds of analysis that you want.
External reference
Setup
To set up the AWS integration and discover the AWS service, go to AWS Integration Discovery Profile and select GLUE. AWS Glue databases, tables, crawlers, jobs, DevEndpoints, and MLTransforms are discovered.
Event support
CloudTrail event support
- Supported
- Configurable in OpsRamp AWS Integration Discovery Profile.
CloudWatch alarm support
- Supported
- Configurable in OpsRamp AWS Integration Discovery Profile.
Supported metrics
| OpsRamp Metric | AWS Metric | Metric Display Name | Unit | Aggregation Type | Description |
|---|---|---|---|---|---|
| aws_glue_glue_driver_aggregate_bytesRead | glue.driver.aggregate.bytesRead | glue driver aggregate bytes Read | Bytes | Sum | The number of bytes read from all data sources by all completed Spark tasks running in all executors |
| aws_glue_glue_driver_aggregate_elapsedTime | glue.driver.aggregate.elapsedTime | glue driver aggregate elapsed Time | Milliseconds | Sum | The ETL elapsed time in milliseconds (does not include the job bootstrap times) |
| aws_glue_glue_driver_aggregate_numCompletedStages | glue.driver.aggregate.numCompletedStages | glue driver aggregate num Completed Stages | Count | Sum | The number of completed stages in the job |
| aws_glue_glue_driver_aggregate_numCompletedTasks | glue.driver.aggregate.numCompletedTasks | glue driver aggregate num Completed Tasks | Count | Sum | The number of completed tasks in the job |
| aws_glue_glue_driver_aggregate_numFailedTasks | glue.driver.aggregate.numFailedTasks | glue driver aggregate num Failed Tasks | Count | Sum | The number of failed tasks |
| aws_glue_glue_driver_aggregate_numKilledTasks | glue.driver.aggregate.numKilledTasks | glue driver aggregate num Killed Tasks | Count | Sum | The number of tasks killed |
| aws_glue_glue_driver_aggregate_recordsRead | glue.driver.aggregate.recordsRead | glue driver aggregate records Read | Count | Sum | The number of records read from all data sources by all completed Spark tasks running in all executors |
| aws_glue_glue_driver_aggregate_shuffleBytesWritten | glue.driver.aggregate.shuffleBytesWritten | glue driver aggregate shuffle Bytes Written | Bytes | Sum | The number of bytes written by all executors to shuffle data between them since the previous report |
| aws_glue_glue_driver_aggregate_shuffleLocalBytesRead | glue.driver.aggregate.shuffleLocalBytesRead | glue driver aggregate shuffle Local Bytes Read | Bytes | Sum | The number of bytes read by all executors to shuffle data between them since the previous report |
| aws_glue_glue_driver_BlockManager_disk_diskSpaceUsed_MB | glue.driver.BlockManager.disk.diskSpaceUsed_MB | glue driver Block Manager disk disk Space Used_MB | Megabytes | Average | The number of megabytes of disk space used across all executors |
| aws_glue_glue_driver_ExecutorAllocationManager_executors_numberAllExecutors | glue.driver.ExecutorAllocationManager.executors.numberAllExecutors | glue driver Executor Allocation Manager executors number All Executors | Count | Average | The number of actively running job executors |
| aws_glue_glue_driver_ExecutorAllocationManager_executors_numberMaxNeededExecutors | glue.driver.ExecutorAllocationManager.executors.numberMaxNeededExecutors | glue driver Executor Allocation Manager executors number Max Needed Executors | Count | Maximum | The number of maximum (actively running and pending) job executors needed to satisfy the current load |
| aws_glue_glue_driver_jvm_heap_usage | glue.driver.jvm.heap.usage | glue driver jvm heap usage | Percent | Average | The fraction of memory used by the JVM heap for the driver (also available as glue.ALL.jvm.heap.usage for all executors) |
| aws_glue_glue_driver_jvm_heap_used | glue.driver.jvm.heap.used | glue driver jvm heap used | Bytes | Average | The number of memory bytes used by the JVM heap for the driver (also available as glue.ALL.jvm.heap.used for all executors) |
| aws_glue_glue_driver_s3_filesystem_read_bytes | glue.driver.s3.filesystem.read_bytes | glue driver s3 filesystem read_bytes | Bytes | Sum | The number of bytes read from Amazon S3 by the driver (also available as glue.ALL.s3.filesystem.read_bytes for all executors) |
| aws_glue_glue_driver_s3_filesystem_write_bytes | glue.driver.s3.filesystem.write_bytes | glue driver s3 filesystem write_bytes | Bytes | Sum | The number of bytes written to Amazon S3 by the driver (also available as glue.ALL.s3.filesystem.write_bytes for all executors) |
| aws_glue_glue_driver_system_cpuSystemLoad | glue.driver.system.cpuSystemLoad | glue driver system cpu System Load | Percent | Average | The fraction of CPU system load used by the driver (also available as glue.ALL.system.cpuSystemLoad for all executors) |
| aws_glue_glue_driver_streaming_numRecords | glue.driver.streaming.numRecords | glue driver streaming num Records | Count | Average | The number of records received in a micro-batch (streaming jobs only, Glue version 2.0+) |
| aws_glue_glue_driver_streaming_batchProcessingTimeInMs | glue.driver.streaming.batchProcessingTimeInMs | glue driver streaming batch Processing Time In Ms | Milliseconds | Average | The time it takes to process the batches in milliseconds (streaming jobs only, Glue version 2.0+) |