AWS Glue is a fully managed ETL (extract, transform, and load) service that makes it simple and cost-effective to categorize your data, clean it, enrich it, and move it reliably between various data stores.
AWS Glue consists of a central metadata repository known as the AWS Glue Data Catalog, an ETL engine that automatically generates Python or Scala code, and a flexible scheduler that handles dependency resolution, job monitoring, and retries. AWS Glue is serverless, so there is no infrastructure to set up or manage.
AWS Glue is designed to work with semi-structured data. It introduces a component called a dynamic frame, which you can use in your ETL scripts. A dynamic frame is similar to an Apache Spark dataframe, except that each record is self-describing, so no schema is required initially. With dynamic frames, you get schema flexibility and a set of advanced transformations specifically designed for dynamic frames.
You can convert between dynamic frames and Spark dataframes to take advantage of both AWS Glue and Spark transformations to do the kinds of analysis that you want.
External reference
Setup
To set up the AWS integration and discover the AWS service, go to AWS Integration Discovery Profile and select GLUE. AWS Glue databases, tables, crawlers, jobs, DevEndpoints, and MLTransforms are discovered.
Event support
CloudTrail event support
- Supported
- Configurable in OpsRamp AWS Integration Discovery Profile.
CloudWatch alarm support
- Supported
- Configurable in OpsRamp AWS Integration Discovery Profile.
Supported metrics
| OpsRamp Metric | Metric Display Name | Unit | Aggregation Type |
|---|---|---|---|
| aws_glue_glue_jvm_heap_usage Number of memory bytes used by the JVM heap for the driver, the executor identified by executorId, or ALL executors. | glue jvm heap usage | None | Average |
| aws_glue_glue_jvm_heap_used Number of memory bytes used by the JVM heap for the driver, the executor identified by executorId, or ALL executors. | glue jvm heap used | None | Average |
| aws_glue_glue_s3_filesystem_read_bytes Number of bytes read from Amazon S3 by the driver, an executor identified by executorId, or ALL executors since the previous report (aggregated by the AWS Glue Metrics Dashboard as the number of bytes read during the previous minute). | glue s3 file system read bytes | Count | Average |
| aws_glue_glue_s3_filesystem_write_bytes Number of bytes written to Amazon S3 by the driver, an executor identified by executorId, or ALL executors since the previous report (aggregated by the AWS Glue Metrics Dashboard as the number of bytes written during the previous minute). | glue s3 filesystem write bytes | Count | Average |
| aws_glue_glue_system_cpuSystemLoad The fraction of CPU system load used (scale: 0-1) by the driver, an executor identified by executorId, or ALL executors. | glue system cpu System Load | None | Average |