Spark - Statistics
Description
It monitors the spark job,stage,RDD and executor parameters.Job parameters : Spark Job Num Tasks, Spark Job Num Active Tasks, Spark Job Num Skipped Tasks, Spark Job Num Failed Tasks, Spark Job Num Completed Tasks, Spark Job Num Active Stages, Spark Job Num Completed Stages, Spark Job Num Skipped Stages, Spark Job Num Failed Stages.Stage parameters : Spark Stage Num Active Tasks, Spark Stage Num Complete Tasks, Spark Stage Num Failed Tasks, Spark Stage Executor Run Time, Spark Stage Input Bytes, Spark Stage Input Records, Spark Stage Output Bytes, Spark Stage Output Records, Spark Stage Shuffle Read Bytes, Spark Stage Shuffle Read Records, Spark Stage Shuffle Write Bytes, Spark Stage Shuffle Write Records, Spark Stage Memory Bytes Spilled.RDD parameters : Spark Rdd Memory Used, Spark Rdd Disk Used.Executor parameters : Spark Executor Memory Used, Spark Executor Disk Used, Spark Executor Active Tasks, Spark Executor Failed Tasks, Spark Executor Completed Tasks, Spark Executor Total Tasks, Spark Executor Total Duration, Spark Executor Total Input Bytes, Spark Executor Total Shuffle Read, Spark Executor Total Shuffle Write. Validated on Spark v2.3.1
Prerequisites
No prerequisite
Supported Metric
| Monitor Name | Metric Name | Metric Display Name | Unit | Description |
|---|---|---|---|---|
| Spark - Executor Statistics | spark.executor.active_tasks | Spark Executor Active Tasks | NULL | Number of active tasks in the application's executors |
| spark.executor.total_shuffle_write | Spark Executor Total Shuffle Write | NULL | Total number of shuffled bytes in the application's executors | |
| spark.executor.failed_tasks | Spark Executor Failed Tasks | NULL | Number of failed tasks in the application's executors | |
| spark.executor.total_shuffle_read | Spark Executor Total Shuffle Read | NULL | Total number of bytes read during a shuffle in the application's executors | |
| spark.executor.completed_tasks | Spark Executor Completed Tasks | NULL | Number of completed tasks in the application's executors | |
| spark.executor.memory_used | Spark Executor Memory Used | NULL | Amount of memory used for cached RDDs in the application's executors | |
| spark.executor.total_tasks | Spark Executor Total Tasks | NULL | Total number of tasks in the application's executors | |
| spark.executor.total_duration | Spark Executor Total Duration | NULL | Fraction of time (ms/s) spent by the application's executors executing tasks | |
| spark.executor.disk_used | Spark Executor Disk Used | NULL | Amount of disk space used by persisted RDDs in the application's executors | |
| spark.executor.total_input_bytes | Spark Executor Total Input Bytes | NULL | Total number of input bytes in the application's executors | |
| Spark - Job Statistics | spark.job.num_skipped_stages | Spark Job Num Skipped Stages | NULL | Number of skipped stages in the application |
| spark.job.num_failed_tasks | Spark Job Num Failed Tasks | NULL | Number of failed tasks in the application | |
| spark.job.num_completed_stages | Spark Job Num Completed Stages | NULL | Number of completed stages in the application | |
| spark.job.num_active_stages | Spark Job Num Active Stages | NULL | Number of active stages in the application | |
| spark.job.num_failed_stages | Spark Job Num Failed Stages | NULL | Number of failed stages in the application | |
| spark.job.num_skipped_tasks | Spark Job Num Skipped Tasks | NULL | Number of skipped tasks in the application | |
| spark.job.num_tasks | Spark Job Num Tasks | NULL | Number of tasks in the application | |
| spark.job.num_active_tasks | Spark Job Num Active Tasks | NULL | Number of active tasks in the application | |
| spark.job.num_completed_tasks | Spark Job Num Completed Tasks | NULL | Number of completed tasks in the application | |
| Spark - RDD Statistics | spark.rdd.memory_used | Spark Rdd Memory Used | NULL | Amount of memory used in the application's persisted RDDs |
| spark.rdd.disk_used | Spark Rdd Disk Used | NULL | Amount of disk space used by persisted RDDs in the application | |
| Spark - Stage Statistics | spark.stage.shuffle_read_records | Spark Stage Shuffle Read Records | NULL | Number of records read during a shuffle in the application's stages |
| spark.stage.shuffle_write_records | Spark Stage Shuffle Write Records | NULL | Number of shuffled records in the application's stages | |
| spark.stage.input_bytes | Spark Stage Input Bytes | NULL | Input bytes in the application's stages | |
| spark.stage.shuffle_read_bytes | Spark Stage Shuffle Read Bytes | NULL | Number of bytes read during a shuffle in the application's stages | |
| spark.stage.shuffle_write_bytes | Spark Stage Shuffle Write Bytes | NULL | Number of shuffled bytes in the application's stages | |
| spark.stage.num_active_tasks | Spark Stage Num Active Tasks | NULL | Number of active tasks in the application's stages | |
| spark.stage.output_records | Spark Stage Output Records | NULL | Output records in the application's stages | |
| spark.stage.memory_bytes_spilled | Spark Stage Memory Bytes Spilled | NULL | Number of bytes spilled to disk in the application's stages | |
| spark.stage.num_complete_tasks | Spark Stage Num Complete Tasks | NULL | Number of complete tasks in the application's stages | |
| spark.stage.input_records | Spark Stage Input Records | NULL | Input records in the application's stages | |
| spark.stage.output_bytes | Spark Stage Output Bytes | NULL | Output bytes in the application's stages | |
| spark.stage.num_failed_tasks | Spark Stage Num Failed Tasks | NULL | Number of failed tasks in the application's stages | |
| spark.stage.executor_run_time | Spark Stage Executor Run Time | NULL | Fraction of time (ms/s) spent by the executor in the application's stages |