Cloud Dataflow is a fully-managed service for transforming and enriching data in stream (real-time) and batch (historical) modes with equal reliability and expressiveness – no more complex workarounds or compromises needed. With its serverless approach to resource provisioning and management, you have access to virtually limitless capacity to solve your biggest data processing challenges, while paying only for what you use.

Cloud Dataflow unlocks transformational use cases across industries, including:

  • Check Clickstream, Point-of-Sale, and segmentation analysis in retail.
  • Check Fraud detection in financial services.
  • Check Personalized user experience in gaming.
  • Check IoT analytics in manufacturing, healthcare, and logistics.

Setup

To set up the OpsRamp Google integration and discover the Google service, go to Google Integration Discovery Profile and select GOOGLE/Dataflow Job.

Metrics

OpsRamp MetricMetric Display NameUnitAggregation Type
google_dataflow_job_current_num_vcpus

Number of vCPUs currently being used by this Dataflow job. This is the current number of workers times the number of vCPUs per worker.
Current number of vCPUs in useCountAverage
google_dataflow_job_data_watermark_age

Age (time since event timestamp) of the most recent item of data that has been fully processed by the pipeline.
Data watermark ageSecondsAverage
google_dataflow_job_elapsed_time

Duration that the current run of this pipeline has been in the Running state so far, in seconds. When a run completes, this stays at the duration of that run until the next run starts.
Elapsed timeSecondsAverage
google_dataflow_job_element_count

Number of elements added to the pcollection so far.
Element countCountAverage
google_dataflow_job_error_count

Number of errors that happened so far.
Error countCountAverage
google_dataflow_job_estimated_byte_count

Estimated number of bytes added to the pcollection so far. Dataflow calculates the average encoded size of elements in a pcollection and mutiplies it by the number of elements.
Estimated byte countBytesAverage
google_dataflow_job_is_failed

Has this job failed.
FailedCountAverage
google_dataflow_job_status

Current state of this pipeline (for example: RUNNING, DONE, CANCELLED, FAILED, ...). Not reported while the pipeline is not running.
StatusStringAverage
google_dataflow_job_system_lag

Current maximum duration that an item of data has been awaiting processing, in seconds.
System lagSecondsAverage
google_dataflow_job_total_memory_usage_time

Total GB seconds of memory allocated to this Dataflow job.
Total memory usage timeGB.secondsAverage
google_dataflow_job_total_pd_usage_time

Total GB seconds for all persistent disk used by all workers associated with this Dataflow job.
Total PD usage timeGB.secondsAverage
google_dataflow_job_total_vcpu_time

Total vCPU seconds used by this Dataflow job.
Total vCPU timeSecondsAverage
google_dataflow_job_billable_shuffle_data_processed

Billable bytes of shuffle data processed by this Dataflow job, in bytes Sampled every 60 seconds.
Job Billable Shuffle Data ProcessedBytesAverage
google_dataflow_job_total_shuffle_data_processed

Total bytes of shuffle data processed by this Dataflow job.
Job Total Shuffle Data ProcessedBytesAverage
google_dataflow_job_total_streaming_data_processed

Total bytes of shuffle data processed by this Dataflow job.
Job Total Streaming Data ProcessedBytesAverage
google_dataflow_job_user_counter

A user-defined counter metric.
Job User CounterCountAverage
google_dataflow_job_current_shuffle_slots

Current shuffle slots used by this Dataflow job. Sampled every 60 seconds. After sampling, data is not visible for up to 180 seconds.
Current shuffle slots in useCountAverage
google_dataflow_job_elements_produced_count

Number of elements produced by each PTransform. Sampled every 60 seconds. After sampling, data is not visible for up to 180 seconds.
Elements ProducedCountAverage
google_dataflow_job_estimated_bytes_produced_count

Estimated total byte size of elements produced by each PTransform. Sampled every 60 seconds. After sampling, data is not visible for up to 180 seconds.
Estimated Bytes ProducedCountAverage

Event support

  • Supported
  • Configurable in OpsRamp Google Integration Discovery Profile.

External reference