Prometheus Remote Write Integration

OpsRamp allows you to ship the Prometheus (OpenMetricFormat) metrics to the SaaS platform with the use of Prometheus Remote Write integration.

Prerequisites

An environment instrumented with Prometheus that has access to the OpsRamp portal.

Add a Prometheus Remote Write integration

To install Prometheus Remote Write integration:

Click All Clients and from the drop-down list, select a client.
Go to Setup > Integrations and Apps.
From Available Integrations, click Monitoring and search for Prometheus Remote Write.

Click Install on the PROMETHEUS REMOTE WRITE INTEGRATION page.
The Install Prometheus Remote Write Integration page is displayed.
Enter a Name for the Prometheus Remote Write integration. You can also upload a logo and click Install.
The Prometheus Remote Write integration is installed.
Follow below given on screen instructions displayed after installing the integration:

Constraints

The number of active series per metric per client is 50000. You can avoid the limit by configuring Prometheus to filter metrics. For example, use the following configuration to limit apiserver_request_duration_seconds_bucket and etcd_request_duration_seconds_bucket metrics:

remoteWrite:
  - url: http://portal.opsramp.net/prom/push
    authorization:
      type: Bearer
      credentials: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    headers:
      tenantId: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
      integrationUUID: XXXXXXXXXXXXXXXXXXXXXXXXXXXX-XXXXXXXXXXXX
    writeRelabelConfigs:
    - action: drop
      regex: apiserver_request_duration_seconds_bucket
      sourceLabels:
      - __name__
    - action: drop
      regex: etcd_request_duration_seconds_bucket
      sourceLabels:
      - __name__

Configuration

Remote Write Tuning

Below are the recommended Prometheus Remote-Write Configuration tuning:

Configuration Parameter	Description
capacity	How many samples are queued in memory per shard before blocking reading from the WAL. When the WAL is blocked, samples cannot be appended to any shards and all throughput ceases.
max_shards	The maximum number of shards, or parallelism, Prometheus uses for each remote write queue.
min_shards	The minimum number of shards used by Prometheus, which is the number of shards used when remote write starts.
max_samples_per_send	The maximum number of samples sent per batch, which can be adjusted depending on the back end in use.
batch_send_deadline	The maximum time interval between sends for a single shard, in seconds.
min_backoff	The minimum time to wait before retrying a failed request, in seconds.
max_backoff	The maximum time to wait before retrying a failed request, in seconds.

See: https://prometheus.io/docs/practices/remote_write/ for more details.

Configuration examples to filter metrics pushed to remote write

Prometheus can be configured to filter the metrics to be pushed to a remote endpoint like the agent. See the Prometheus configuration remote_write property documentation for more information.

Drop samples example

The following writeRelabelConfigs configuration drops samples with a metric name that starts with go:

remote_write:
  - url: http://portal.opsramp.net/prom/push
    authorization:
      type: Bearer
      credentials: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    headers:
      tenantId: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
      integrationUUID: XXXXXXXXXXXXXXXXXXXXXXXXXXXX-XXXXXXXXXXXX
    writeRelabelConfigs:
    - sourceLabels: [__name__]
      regex: 'go_.*'
      action: drop

Keep samples example

The following writeRelabelConfigs configuration keeps samples with a metric name that starts with go:

remote_write:
  - url: http://portal.opsramp.net/prom/push
    authorization:
      type: Bearer
      credentials: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    headers:
      tenantId: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
      integrationUUID: XXXXXXXXXXXXXXXXXXXXXXXXXXXX-XXXXXXXXXXXX
    writeRelabelConfigs:
      - sourceLabels: [__name__]
        regex: 'go_.*'
        action: keep

Replace samples example

You can also change a label using the replace action before pushing it to a remote endpoint. The following example replaces samples with a metric name that starts with go with the name sample_go_label:

remote_write:
  - url: http://portal.opsramp.net/prom/push
    authorization:
      type: Bearer
      credentials: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    headers:
      tenantId: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
      integrationUUID: XXXXXXXXXXXXXXXXXXXXXXXXXXXX-XXXXXXXXXXXX
    writeRelabelConfigs:
    - sourceLabels: [__name__]
      regex: 'go_.*'
      action: replace
      targetLabels: sample_go_label