Documentation is now available for the Fall 2020 Update release!

Prometheus

Describes how to set up an integration to ingest Prometheus monitoring events.

Leave Feedback

Introduction

Prometheus is an open-source software application used for event monitoring and alerting.

Validated Version: Prometheus 2.14.0

OpsRamp configuration

Configuration involves:

  1. Installing the integration.
  2. Configuring the integration.

Step 1: Install the integration

To install:

  1. From All Clients, select a client.
  2. Go to Setup > Integrations > Integrations.
  3. From Available Integrations, select Monitoring > Prometheus.
  4. Click Install.

Step 2: Configure the integration

To configure the integration:

  1. From the API > Authentication section, copy the following:
    • Tenant Id
    • Token
    • Webhook URL
      Note: These settings are used to create an HTTP Request template.
  2. From the API > Map Attributes section, configure the mapping attributes.
    Note: These parameters are used for the third-party’s software configuration.

Configuring the map attributes

To configure the mapping attributes:

  1. In OpsRamp Entity, specify Alert.
  2. In OpsRamp Property, select the property from the drop-down.
  3. Click + to define the mappings.
  4. From Create Alert Mappings on Status, define the mappings, parsing conditions, and default values.
  5. Click Save.

The following table shows attribute mappings:

Property Mappings
Third-Party EntityOpsRamp EntityThird-Party PropertyOpsRamp PropertyThird-Party Property ValueOpsRamp Property Value
AlertALERTseverityalert.currentStatecriticalCritical
AlertALERTmetricalert.serviceNamecontainer_memory_usage_bytesNA
AlertALERTdescriptionalert.descriptiontesting alert1NA
AlertALERTsummaryalert.subjectHigh Memory UsageNA

The following mappings examples are based on a custom label and values and reflect the Prometheus Alert Browse configuration examples.

  • severity: critical
  • metric: container_memory_usage_bytes
  • description: testing alert1
  • annotations
  • summary: High Memory Usage

Prometheus configuration

Routing the Prometheus alerts to OpsRamp is configured via the YAML definition used during deployment. Based on the requirement, you can configure Prometheus using either of the following scenarios:

  • Configure without defining a label for the app and forward all generated alerts of Prometheus to OpsRamp.
  • Configure with a label defined for the app and forward selective generated alerts to OpsRamp.

Configuring without defining App label

Configuration without defining App label involves configuring only Prometheus Alert Manager. OpsRamp webhook becomes the default receiver in the Alert Manager configuration. A label is not defined and all alerts with severity level of error/warning that Prometheus generates are forwarded to OpsrRamp. As a result, configuring alert rules is not required.

Configure Prometheus Alert Manager

Alert Manager is the receiver to route the alerts.

To configure Alerts in Prometheus,

  1. Get the Webhook URL from the Opsramp configuration.
  2. Use the Webhook URL in the Prometheus Alert Manager configuration to map the YAML file:
{
receivers:
- name: opsramp-webhook
   webhook_configs:
   - url: https://<apiserver>/integrations/alertsWebhook/client_123/alerts?vtoken=<token>
      send_resolved: true
route:
  group_by:
  - alertname
  group_interval: 1m
  group_wait: 30s
  receiver: opsramp-webhook
  repeat_interval: 2m
  routes:
  - receiver: opsramp-webhook
    match_re:
      severity: error|warning
}

Configuring with App label defined

With App label defined only selective alerts are forwarded from Prometheus to OpsRamp. Configuration involves the following:

  1. Configuring Prometheus Alert Manager.
  2. Configuring alert rules.

Step 1: Configure Prometheus Alert Manager

Alert Manager is the receiver to route the alerts.

To configure Alerts in Prometheus,

  1. Get the Webhook URL from the Opsramp configuration.
  2. Use the following Webhook URL in the Prometheus Alert Manager configuration to map the YAML file:
kind: ConfigMap

apiVersion: v1 

metadata: 
    name: alertmanager-config 
    namespace: monitoring 
    data: 
      config.yml: 

global: 

templates: 
- '/etc/alertmanager/*.tmpl'

receivers: 
    - name: default-receiver 
    - name: opsramp-webhook
      webhook_configs:
        - url: "https://<webhook_url>/integrations/alertsWebhook/client_14/alerts?vtoken=<TokenValue>"

route: 
  group_wait: 10s 
  group_interval: 5m 
  receiver: default-receiver 
  repeat_interval: 3h

routes:
  - receiver: opsramp-webhook
  match_re:
  app: opsramp

Step 2: Configure alert rules

This step configures the alert rules in the Prometheus Alert Manager. Filtering rules are created by using Alerting profiles. The alert rules are labeled as OpsRamp alerts for the receiver.

To configure alert rules, add the required Opsramp labels in the prometheus.rules file (config map for alert rules) so that you can map the alerts generated from these rules to the corresponding OpsRamp entities in the OpsRamp alert browser:

YAML file

    
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: prometheus-server-conf
      labels:
        name: prometheus-server-conf
      namespace: monitoring
    data:
      prometheus.rules: |-
        groups:
        - name: devopscube demo alert
          rules:
          - alert: High Pod Memory
            expr: sum(container_memory_usage_bytes) > 1
            for: 1m
            **labels:**
     **severity: critical**
     **app: opsramp**
     **metric: container_memory_usage_bytes**
     **description: testing alert1**
            annotations:
              **summary: High Memory Usage**
        - name: devopscude demo alert2
          rules:
          - alert: High Pod Memory2
            expr: sum(container_memory_usage_bytes) > 2
            for: 1m
           ** labels:**
     **severity: VeryCritical**
     **app: opsramp**
     **metric: container_memory_usage_bytes**
     **description: testing alert2**
            annotations:
              **summary: High Memory Usage2**
      prometheus.yml: |-
        global:
          scrape_interval: 5s
          evaluation_interval: 5s
        rule_files:
          - /etc/prometheus/prometheus.rules
        alerting:
          alertmanagers:
          - scheme: http
            static_configs:
            - targets:
              - "alertmanager.monitoring.svc:9093"
        scrape_configs:
          - job_name: 'kubernetes-apiservers'
    
            kubernetes_sd_configs:
            - role: endpoints
            scheme: https
    
            tls_config:
              ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    
            relabel_configs:
            - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
              action: keep
              regex: default;kubernetes;https
    
          - job_name: 'kubernetes-nodes'
    
            scheme: https
    
            tls_config:
              ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    
            kubernetes_sd_configs:
            - role: node
    
            relabel_configs:
            - action: labelmap
              regex: __meta_kubernetes_node_label_(.+)
            - target_label: __address__
              replacement: kubernetes.default.svc:443
            - source_labels: [__meta_kubernetes_node_name]
              regex: (.+)
              target_label: __metrics_path__
              replacement: /api/v1/nodes/${1}/proxy/metrics
    
          - job_name: 'kubernetes-pods'
            kubernetes_sd_configs:
            - role: pod
    
            relabel_configs:
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              action: keep
              regex: true
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
              action: replace
              target_label: __metrics_path__
              regex: (.+)
            - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
              action: replace
              regex: ([^:]+)(?::d+)?;(d+)
              replacement: $1:$2
              target_label: __address__
            - action: labelmap
              regex: __meta_kubernetes_pod_label_(.+)
            - source_labels: [__meta_kubernetes_namespace]
              action: replace
              target_label: kubernetes_namespace
            - source_labels: [__meta_kubernetes_pod_name]
              action: replace
              target_label: kubernetes_pod_name
    
          - job_name: 'kubernetes-cadvisor'
    
            scheme: https
    
            tls_config:
              ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    
            kubernetes_sd_configs:
            - role: node
    
            relabel_configs:
            - action: labelmap
              regex: __meta_kubernetes_node_label_(.+)
            - target_label: __address__
              replacement: kubernetes.default.svc:443
            - source_labels: [__meta_kubernetes_node_name]
              regex: (.+)
              target_label: __metrics_path__
              replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
    
          - job_name: 'kubernetes-service-endpoints'
    
            kubernetes_sd_configs:
            - role: endpoints
    
            relabel_configs:
            - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
              action: keep
              regex: true
            - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
              action: replace
              target_label: __scheme__
              regex: (https?)
            - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
              action: replace
              target_label: __metrics_path__
              regex: (.+)
            - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
              action: replace
              target_label: __address__
              regex: ([^:]+)(?::d+)?;(d+)
              replacement: $1:$2
            - action: labelmap
              regex: __meta_kubernetes_service_label_(.+)
            - source_labels: [__meta_kubernetes_namespace]
              action: replace
              target_label: kubernetes_namespace
            - source_labels: [__meta_kubernetes_service_name]
              action: replace
              target_label: kubernetes_name

Sample payload

{ "receiver": "opsramp-webhook", "status": "firing", "alerts": 
    [{ 
        "status": "firing", "labels": 
            { "alertname": "High Pod Memory", "app": "opsramp", "severity": "slack" }, 
        "annotations": { "summary": "High Memory Usage" }, 
        "startsAt": "2019-09-19T08:14:52.059731582Z", 
        "endsAt": "0001-01-01T00:00:00Z", 
        "generatorURL": "[http://prometheus-deployment-7bc6dc6f77-ds6j2:9090/graph?g0.expr=sum%28container_memory_usage_bytes%29+%3E+1u0026g0.tab=1](http://prometheus-deployment-7bc6dc6f77-ds6j2:9090/graph?g0.expr=sum%28container_memory_usage_bytes%29+%3E+1u0026g0.tab=1)", 
        "fingerprint": "243ccc9d065e8b26" 
        }, 
    { 
        "status": "firing", 
        "labels": 
            { "alertname": "Low Containers Count", "app": "opsramp", "severity": "page" }, 
        "annotations": { "summary": "Low Container Count" }, 
        "startsAt": "2019-09-19T08:14:53.135072669Z", 
        "endsAt": "0001-01-01T00:00:00Z", 
        "generatorURL": "[http://prometheus-deployment-7bc6dc6f77-ds6j2:9090/graph?g0.expr=sum%28kubelet_running_container_count%29+%3C+40u0026g0.tab=1](http://prometheus-deployment-7bc6dc6f77-ds6j2:9090/graph?g0.expr=sum%28kubelet_running_container_count%29+%3C+40u0026g0.tab=1)", 
        "fingerprint": "a95e6f948c14554a" 
     }
     ], 
  "groupLabels": { }, 
  "commonLabels": { "app": "opsramp" }, 
  "commonAnnotations": { }, 
  "externalURL": "[http://alertmanager-7b6d855bd8-7mvf2:9093](http://alertmanager-7b6d855bd8-7mvf2:9093/)", 
  "version": "4", 
  "groupKey": "{}/{app=~\" ^ ( ? : opsramp) $\"}:{}" 

}

 

What to do next

View alerts in OpsRamp.

  1. From the Workspace drop-down in the OpsRamp Console, navigate to Alerts.
  2. On the Alerts page, search using Prometheus as the Source name.
    The related alerts are displayed.
  3. Click Alert ID to view.
Prometheus Integration

Prometheus Integration