Overview

Annotation-Based Auto-Monitoring enables zero-configuration monitoring of Kubernetes workloads by reading monitoring intent directly from pod annotations. Instead of configuring each workload through the OpsRamp console, you annotate your Kubernetes pods with a standard annotation, and metrics collection starts automatically within 60 seconds.

This feature is designed for platform engineers, SREs, and Kubernetes administrators who want to:

  • Monitor applications immediately upon deployment, with no manual configuration step
  • Keep monitoring configuration co-located with application manifests (GitOps-friendly)
  • Securely pass credentials using Kubernetes Secrets, without exposing plaintext values
  • Automatically stop monitoring when pods are removed

Prerequisites

RequirementDetails
OpsRamp AgentThe OpsRamp Kubernetes 2.0 Agent (master and worker) must be installed and running in the Kubernetes cluster.
Agent VersionUse OpsRamp Kubernetes 2.0 Agent version 22.0.0 or later, which includes annotation-based discovery support.
RBAC PermissionsThe master agent's ServiceAccount must have the get permission on Kubernetes Secrets in the namespaces where annotated pods are deployed.
Supported ReceiverThe workload must use one of the supported receiver types.

Supported receiver types

Receiver TypeDescription
mongodbCollects metrics from MongoDB database instances.
redisCollects metrics from Redis cache and datastore instances.
mysqlCollects metrics from MySQL database instances.
nginxCollects metrics from NGINX web server instances.
haproxyCollects metrics from HAProxy load balancer instances.
kafkaCollects metrics from Apache Kafka broker instances.
jmxCollects JMX metrics from Java-based applications.
apachesparkCollects metrics from Apache Spark clusters.
corednsCollects metrics from CoreDNS servers.
prometheusCollects metrics by scraping generic Prometheus endpoints.

How it works

When you add a single hpe.opsramp.com annotation to a pod, the following process occurs:

  1. The master agent detects the pod and reads the annotation.
  2. It waits 60 seconds (a debounce window, so a rolling deployment of many pods does not trigger repeated restarts) and writes the monitoring config to a ConfigMap.
  3. The worker agent on that node picks it up and starts scraping metrics.

When you delete the pod, the same process runs in reverse, and monitoring stops automatically. Pods that were already running before you installed the agent are also picked up through a one-time startup sync.

Configuration

To configure annotation-based monitoring for a pod, follow these steps:

Step 1: Add the annotation

Add the hpe.opsramp.com annotation to your pod’s (or Deployment/StatefulSet’s) metadata, with a JSON value describing what to monitor:

metadata:
  annotations:
    hpe.opsramp.com: |
      {
        "<receiver-type>": {
          "instances": [
            {
              "host": "%%host%%",
              "port": <port-number>,
              "collection_interval": "<interval>"
            }
          ]
        }
      }

Required fields:

FieldRequiredDescription
<receiver-type>YesSpecifies the receiver type. Use one of the supported receiver types, such as mongodb, redis, or mysql.
instancesYesDefines one or more instance configurations for the selected receiver.
hostYesSpecify %%host%% to automatically resolve the monitored pod's IP address.
portYesSpecifies the port on which the monitored workload is listening.
collection_intervalNoSpecifies how often metrics are collected (for example, 30s or 60s). If omitted, the receiver's default collection interval is used.
username / passwordNoProvide credentials only if the workload requires authentication. See Step 3 for configuration details.
tlsNoSet to true to connect to the workload using TLS.

A few receivers need one extra field; otherwise, a sensible default applies:

ReceiverFieldDefault Value (If Omitted)
nginxhttpPath/nginx_status
haproxyhttpPath/stats
prometheusmetrics_path/metrics

You can monitor more than one instance from the same pod by adding more entries to instances:

"redis": {
  "instances": [
    { "host": "%%host%%", "port": 6379 },
    { "host": "%%host%%", "port": 6380 }
  ]
}

Step 2: Reference dynamic values with placeholders

Two placeholders are available so the same annotation works across every replica of a workload:

PlaceholderResolves ToExample
%%host%%The IP address of the monitored pod.10.244.0.15
%%env_VARNAME%%The value of the environment variable VARNAME defined in the pod specification.%%env_REDIS_PASS%% resolves to the value of the REDIS_PASS environment variable.

Step 3: Secure credentials with Kubernetes secrets

If your workload needs a username or password, pull it from a Secret via an environment variable — never write it directly into the annotation.

spec:
  containers:
    - name: myapp
      env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: my-secret
              key: password

Then reference the env var in your annotation:

"password": "%%env_DB_PASSWORD%%"

Secret-backed values are fetched only at collection time and never written to disk in plaintext. Avoid configMapKeyRef or hardcoded values for anything sensitive because the system stores and resolves those values directly, with no protection.

Step 4: Apply the annotation to your pod

kubectl apply -f my-deployment.yaml

Wait approximately 60 seconds, and then verify:

Step 5: Verify metrics are being collected

kubectl get cm opsramp-agent-<node-name> -n opsramp-agent -o yaml

Look for your workload under the raw-workloads key — then check the OpsRamp console to confirm metrics are arriving on the resource.

Examples

MongoDB with Secret-backed credentials

This is the most complete example, since it shows Secrets, the MONGO_INITDB_ROOT_* bootstrap variables, and the annotation together:

apiVersion: v1
kind: Secret
metadata:
  name: mongo-secret
  namespace: monitoring
stringData:
  user: opsramp_monitor
  password: MonitorPass456
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb
  namespace: monitoring
spec:
  template:
    metadata:
      annotations:
        hpe.opsramp.com: |
          {
            "mongodb": {
              "instances": [{
                "host": "%%host%%",
                "port": 27017,
                "username": "%%env_MONGO_USER%%",
                "password": "%%env_MONGO_PASS%%",
                "collection_interval": "60s",
                "tls": false
              }]
            }
          }          
    spec:
      containers:
        - name: mongodb
          image: mongo:7.0
          ports:
            - containerPort: 27017
          env:
            - name: MONGO_USER
              valueFrom:
                secretKeyRef:
                  name: mongo-secret
                  key: user
            - name: MONGO_PASS
              valueFrom:
                secretKeyRef:
                  name: mongo-secret
                  key: password
            - name: MONGO_INITDB_ROOT_USERNAME
              valueFrom:
                secretKeyRef:
                  name: mongo-secret
                  key: user
            - name: MONGO_INITDB_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mongo-secret
                  key: password

Redis

metadata:
  annotations:
    hpe.opsramp.com: |
      {
        "redis": {
          "instances": [{
            "host": "%%host%%",
            "port": 6379,
            "password": "%%env_REDIS_PASS%%",
            "collection_interval": "60s"
          }]
        }
      }

NGINX

metadata:
  annotations:
    hpe.opsramp.com: |
      {
        "nginx": {
          "instances": [{
            "host": "%%host%%",
            "port": 80,
            "httpPath": "/nginx_status",
            "collection_interval": "60s"
          }]
        }
      }

Prometheus endpoint

metadata:
  annotations:
    hpe.opsramp.com: |
      {
        "prometheus": {
          "instances": [{
            "host": "%%host%%",
            "port": 8080,
            "metrics_path": "/metrics",
            "collection_interval": "30s"
          }]
        }
      }

HAProxy

metadata:
  annotations:
    hpe.opsramp.com: |
      {
        "haproxy": {
          "instances": [{
            "host": "%%host%%",
            "port": 8404,
            "httpPath": "/stats",
            "collection_interval": "60s"
          }]
        }
      }

Multiple instances per receiver

You can monitor multiple instances of the same receiver type from a single pod:

metadata:
  annotations:
    hpe.opsramp.com: |
      {
        "redis": {
          "instances": [
            {
              "host": "%%host%%",
              "port": 6379,
              "collection_interval": "60s"
            },
            {
              "host": "%%host%%",
              "port": 6380,
              "collection_interval": "60s"
            }
          ]
        }
      }

Troubleshooting

SymptomVerification / Resolution
No metrics are collected after 60+ seconds.Check the master agent logs by running kubectl logs -n opsramp-agent <master-pod> | grep annotation-discovery. Then verify that the ConfigMap contains a raw-workloads entry for the workload.
%%env_VARNAME%% is not resolved.Verify that the environment variable name matches exactly (case-sensitive), the referenced Secret exists in the pod's namespace, and the master agent's ServiceAccount has get permission on Secrets in that namespace.
MongoDB authentication fails.Ensure the monitoring user is configured with SCRAM-SHA-1 authentication and has the clusterMonitor role on the admin database and the read role on the local database.
An existing pod is not discovered.Check the master agent logs for the message initial pod sync processed N running pods. Also verify that the annotation key is exactly hpe.opsramp.com without any additional prefix or suffix.
ConfigMap entry is not removed after a pod is deleted.Restart the master agent pod to trigger a full synchronization and remove stale ConfigMap entries.

Limitations

LimitationDetails
Debounce DelayAfter an annotation is added or updated, metrics collection may take up to 60 seconds to begin.
Single Annotation KeyEach pod supports only one hpe.opsramp.com annotation. To monitor multiple receiver types for the same pod, define all receiver configurations within a single JSON object.
Node-Scoped CollectionMetrics are collected only by the worker agent running on the same Kubernetes node as the monitored pod. Cross-node metrics collection is not supported.
No UI ConfigurationAnnotation-based workload monitoring is configured entirely through Kubernetes manifests. Configuration and management through the OpsRamp UI are not supported.