Some applications produce log entries that span multiple lines. Common examples include Java stack traces, Python tracebacks, and multi-line JSON objects. By default, the agent treats each line as a separate log entry, which breaks these into fragmented, hard-to-read pieces in the Logs UI.

Multiline configuration lets you define rules that recombine related lines into a single, coherent log entry before it is sent to the OpsRamp Platform.

Prerequisites

• Log Management is enabled for your client.
• Pod log collection is enabled. For details, see Configuring Pod Log Collection.
• OpsRamp Kubernetes 2.0 Agent version 22.0.0 or later.

How it works

You define multiline rules as an array under multiline in the opsramp-logs-user-config ConfigMap. Each rule specifies:

• A pattern that identifies where a log entry starts or ends.
• Optional filters that scope the rule to a specific namespace, pod, or container.

You can define multiple rules in the same array, each targeting a different application or log format.

kubectl edit configmap opsramp-logs-user-config -n <agent-installed-namespace>

pods:
  enable: true
  multiline:
    - line_start_pattern: '<regex>'
      namespaces:
        - <namespace>
      pods:
        - <workload-name>
      containers:
        - <container-name>

Field reference

Scoping rules with filters

Filters narrow which pods a rule applies to. All three dimensions—namespace, pod, and container—must match (they work as AND conditions), while multiple values within a single filter are OR-ed.

1. Namespaces

namespaces:
  - production
  - staging

The rule applies if the log comes from either namespace listed.

2. Pods

Provide the workload name — the Deployment, StatefulSet, DaemonSet, or Job name — and not the full pod name. You do not need to include the replica hash suffix.

pods:
  - payment-service

This automatically matches pods like payment-service-6b8f9c7d5-abc12.

3. Containers

Provide the exact container name as defined in spec.containers[].name in the pod manifest.

containers:
  - app
  - sidecar

4. Omitting filters

Warning: Risk of global rules

A rule with no namespaces, pods, or containers filters applies to every pod in the cluster which can cause serious problems if different applications use different log formats.

Example: a global rule with a timestamp pattern:

pods:
  enable: true
  multiline:
    - line_start_pattern: '^\d{4}-\d{2}-\d{2}'

This works correctly for Java apps that start every entry with a date like 2024-06-10. But consider what happens to an application whose logs do not match that pattern, like Nginx access logs: every line gets merged into a single, ever-growing entry, because none of them ever matches the start pattern.

How to avoid this

Prefer scoped rules over global ones:

pods:
  enable: true
  multiline:
    # Scoped to Java apps using this timestamp format
    - line_start_pattern: '^\d{4}-\d{2}-\d{2}'
      pods:
        - payment-service
        - order-service
    # Scoped to Python apps with bracket timestamps
    - line_start_pattern: '^\['
      pods:
        - ml-worker
        - cache-service

With this approach, pods not listed in any rule (like Nginx) are unaffected; their logs pass through as normal single-line entries.

If you must use a global rule, place more specific rules before it, and make sure the global pattern matches the majority of applications in your cluster:

pods:
  enable: true
  multiline:
    # Specific rule for Python apps — evaluated first
    - line_start_pattern: '^\['
      pods:
        - ml-worker
    # Global fallback — only safe if most apps use this format
    - line_start_pattern: '^\d{4}-\d{2}-\d{2}'

Even with this ordering, any pod whose logs do not match the global pattern will still have all its lines merged. Use global rules with caution.

Pattern reference

line_start_pattern — a new entry starts whenever a line matches. All subsequent non-matching lines are appended to it.

line_end_pattern — an entry ends whenever a line matches. The next line starts a new entry.

Examples

Example 1: Java stack traces (using line_start_pattern)

pods:
  enable: true
  multiline:
    - line_start_pattern: '^\d{4}-\d{2}-\d{2}'
      pods:
        - payment-service

This rule means: “A new entry starts whenever a line begins with a date like 2024-06-10. Everything else belongs to the previous entry."

Result in OpsRamp: the first four lines appear as a single log entry; the fifth starts a new one.

Example 2: Multi-line JSON logs (using line_end_pattern)

pods:
  enable: true
  multiline:
    - line_end_pattern: '^}'
      pods:
        - json-logger

This rule means: “An entry ends whenever a line starts with }. The next line starts a new entry."

Result in OpsRamp: the first five lines form one JSON log entry; the sixth starts the next.

Example 3: Multiple rules for different applications

pods:
  enable: true
  multiline:
    # Rule 1: Java apps in production
    - line_start_pattern: '^\d{4}-\d{2}-\d{2}'
      namespaces:
        - production
      pods:
        - payment-service
      containers:
        - app
    # Rule 2: Python tracebacks for ML worker
    - line_start_pattern: '^Traceback|^\d{4}-\d{2}-\d{2}'
      pods:
        - ml-worker
    # Rule 3: Bracket-prefixed logs in the monitoring namespace
    - line_start_pattern: '^\['
      namespaces:
        - monitoring

Example 4: Scoping a rule to namespace, pod, and container together

pods:
  enable: true
  multiline:
    - line_start_pattern: '^\d{4}-\d{2}-\d{2}'
      namespaces:
        - production
      pods:
        - payment-service
      containers:
        - app

Common use cases

Important notes

• Each rule must specify exactly one of line_start_pattern or line_end_pattern. Do not specify both, and do not omit both.
• The multiline field accepts an array, so multiple rules can coexist, each scoped to different workloads or namespaces.
• Logs that do not match any multiline rule are sent as single-line entries; no logs are dropped.
• Changes to the ConfigMap are picked up automatically within a few minutes. No manual agent restart is required.
• Multiline configuration for pod logs is separate from multiline configuration for node logs. Node logs continue to use the existing single-object multiline format defined inside each configs entry. For details, see Node Logs (Syslog).

Troubleshooting

If you encounter kube events issues, see the Troubleshooting documentation.