An alert escalation policy can be created to support operational requirements. For example, a policy can be created to escalate critical network issues to a network engineer.
Both OpsQ View and OpsQ Manage permissions are required to access the Alert Escalation Policy.
A training file is required to suppress specific alerts or to snooze specific alerts. The training file must include examples of alerts that need to be suppressed. See Alert Management Training File for more information.
Step 1: Define name and scope
In this step, define the name, mode, and partner or client for the escalation policy.
Ensure the partner you want to create an alert escalation policy is selected in the Partner list (upper-left corner).
Go to Setup > Alerts > Alert Escalation.
Click Create New or + Add, depending on whether you have any existing policies.
Enter a Name and Description.
From the Mode list, select a policy mode.
Policy Mode Description ON The policy drives automated actions on alerts. OFF The policy is inactive and does not affect alerts. You can use this mode to review a newly defined policy before choosing one of the other modes. Recommend The policy creates a recommendation for actions that you should take on the alert. Recommendations are based on OpsRamp learning from historical alerts. The recommendation includes a link to take the action. Observed This mode permits you to simulate a policy without affecting alerts.
The policy creates an observed alert, which simulates the original alert. The observed alert shows the actions that would have been taken on the original alert if the policy were in On mode. The observed alert includes a link to the original alert.
Recommend and Observed modes apply to incident actions.
Select the partner that receives escalations from this policy.
Click Next: Select Resources.
Step 2: Select resources
Select the resources where alerts match a policy:
Select resources for the escalation policy.
- Resources can be selected from one or more clients.
- Add up to 25 resources.
- To escalate alerts for users of a specific client, add only resources from that client.
(Optional) Filter the resources by:
- Resource Name
- Resource Type
- Service Group
- Device Group
Click Next: Define Alert Conditions.
When choosing a parent service group or parent resource group, select all of the child group resources that form the escalation policy.
Step 3: Define alert and resource conditions
Filter the type of alerts which occur on the selected resources. If no conditions are defined in this section, all alerts on the selected resources will match this policy.
Choose from Any or All of the following conditions to apply a filter for the alerts.
Select Native Attributes or Custom Attributes depending on your requirement.
Native Attributes are the predefined attributes and Custom Attributes are user-defined attributes.
Select the required attribute, logical operator, and enter the value. Click + to add multiple filter conditions.
Click Next: Define Escalation Rules.
The alert property Client: Service Name is applicable only to the partner scope policy. As an example, to escalate alerts of a service named Windows Maintenance Support, add the condition Client: Service Name:
Step 4: Define escalation rules
Escalation rules are actions to be taken when an alert is not acknowledged within a time period.
List of escalation rules including the description:
|Escalate directly as needed||Select the users who should be directly contacted on an on-demand basis. Escalating directly is a manual process. No automatic actions are taken.|
If you select Escalate directly as needed, you will only select users who receive notifications.
|Escalate alert automatically until Acknowledged, Closed, Suppressed, or Ticketed||Escalate an alert as a notification until an action is taken. No further level of escalation can be added since the alert escalation will end as soon as the incident is created.|
|Escalate alert automatically until Acknowledged, Closed, or Suppressed||Escalate an alert as notification even if an incident is created.|
To configure escalation rules with actions:
Alert Elapsed Timeline: Configure the time interval to escalate an alert after it is generated.
- Select Immediately on the timeline to escalate an alert as soon as the alert is generated. If the alert does not have correlation or first response policies, the algorithm immediately moves the alert to Alert Escalation. If the alert does have correlation and first response policies, the algorithm starts the escalation process and finishes after the correlation is complete.
- Select Wait Minutes/Hours on the timeline to escalate after the alert has elapsed and the policy conditions have been met.
Escalate as Notification: Select to send continuous notifications until an action (acknowledge, suppress, close or create Incident) is taken on the alert.
For example, if an alert arrives at 10:00 AM, a notification is sent to the user. If the user does not acknowledge the alert, another notification is sent after five minutes (10:05 AM). Notifications are continuously sent every five minutes. When the user acknowledges the alert at 10:20 AM, no further notifications are sent.
Escalate as Incident: Select to escalate the alert as an incident. The escalation stops immediately and no more escalations are sent.
Escalate as notification
You can escalate an alert as a notification.
Select Escalate As Notifications to send periodic notifications to users to make sure that an alert is acknowledged.
Send Alert Notifications to: Configure the users who be sent the notifications. To select the users, click Select Users.
Notification Priority: An escalation notification carries a priority to determine which channel for delivery of notifications. As an example, policy P1 is configured to send Normal priority notifications for all the matching alerts to user A. User A can specify that all normal notifications are received using email (and not using SMS or Voice).
- Repeat Notification Frequency: Configure repeat notification frequency for selected users.
The repeat notification is sent to the user as Set Repeat Frequency even if an alert is escalated
to a higher level. The user will no longer receive notifications after a certain number of repeated notifications.
- The default repeat frequency is 15 minutes and the number of notifications is 2.
- The minimum number of notifications is two and maximum is 10.
- Alert State Transition: Alerts can be escalated based on the alert state transition which allows notifications on selected alert state changes. For example, you can escalate alerts as a notification when the alert state changes from warning to critical. The notification is a one-time notification sent at the time of the state transition.
- Repeat Notification Frequency: Configure repeat notification frequency for selected users. The repeat notification is sent to the user as Set Repeat Frequency even if an alert is escalated to a higher level. The user will no longer receive notifications after a certain number of repeated notifications.
Add Escalation: Adding escalation levels will escalate the alert to the next level if it is not acknowledged at a prior level.
- A policy can have multiple escalation notifications.
- When a level 2 escalation is added to notify users, level 1 and level 2 users receive repeated notifications according to the repeat notification frequency.
Escalate as incident
You cannot escalate an alert that displays an OK state as an incident.
Select Escalate As Incident to automatically create an incident from an alert and assign it to the user.
For New Incidents: Configure the properties of the incident that is created when an alert’s condition matches this policy.
- A new incident is created for an alert if there is no open incident existing for the alert.
- The incident property tokens available in the auto incident form can be used to customize the subject and description of the incident.
- Enable Continous Learning to automatically escalate incidents to appropriate groups, priority, or category using machine learning.
For Created Incidents: If an open incident (an incident in any state other than Closed) is available for the alert, the incident is updated instead of creating a new incident.
Update Incident: Configure how to update existing incident of an escalated alert. The incident gets updated every time an escalated alert repeats with a state change.
Update incident with latest alert description when alert state change: The latest alert description is appended to the conversation of the incident. No change to the status of the incident is made based on escalated alert’s state.
- Resolve incident when an alert heals: The latest alert description is appended to the conversation of the incident. The incident status is changed to Resolved when an escalated alert’s state changes to OK.
- Update incident priority based on these rules: The priority of an incident is updated according to the rule configured for the alert severity change.
Notify: Configure which updates to an incident should be notified to users.
- For every alert update: Send an incident notification when any update of an escalated alert is appended to the incident.
- Only when the alert state changed: Send incident notification only when escalated alert heals and the update is appended to the incident.
Step 5: Review
Review and save the alert escalation policy.
A summary of all sections of the escalate alert policy is available for review and editing:
Click Edit to edit any sections.