An alert escalation policy can be created to support operational requirements. For example, a policy can be created to escalate critical network issues to a network engineer.
Alert escalation policies are created by:
- Defining the name and scope.
- Selecting the resources where the alert matches this policy.
- Defining alert conditions.
- Defining the escalation rules.
An alert token is a placeholder added to an alert that is escalated as an Incident so that the Incident includes the data the token represents. For example, if
$alert.serviceName (Alert metric) is added to an incident, after the alert is escalated as incident, the incident includes the alert metric name.
Tokens are divided into three categories:
- Alert: Alert category is divided into Alert specific tokens and Alert’s Resource tokens
- Policy: Policy category consists of the Policy name token.
- Functions: Functions category consists of the Substring token. Substring is a string between pointer for reference start character and reference end character. Reference pointer is the delimiter.The substring function token allows you to create dynamic description for an Incident. Delimiter is used to specify boundaries for identifying appropriate string in a data stream. startDelimiter indicates the beginning element in a character string. endDelimiter indicates the end element in a character string.
For example, a user wants to extract Site Account from the Alert Description and set the account number to Account Id of the Incident record:
To extract the Account ID from the Alert Description, use the substring
($function.substring(<String>,<startDelimiter>,<endDelimiter>) ) function. Enter the
$alert.description token as an input
AT SITE ACCOUNT NO=, and endDelimiter does not need to be configured to indicate a natural end delimiter.
Both OpsQ View and OpsQ Manage permissions are required to access the Alert Escalation Policy.
A training file is required to suppress specific alerts or to snooze specific alerts. The training file must include examples of alerts that need to be suppressed. See Alert Management Training File for more information.
Step 1: Define name and scope
- From All Clients, select the client.
- Go to Setup > Alert Management > Alert Escalation.
- Click the Add button and enter Name and Description.
- Select the required mode from the Mode drop-down list.
- Choose the organization whose users from the organization who will receive the escalations from this policy. For example: If you choose a partner organization, only partner users can receive alert escalations.
- Select Next: Select Resources.
Step 2: Select resources
Select the resources where alerts match a policy:
Select resources for the escalation policy.
- Resources can be selected from one or more clients.
- Add up to 25 resources.
- To escalate alerts for users of a specific client, add only resources from that client.
(Optional) Filter the resources by:
- Resource Name
- Resource Type
- Service Group
- Device Group
Select Next: Define Alert Conditions.
When choosing a parent service group or parent resource group, select all of the child group resources that form the escalation policy.
Step 3: Define alert and resource conditions
Filter the type of alerts which occur on the selected resources. If no conditions are defined in this section, all alerts on the selected resources will match this policy.
- Add conditions for an alert based on the alert properties listed.
- Click Add to add multiple conditions.
- Choose if Any or All conditions should be applied to filter the alerts.
- Click Next: Define Escalation Rules.
The alert property Client: Service Name is applicable only to the partner scope policy. As an example, to escalate alerts of a service named Windows Maintenance Support, add the condition Client: Service Name:
Step 4: Define escalation rules
Escalation rules are actions to be taken when an alert is not acknowledged within a time period.
List of escalation rules including the description:
|Escalate directly as needed||Select the users who should be directly contacted on an on-demand basis. Escalating directly is a manual process. No automatic actions are taken.|
|Escalate alert automatically until it is acknowledged, closed, suppressed, or ticketed||Escalate an alert as notification until an action is taken. No further level of escalation can be added since the alert escalation will end as soon as the incident is created.|
|Escalate alert automatically until it is acknowledged, closed, or suppressed||Escalate an alert as notification even if an incident is created.|
To configure escalation rules:
Alert Elapsed Timeline: Configure the time interval to escalate an alert after it is generated.
- Select Immediately on the timeline to escalate an alert as soon as the alert is generated. If the alert does not have correlation or first response policies, the algorithm immediately moves the alert to Alert Escalation. If the alert does have correlation and first response policies, the algorithm starts the escalation process and finishes after the correlation is complete.
- Select Wait Minutes/Hours on the timeline to escalate after the alert has elapsed and the policy conditions have been met.
Escalate as Notification: Select to send continuous notifications until an action (acknowledge, suppress, close or create Incident) is taken on the alert.
For example, if an alert arrives at 10:00 AM, a notification is sent to the user. If the user does not acknowledge the alert, another notification is sent after 5 minutes (10:05 AM). Notifications are continuously sent every 5 minutes. When the user acknowledges the alert at 10:20 AM, no further notifications are sent.
Escalate as Incident: Select to escalate the alert as an incident. The escalation stops immediately and no more escalations are sent.
Escalate as notification
Select Escalate As Notifications to send periodic notifications to users to make sure that an alert is acknowledged.
Send Alert Notifications to: Configure the users who be sent the notifications. To select the users, click Select Users.
Notification Priority: An escalation notification carries a priority to determine which channel for delivery of notifications. As an example, policy P1 is configured to send Normal priority notifications for all the matching alerts to user A. User A can specify that all normal notifications are received using email (and not using SMS or Voice).
- Repeat Notification Frequency: Configure repeat notification frequency for selected users.
The repeat notification is sent to the user as Set Repeat Frequency even if an alert is escalated
to a higher level. The user will no longer receive notifications after a certain number of repeated notifications.
- The default repeat frequency is 15 minutes and the number of notifications is 2.
- The minimum number of notifications is 2 and maximum is 10.
- Alert State Transition: Alerts can be escalated based on the alert state transition which allows notifications on selected alert state change. For example, user can escalate alerts as a notification when alert state changes from warning to critical. The notification is a one-time notification sent at the time of the state transition.
- Repeat Notification Frequency: Configure repeat notification frequency for selected users. The repeat notification is sent to the user as Set Repeat Frequency even if an alert is escalated to a higher level. The user will no longer receive notifications after a certain number of repeated notifications.
Add Escalation: Adding escalation levels will escalate the alert to the next level if it is not acknowledged at a prior level.
- A policy can have multiple escalation notifications.
- When a level 2 escalation is added to notify users, level 1 and level 2 users receive repeated notification according to the repeat notification frequency.
Escalate as incident
You cannot escalate an alert that displays an OK state as Incident.
Select Escalate As Incident to automatically create an incident from an alert and assign it to the user.
For New Incidents: Configure the properties of the incident that is created when an alert’s condition matches this policy.
- A new incident is created for an alert if there is no open incident existing for the alert.
- The incident property tokens available in the auto incident form can be used to customize the subject and description of the incident.
- Enable Continous Learning to automatically escalate incidents to appropriate groups, priority or category using machine learning.
For Created Incidents: If an open incident (an incident in any state other than Closed) is available for the alert, the incident is updated instead of creating a new incident.
Update Incident: Configure how to update existing incident of an escalated alert. The incident gets updated every time an escalated alert repeats with a state change.
Update incident with latest alert description when alert state change: The latest alert description is appended to the conversation of the incident. No change to the status of the incident is made based on escalated alert’s state.
- Resolve incident when an alert heals: The latest alert description is appended to the conversation of the incident. The incident status is changed to Resolved when an escalated alert’s state changes to OK.
- Update incident priority based on these rules: The priority of an incident is updated according to the rule configured for the alert severity change.
Notify: Configure which updates to an incident should be notified to users.
- For every alert update: Send an incident notification when any update of an escalated alert is appended to the incident.
- Only when the alert state changed: Send incident notification only when escalated alert heals and the update is appended to the incident.
Review alert escalation policy
A summary of all sections of the escalate alert policy is available for review and editing: