Efficiently filter problem notifications with alerting profiles

The latest Dynatrace release introduces an important change in the way that notifications are sent out for detected problems in your environment. To date, personalized watch settings have been used to define how alerts are sent out for detected problems. This approach has proven to be cumbersome within large environments. Also, watch settings can’t account for important factors of detected problems, such as severity levels and tag-based filters.

Alerting profiles enable you to set up fine-grained alert-filtering rules that are based on the severity, customer impact, associated tags, and/or duration of detected problems. Alerting profiles enable you to control exactly which conditions result in problem notifications and which don’t. This includes all problem-push notifications that are sent via the Dynatrace mobile app and displayed in the Dynatrace web UI. Alerting profiles can also be used to set up filtered problem-notification integrations with 3rd party messaging systems like Slack, HipChat, and PagerDuty.  These enhancements enable you to better filter and route problem notifications within large environments.

How alerting profiles work

Alerting profiles control the delivery of problem notifications across your organization’s alerting channels by considering predefined filters that are based on problem severity, problem duration, and tags. Alerting profiles control all alerts that appear in the Dynatrace UI, alerts that are sent via the Dynatrace mobile app, and the alerts that are sent via 3rd party integrations, such as Slack and HipChat.

Each of your monitoring environments has a default alerting profile that defines the level of severity that must be met before an alert is sent out for a detected problem. The default alerting profile applies to all users within your environment—if/until your environment admin creates a new alerting profile and assigns it to individual users.

Your environment administrator can create new alerting profiles that provide fine-grained control over the routing of problem alerts for individual teams or for 3rd party problem notification integrations, such as Slack, HipChat, etc.

To view your environment’s alerting profiles, go to Settings > Alerting > Alerting profiles. The list of alerting profiles in the example below includes the Default profile and two customer-defined team-specific profiles (Team Detroit and Team Linz).

Note: The default profile can be modified, but not deleted.

Create an alerting profile

Alerting profiles provide a powerful filtering mechanism for problem notifications. By combining filter criteria, you can create custom profiles that, for example, delay notification of problems in development environments while immediately alerting on problems detected in production environments.

To create an alerting profile

1. Go to Settings > Alerting > Alerting profiles.
2. Type a name for the new profile in the Create new alerting profile field.
3. Click the Create button.
4. On the Edit profile page, click the Create alerting rule button to add a rule to the new profile.
5. Select a severity level from the Problem severity level drop list.
   Severity options include Availability, Error, Performance, or Resource.

   

6. Type in a number of minutes to define how long a problem duration should last before an alert is sent out. This enables you to avoid alerts for fleeting, low-severity problems that don’t affect customer experience and therefore don’t require immediate attention.
7. You can further filter problems down to specific monitored entities in your environment by selecting an existing tag from the Select tag field. The Filter problems by tag drop list enables you to define a logical operator for the tags you select. Only include entities that have all tags (i.e., AND) enables you to create combined tag rules. Include entities that have at least one tag (i.e., OR) enables you to create either/or rules for tags.

Alerting profile example

Say you want to receive immediate alerts on availability problems that involve production entities, but you only want to receive alerts on development availability problems that are open longer than 15 minutes. To model this example, you would create two availability filtering rules in your alerting profile, as shown below.

Once you’ve created a new alerting profile, you need to assign the profile to specific users. The assigned profile will control the problem notifications that are sent to these users and the problem tally that appears in the menu bar.

Third-party notification integrations

All third-party problem notification integrations now support alerting profiles. The seamless use of alerting profiles for third-party problem integrations allows efficient filtering regardless of the alerting channel your team uses (web UI, Dynatrace mobile app, Slack, HipChat, ServiceNow, etc). See Slack example below.

Migrate from watch settings to alerting profiles

When you enable alerting profiles, all existing watch settings, including the impact-level watch settings above, will be overwritten by the rules in your active alerting profiles—you will no longer be able to view or access your existing watch settings. Additionally, any tag-based filters or configurations you have set up for 3rd party messaging integrations will also be overwritten by your alerting profiles. Until you set up custom alerting profiles for specific integrations and users, notifications will be sent out based on the filtering rules set up in the default alerting profile.

As existing personalized watch settings can’t be migrated automatically to alerting profiles, you must manually migrate your watch settings. Note that newly created environments (version 125 and above) won’t need this migration step—alerting profiles are enabled by default.

To enable alerting profiles in an existing environment

1. Have your environment administrator select Problems from the navigation menu.
2. Click the Enable alerting profiles button (see below).

   

Automated migration of watch settings to alerting profiles

The use of filtering rules based on severities, problem duration, and tags provides a powerful mechanism for routing and filtering problem alerts for all teams in your organization. Within the next several releases, your Dynatrace environment will be automatically migrated to alerting profiles, watch settings will be entirely deprecated, and your personalized watch settings will be lost.

To immediately benefit from alerting profiles, ask your Dynatrace environment administrator to enable them in your environment.