Trigger Dynatrace problem alerts from external sources using Events API

Dynatrace OneAgent is the most powerful tool available for full-stack monitoring of events in your software ecosystem. OneAgent provides alerting on more than 100 different problem types. The severity levels of detected and intelligently correlated problem types include resource contentions, slowdowns, errors, availability issues, and more.

Although built-in intelligence enables Dynatrace to raise alerts about all kinds of critical situations, some use cases still require external tools to trigger alerts within Dynatrace.

Generate events and trigger alerts with third-party tools

Until now, the Dynatrace Events API only offered the push of informational events into Dynatrace to enrich root-cause data—it wasn’t able to trigger new problems. With the release of Dynatrace Saas version 1.151, the Dynatrace Events API endpoint is no longer limited to informational events. Third-party API clients and tools can now create all types of severity events and thereby trigger the creation and correlation of alerts for new problems.

For example, imagine that you need to receive an alert from your UPS notifying you that one of the server rooms in your data center has suffered a severe power outage. Or in the unusual case that you need to use a legacy network monitoring system to alert you of network problems, say you may want to push these events into Dynatrace to take advantage of the intelligent problem correlation Dynatrace provides.

Note: Unbound events can’t be sent to Dynatrace as they can’t be correlated with auto-detected topology.

In the example of a power outage in a specific room within your data center, such an event would provide valuable root-cause information, even if Dynatrace has already alerted the outage of several hosts through standard availability alerting.

External alerts via the Events API

Now let’s take a look at how such an external alert can be triggered through the Dynatrace Events API endpoint. We’ll choose an availability event, as power outages typically cause severe problems in traditional infrastructure.

As you can see in the example JSON event payload below, we’ll target our event at all hosts that have the tag room23. This will enable us to focus on components that will be affected by a power outage in server room 23.  The event payload includes a list of key-value properties that provide details about the detected incident.

It’s important to note that all external events must target at least one Smartscape component, which can be a host or other component, such as a service, application, or even a single process.

This HTTP POST call opens an availability problem for the example power outage in server room 23:

HTTP POST https://<YOUR_ENVIRONMENT>/api/v1/events/?Api-Token=<YOUR_API_TOKEN>

Payload:

{

  "title": "Power outage",

  "source" : "Power control monitoring",

  "description" : "A power outage was detected in server room 23",

  "eventType": "AVAILABILITY_EVENT",

   "attachRules":{

                               "tagRule":[{

               "meTypes":["HOST"],

            "tags":["room23"]

        }]

  },

  "customProperties":{

    "Power out time": "12:00"

  }

}

The problem generated in this example automatically closes after 15 minutes if no other active events are correlated with it. If you need to overwrite the standard timeout of your custom event, use the timeoutMinutes attribute to set a different timeout period.

The Events API also allows you to refresh the event while it’s still open. Say you set a timeout of 30 minutes but discover that the incident is ongoing. You can send the same event again and it will refresh for another 30-minute duration. This refresh mechanism allows external event sources to keep events open while incidents are ongoing.

Problem creation in Dynatrace

Once the event is accepted by the Events API endpoint, all your hosts receive your custom event and open a problem, as shown below.

Clicking the availability problem reveals the custom event name along with the custom key-value properties we provided with the event payload.

The newly introduced event types greatly enhance your options for seamlessly integrating external event sources into Dynatrace problem correlation. The additional event types can be used by third-party tools. They can also be pushed from cloud deployment platforms.

To implement your own use cases, please refer to our Events API help page and read more about how to push external events to Dynatrace.

To learn more about the different event severity levels and the difference between single events and correlated problems, refer to our event type help page.