Prometheus Alertmanager (v2)

Receive Alertmanager events and turn them into Dynatrace Problems

Configuration

In Dynatrace, create a monitoring configuration for this extension, this extension should run on a single Activegate, later Prometheus Alertmanager will need to be configured to send events to this specific Activegate.

The parameters are:

• Group name - A user friendly group name for the Custom Devices to be created by this endpoint
• API Token - A token to create custom devices, read and write problems (DataExport, problems.read, problems.write)
• Listening Port - The port where the Activegate will listen to, Alertmanager needs to reach this port.
• Problem opening severities - The list of "severity" labels that should send problem opening events. For instance critical. All other severity labels will send Custom Info Events instead.

Alertmanager

In Alertmanager, create a webhook config where the url field points to the Dynatrace Activegate, example: http://<my.ag.ip.address>:9393/webhook.

Example receiver configuration:

receivers:
- name: 'dynatrace'
  webhook_configs:
  - url: 'http://100.94.253.97:9393/webhook'

The endpoint always ends in /webhook. Note that problems will only be created for alerts where label called severity matches the ones configured in Dynatrace.

All other alerting options and routes are to be configured in Alertmanager.

Troubleshooting

You can troubleshoot the integration by sending a POST request directly to the Activegate:

curl -H 'Content-Type: application/json' -d '{"receiver": "dynatrace", "status": "firing", "alerts": [{"status": "firing", "labels": {"alertname": "TargetDown", "namespace": "kube-system", "service": "kubelet", "severity": "warning"}, "annotations": {"message": "11.11% of the kubelet/kubelet targets in kube-system"}, "startsAt": "2021-03-19T01:35:45.72Z", "endsAt": "0001-01-01T00:00:00Z", "generatorURL": "http://openshift.com", "fingerprint": "e425bb91067b6c9e"}], "groupKey": "{}:{alertname=\"Test Alert\", cluster=\"Cluster 02\", service=\"kubelet", "groupLabels": {"alertname": "Test Alert", "cluster": "Cluster 02", "service": "Service 02"}, "commonLabels": {"alertname": "Test Alert", "cluster": "Cluster 02", "service": "Service 02"}, "commonAnnotations": {"annotation_01": "annotation 01", "annotation_02": "annotation 03"}, "externalURL": "http://8598cebf58a1:9093"}' http://<ACTIVEGATE_IP_ADDRESS>:9393/webhook