New event type helps avoid unnecessary alerts for planned host downscaling

To enable you to automatically detect the shutdown of spot instances and the scaling up or down of third-party autoscaling solutions, we’ve introduced a new event type.

Modern service infrastructure depends heavily on IT’s ability to dynamically scale the number of hosts up or down, depending on the expected workload. Cloud providers, such as AWS, Azure, and GCP, help to automate the process of upscaling or downscaling compute power by providing autoscaling groups.

Beyond cloud provider solutions, there are multiple additional frameworks and tools that help IT departments to dynamically adapt their compute power dynamically.

In some cases, we’ve seen these tools attempt to reduce the price of the additional compute power through the use of short-lived spot instances. As the expected behavior of spot instances is that they are shut down within 5 minutes of their creation, the traditional strategy of availability alerting isn’t viable.

To avoid false-positive alerts, Dynatrace availability alerting for servers automatically detects the planned downscaling of AWS spot instances. However while you can automatically detect the shutdown of spot instances and the upscaling and downscaling of autoscaling groups, it’s been nearly impossible so far to do this for any third-party autoscaling framework.

To enable you to automatically detect the shutdown of spot instances and the scaling up or down of third-party autoscaling solutions, we’ve introduced a new event type. This event type lets you weave Dynatrace into your own autoscaling frameworks and inform Dynatrace about hosts that will soon be terminated.

Weave Dynatrace into any third-party autoscaling framework and avoid alert spam

In Dynatrace version 1.188, we’ve introduced a new event type called MARKED_FOR_TERMINATION. This allows for simple integration into any third-party orchestration of your autoscaling framework.

By pushing this event type to any monitored server, you can inform Dynatrace Davis® that a host is planned for shutdown or will be terminated within the next 60 minutes.

For example, if you’d like to scale down several hosts within the next few minutes, all you need do is send this event to all the hosts you want to terminate. You can either:

• Send the event to each of the hosts that are to be shut down.
• Use a tagging filter to select the group of hosts that are to be shut down.

Once the event is registered on a host, Dynatrace will not alert on any regular or irregular termination of that host within the next 60 minutes. See the example below:

The MARKED_FOR_TERMINATION event, therefore, helps to avoid unwanted availability alerts where hosts are already scheduled for regular termination.

Example: Push the event to a single host or to a group of tagged hosts

The example below shows how to push a MARKED_FOR_TERMINATION event to a single host or to a group of tagged hosts using a simple cURL call:

curl -i -X POST \
   -H "Authorization:Api-Token <YOUR_API-TOKEN>" \
   -H "Content-Type:application/json" \
   -d \
'{
  "eventType": "MARKED_FOR_TERMINATION",
  "description" : "Host will be shut down",
  "timeoutMinutes": 20,
  "attachRules": {
    "entityIds": [
      "HOST-8A3055B13A8353FA"
    ],
    "tagRule": [
     
    ]
  },
  "source": "OpsControl"
}
' \
 'https://<YOUR_ENVIRONMENT>.live.dynatrace.com/api/v1/events'

Example: Push the event to a tagged group of hosts

To send the same termination event to a tagged group of hosts, use the cURL command shown below:

curl -i -X POST \
   -H "Authorization:Api-Token <YOUr_API_TOKEN>" \
   -H "Content-Type:application/json" \
   -d \
'{
  "eventType": "MARKED_FOR_TERMINATION",
  "description" : "Host will be shut down",
  "timeoutMinutes": 20,
  "attachRules": {
    "entityIds": [
    ],
    "tagRule": [
      {
        "meTypes": [
          "HOST"
        ],
        "tags": [
          {
            "context": "CONTEXTLESS",
            "key": "forsaken"
          }
        ]
      }
    ]
  },
  "source": "OpsControl"
}
' \
'https://<YOUR_ENVIRONMENT>.live.dynatrace.com/api/v1/events'

Summary

Automatically scaling your service infrastructure is one of the crucial aspects of managing your IT service infrastructure. Dynatrace automatically identifies spot instances to avoid availability alerts on those compute nodes that are already planned for termination.

With this new event type, your DevOps teams can weave Dynatrace into their autoscaling frameworks and inform Dynatrace about hosts that will soon be terminated. Thus, your teams can easily eliminate false-positive alerts for planned downscaling.