How StackStorm enables auto-remediation with Dynatrace

Ensuring the performance and 24/7 availability of large-scale software applications in highly dynamic environments is a huge challenge. AI-powered, full stack performance monitoring of your complete environment allows you to detect problems that may impact the experience of your customers and, as a consequence, the success of your digital business. Monitoring alone, however, isn’t always enough to keep your applications running at peak performance. This is why Dynatrace supports self-healing IT and automated remediation for when problems arise. Using Dynatrace with an automation platform like StackStorm enables you to define and run remediation actions as soon as problems are auto-detected by Dynatrace.

StackStorm automated remediation

The Dynatrace StackStorm integration provides you with powerful auto-remediation capabilities. StackStorm is an event-driven open-source automation platform built for DevOps that allows you to define rules and workflows to automate your IT processes. StackStorm can troubleshoot detected problems, run automated rules to address detected problems, and notify human IT operators if required. For example, you might want to automate and customize your CI/CD pipelines or send information about important system events on Slack—there are numerous integrations available for StackStorm.

Auto-remediation with Dynatrace

Before showing you how to leverage StackStorm in combination with Dynatrace to self-heal your environment and your applications, let’s set up a StackStorm installation.

Install StackStorm

StackStorm provides a convenient one-line installation on 64-bit Linux machines. It’s best to begin with a clean system so that StackStorm installation doesn’t interfere with other software.

1. Enter the following command to install StackStorm on your VM or on-premise Linux machine:
   $ curl -sSL https://stackstorm.com/packages/install.sh | bash -s -- --user=st2admin --password='Ch@ngeMe'
2. An installation success message appears. Run the version check as follows to verify the installation:
   $ st2 --version
3. Once you see your StackStorm version in the console, you can access your StackStorm dashboard directly via the browser by simple entering your hostname. Login to your StackStorm dashboard by providing your credentials that you entered in Step 1.
   

Define remediation rules

StackStorm executes remediation actions based on specific triggers. A trigger and its associated action is called a rule. Rules are like “if-this-then-that” workflows that trigger each time a condition is met. Examples of conditions include a specific time, a time interval, or the calling of a specific URL (webhook).

For this auto-remediation scenario, let’s leverage the webhook integration via the StackStorm REST API. This allows us to trigger actions directly from the Dynatrace notification integration.

To create a new rule

1. Type a name for the rule and the name of the pack that contains the available triggers and actions. For this example, we’ll use the built-in trigger core.st2.webhook.
2. Define a name (webhook demo in this example) and url (webhook-demo) for the Trigger. When this URL is called, this rule will be triggered, and the defined actions will execute.
3. Define the action to be executed when the trigger URL is called. For this scenario, the action should restart the Apache process since it’s assumed that this trigger will be called whenever there are problems with the Apache process. Set the action name to linux.service, set act to restart, and set the service to apache2. Remember to select the sudo checkbox, since the restart may fail with insufficient privileges.
   

Execute a remediation rule via StackStorm

Note: Before trying out this webhook, be sure to create an API-key that allows for authorization within the REST API. To create an API key via the command line, type the following command:
$ st2 apikey create -k -m '{"used_by": "my webhook integration"}'

For more details about API keys, see StackStorm documentation.

Restart your Apache process with a simple REST API call. The REST endpoint must be called using the POST HTTP method. Postman is used in the example below to generate the POST request. Simply enter the URL, set the request type to POST, and include your St2-Api-Key in the header or as a parameter.

If the response you receive has an empty body, then you know that your code was executed successfully.

Integration with Dynatrace

To set up problem-notification integration with StackStorm

1. In Dynatrace, navigate to Settings > Integration > Problem notifications.
2. Click Set up notifications and select Custom integration.
3. Enter a Name for the integration and the Webhook URL, looking similar to https://<yourhostname>/api/v1/webhooks/<url-of-your-webhook>
4. TypeSt2-Api-Key in the Additional HTTP headers section.
   Note: You don’t need to specify a payload. A POST request without payload is enough for the webhook to trigger the defined action in StackStorm.
5. Click Send test notification to confirm that you’ve set up the integration correctly.
6. Click Save to store the custom integration.

Your StackStorm integration is now complete. Within StackStorm you should now see the rules that have been triggered either by Dynatrace or Postman.

Conclusion

In this blog post, you learned how to integrate StackStorm with Dynatrace and create rules that execute when triggered via the StackStorm REST API. So each time Dynatrace detects a problem, StackStorm can automatically run remediation actions. With this integration, you can define your own remediation actions and enable your self-healing applications.