There are no secrets that time doesn’t reveal—how to use the token management API to rotate your secrets!

Remember the saying that “there are no secrets that time doesn’t reveal”? Well, this statement also holds true when it comes to data security—despite all the technological improvements that have been made over the past decade. We all know “a “friend” who accidentally pushed a secret token into a public Github repository.

Managing your secrets and API tokens can be cumbersome and an error-prone process. Today’s microservices and cloud ecosystems require DevOps teams to handle a huge number of distributed API and access secrets. Without a dedicated strategy, you quickly lose control of all the secrets you’re depending on across your environment. One result can be an ever-increasing number of secrets as everyone becomes reluctant to delete old secrets for fear of breaking dependencies. Another more serious outcome can be the exposure of your secrets to the outside world and the introduction of business-critical security leaks.

A well-known strategy for addressing these challenges is the introduction of centralized and secure management of all your secrets along with the regular rotation of secrets that are used for automation tasks. Vault by HashiCorp is one such tool that can help you organize your secrets and make them available for your automation scripts.

With the release of Dynatrace version 1.169, we’ve introduced a new API endpoint for token management that enables the implementation of an API token rotation strategy that helps your DevOps teams to further secure their monitoring environments.

This blog post provides an example of how to use the Dynatrace token management API endpoint to implement a rotation strategy using a HashiCorp Vault server.

Create a token secret in a HashiCorp Vault server

Create a token secret in your vault server. Secret fields are customizable. We’ll store this example Dynatrace API token secret in a field called token, as shown below:

vault kv put secret/dynatrace/automation_script1 token=w_u5xy97RK-gaPKwFaadu

Retrieve a secret for an automation script

Whenever you need to use this secret in one of your automation scripts, use the get command in the Vault command line interface, as shown below:

vault kv get secret/dynatrace/automation_script1

In a shell script, it is, of course, more convenient to receive the only the secret itself instead of all the metdata about the secret. This can be done using the field selector with the Vault get command:

vault kv get -field=token secret/dynatrace/automation_script1

Create a Dynatrace master token to rotate an API token

Now that we can create, update, and retrieve a secret through the Vault server, let’s take a look at how we can automatically rotate API token secrets using the Dynatrace token management API endpoint.

First, let’s create a Dynatrace master token with the Token management permission enabled, which allows us to rotate all Dynatrace API tokens in one place:

Then, save the master token in your Vault server as shown below. This master token will be used by the rotation shell script to automatically rotate this automation client API token:

vault kv put secret/dynatrace/master token=nyjtIaZ8R_-Lw8-kpgp7D

Create a Python script for rotating a token

Now let’s create a quick and dirty Python script that automatically rotates all given Dynatrace tokens and updates the respective secret in the Vault server.

The newly released token management API endpoint allows you to automate the management of all your API tokens. Please refer to the Dynatrace API Explorer for details about each available token management operation:

Rotation of Dynatrace tokens involves the following steps

1. Use a Dynatrace master token to receive the information of a given API token.
2. Create a new API token with the same permissions.
3. Update the API token within the password vault.
4. Delete the old API token.

To create the new API token using the Dynatrace token management API and the Dynatrace master token:

curl -X POST "https://demo.dev.dynatracelabs.com/api/v1/tokens" -H "accept: application/json; charset=utf-8" -H "Authorization: Api-Token nyjtIaZ8R_-Lw8-kpgp7D" -H "Content-Type: application/json; charset=utf-8" -d "{ \"name\": \"My Tenant Token\", \"scopes\": [ \"ReadConfig\" ]}"

Next, to update the API token within the password vault:

curl -i -X POST \

-H "X-Vault-Token:s.aWf0evg0WSwOKKxBdweuQsqU" \

-H "Content-Type:application/json" \

-d \

'{ "data" : { "token" : "nyjtIaZ8R_-Lw8-kpgp7D" } }' \

'http://127.0.0.1:8200/v1/secret/data/dynatrace/automation_script1'

To delete the now outdated token in Dynatrace:

curl -X DELETE "https://demo.dev.dynatracelabs.com/api/v1/tokens/6a98d7bc-abb9-44f8-ae6a-73e68e71812a" -H "accept: application/json; charset=utf-8"

The following Python script automates all the steps above. The script is also capable of rotating an array of multiple secrets in a single run.

It’s pretty simple to automate the rotation of your Dynatrace API tokens once a day or week by registering the Python script as a regular cron job in one of your trusted servers.

See the Python script below:

# Example Python script for automated API token rotation in combination 
# with a HashiCorp vault server

import requests

# CONFIGURATION
# configure list of all secrets we want to rotate
SECRETS = [ 'dynatrace/automation_script1', 'dynatrace/automation_script2' ] 
# HashiCorp vault information
VAULT = 'http://YOUR_VAULT_SERVER_ADDRESS:8200/v1/secret/data/'
X_VAULT_TOKEN = 'YOUR_VAULT_ACCESS_KEY'
# Dynatrace master token must have 'token management' permission assigned 
DT_MASTER_TOKEN = 'dynatrace/master'
# Your Dynatrace environment URL
DT_URL = 'https://YOURENVIRONMENT.live.dynatrace.com'
# END CONFIGURATION

def vault_receive_secret(secret_path, vault, token):
    resp = requests.get(url=vault + '' + secret_path, headers={'x-vault-token' : X_VAULT_TOKEN})
    if resp.status_code == 200:
        return True, resp.json()['data']['data']['token'] 
    return False, {}

def dt_receive_token_info(token, dt_url, dt_api_token):
    resp = requests.post(url=dt_url + '/api/v1/tokens/lookup/', json={'token' : token}, headers={'Authorization' : 'Api-Token ' + dt_api_token})
    if resp.status_code == 200:
        return True, resp.json() 
    return False, {}

# fetch master token '/dynatrace/master' from vault
m_ret = requests.get(url=VAULT + '' + DT_MASTER_TOKEN, headers={'x-vault-token' : X_VAULT_TOKEN})
if m_ret.status_code != 200:
    print('There was no master token found within your vault under path /dynatrace/master')
    exit()
else:
    DT_MASTER_TOKEN = m_ret.json()['data']['data']['token']

for secret in SECRETS:
    print('>Rotate: ' + secret)
    # fetch old secret from vault
    success, old_sec = vault_receive_secret(secret, VAULT, X_VAULT_TOKEN)
    if success:
        # receive secret metainfos from Dynatrace
        success, t_info = dt_receive_token_info(old_sec, DT_URL, DT_MASTER_TOKEN)
        if success:
            # create new secret with same scope as old
            resp = requests.post(url=DT_URL + '/api/v1/tokens', json=t_info, headers={'Authorization' : 'Api-Token ' + DT_MASTER_TOKEN})
            if resp.status_code == 201:
                new_token = resp.json()['token'] 
                # store new token in secret vault
                c_ret = requests.post(url=VAULT + secret, json={ 'data' : { 'token': new_token}}, headers={'x-vault-token' : X_VAULT_TOKEN})
                if c_ret.status_code == 200:
                    # if creation was successful delete old Dynatrace token through Dynatrace API
                    d_resp = requests.delete(url=DT_URL + '/api/v1/tokens/' + t_info['id'], headers={'Authorization' : 'Api-Token ' + DT_MASTER_TOKEN})
                    if d_resp.status_code == 204:
                        print('  Old Dynatrace token successfully deleted')
                        print('  Token rotation successful')
                    else:
                        print('  Failed to delete old Dynatrace token')
            else:
                print('Failed to rotate secret: ' + secret)
        else:
            print('  Api token not found in Dynatrace')
    else:
        print('  Secret not found in vault')
 

This script is also available in our Github repository.

The token management API can be used to automate all kinds of use cases related to API tokens in Dynatrace. The example discussed in this blog post demonstrated one of the most important use cases, rotating your tokens on a regular schedule to avoid security issues. Other use cases could be automatically checking all tokens and deleting those that are no longer used or that relate to a given user.