Capture Business Grade Data with AWS Lambda, Python and Dynatrace Business Events

This post explains how to report and capture business grade data using AWS Lambda and the Dynatrace platform.

Watch Andi Grabner and Klaus Enzenhofer discuss Business Events in their Dynatrace Observability Clinic. This video contains an end-to-end walk through of ingesting, exploring, and analyzing your organization’s critical business data.

Why Do I Need Business-Grade Data?

Before we get to how, first the why: Why would you need or want to capture business-grade data in Dynatrace? On that note, what is business-grade data?

Business Grade Data

The official documentation brilliantly describes business-grade data and the sources from where you can capture business events:

• OneAgent
• Web and Mobile Real User Monitoring (RUM)
• External Sources

Some of the coolest properties of Business events:

• Complete: All data, no sampling, no granulation as they age, no extrapolation, and bizevents are available for 3 years.
• Unified: Use DQL and fetch bizevents and you have them all in one place.
• Topologically-aware: Business events are enriched to be fully aware of your environment: attach them to applications, hosts or any custom device OR leave them as “standalone events” — the choice is yours.

This post will focus purely on that 3rd bullet: ingestion from external sources.

Inspire Me

So what can you do with business events?

• Track financial performance (real revenues or profit margins)
• Track long running or complex and asynchronous “flows”. Even processes that take multiple weeks and are not “joined” easily. If there’s a way to “know” that one thing relates to another in a process, it is trackable via business events.
• Push external data to Dynatrace in hi-fidelity: Voice of Customer scores, call centre queue details, survey responses… The possibilities are almost endless.

Our Scenario

The open source projects we support like Keptn and OpenFeature offer hands-on tutorials. We wanted a way to track the usage of these tutorials:

• How many people started each tutorial?
• How many people finished each tutorial?
• Are users stuck at a certain step?
• How long do users spend, on average, on each step?

We did not need to capture any user-specific data or personally identifiable information (PII).

Technical Limitations

Unfortunately the platform that we use to build these in-browser tutorials does not provide a way to install the OneAgent or place the JavaScript RUM tag.

The only mechanism we had available was the ability to use curl to send a POST to an endpoint.

To complicate matters further, everything about these tutorials is public so there is no way to store or use sensitive data like authentication tokens.

Our Solution

We needed a way to storage sensitive OAuth tokens so we required a middleware layer. We opted for Lambda as a serverless platform because these operations are quick and thus we can minimise our carbon footprint by only having compute resources running precisely when we need them.

1. Perform a curl request into Lambda with the tutorial and page name.
2. Lambda pulls sensitive data from AWS Secrets Manager
3. Lambda transforms incoming data to a CloudEvent
4. Lambda pushes JSON payload (CloudEvent) to Dynatrace
5. Build a dashboard in the Dynatrace platform to visual the data

Step 1: Create an OAuth Client

The Dynatrace official docs have excellent instructions on how to create an OAuth client to authenticate. Follow those instructions which should result in you having:

• A Dynatrace tenant (eg. https://abc12345.live.dynatrace.com)
• An OAuth client ID (eg. dt0s02.ABC12345)
• An OAuth client secret (eg. dt0s02.ABC12345.XYZ543SECRET…)
• An OAuth account URN: (eg. urn:dtaccount:1234-abcd-…)

Step 2: Store Dynatrace Details in AWS Secrets Manager

Obviously the details above provide access to read and write business event data to the Dynatrace platform. They are sensitive and secret, so store them in an encrypted form.

• Go to AWS Secrets Manager and Create a secret. Choose Other type of secret
• Add key / value pairs like this:

• Make a note of the secret name and region name (eg. dynatrace-oauth-details and us-east-1). You’ll need these when you code the Lambda function in the following steps.

Step 3: Create the Lambda Function

Create a new AWS Lambda function. You can use any language you are comfortable with. This tutorial will show Python. You are, of course, free to use packages like requests but we chose to have no external dependencies so our code uses urllib3.

Configure the biz_event_to_send dictionary, notice that the data block is empty. During execution, the code copies in the data from the incoming event[‘body’] (ie. the curl request) into the data block.

In other words:

curl -X POST https://my-lambda \
-d '{"tutorial": "OpenFeatureDemo", "pageName": "step1.md"}'

Becomes:

{
  "source": "lambda",
  "type": "com.tutorial.process",
  "datacontenttype": "application/json",
  "data": {
    "tutorial": "OpenFeatureDemo",
    "pageName": "step1.md"
  }
}

The values for secret_name and the region_name which should reflect whatever you called your secret in AWS Secrets manager.

The lambda_handler function may look complex but actually it is simple:

1. Retrieve secret values from AWS secrets manager
2. Use Dynatrace OAuth to get a temporary OAuth access token
3. Use OAuth access token as Authorization
4. Send a second HTTPS POST to Dynatrace, containing the business event data.
5. Get the response code from Dynatrace and return the status + dummy message of bizevent ingestion fired

import json
import urllib3
import boto3
from botocore.exceptions import ClientError

# Configure the following variables
biz_event_to_send = {
  "source": "lambda",
  "type": "com.tutorial.progress",
  "datacontenttype": "application/json",
  "data": {}
}
secret_name = "ag/jao_oauth_details"
region_name = "us-east-1"

##########################################
# DO NOT MODIFY ANYTHING BELOW THIS LINE #
##########################################

oauth_endpoint = "https://sso.dynatrace.com/sso/oauth2/token"

def get_secret(secret_name, region_name):

    # Create a Secrets Manager client
    session = boto3.session.Session()
    client = session.client(
        service_name='secretsmanager',
        region_name=region_name
    )

    try:
        get_secret_value_response = client.get_secret_value(
            SecretId=secret_name
        )
    except ClientError as e:
        # For a list of exceptions thrown, see
        # https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_GetSecretValue.html
        raise e

    # Decrypts secret using the associated KMS key.
    secret = json.loads(get_secret_value_response['SecretString'])

    return secret

def lambda_handler(event, context):

    # Step 1: Get OAuth Access Token from Dynatrace   
    headers = {
        "Content-Type": "application/x-www-form-urlencoded"
    }

    # Retrieve sensitive info from AWS Secrets Manager
    secret = get_secret(secret_name, region_name)

    tenant_url = secret['tenant_url']
    oauth_client_id = secret["oauth_client_id"]
    oauth_client_secret = secret["oauth_client_secret"]
    account_urn = secret["oauth_account_urn"]
    permissions = "storage:events:write storage:events:read"

    http = urllib3.PoolManager()

    oauth_params = {
        "grant_type": "client_credentials",
        "client_id": oauth_client_id,
        "client_secret": oauth_client_secret,
        "resource": account_urn,
        "scope": permissions
    }

    headers = {
        "Content-Type": "application/x-www-form-urlencoded"
    }

    # Step 1: Get Access Token
    access_token_resp = http.request_encode_body(
          method="POST",
          url=oauth_endpoint,
          fields=oauth_params,
          encode_multipart=False
        )

    access_token_json = json.loads(access_token_resp.data.decode('utf-8'))
    access_token_value = access_token_json.get("access_token")

    # Step 2: Use Access Token to push bizevent
    biz_event_url = f"{tenant_url}/api/v2/bizevents/ingest"

    # Add incoming data from tutorial platform
    # to the cloudevent 'data' field
    biz_event_to_send['data'] = event['body']
    biz_event_url = f"{tenant_url}/api/v2/bizevents/ingest"

    # Add the dynamic OAuth token to the Authorization header as the Bearer token
    biz_event_headers = {
        "Authorization": f"Bearer {access_token_value}",
        "Content-Type": "application/json"
    }

    # POST the business event to Dynatrace
    biz_event_resp = http.request(
        method="POST",
        headers=biz_event_headers,
        url=biz_event_url,
        body=bytes(json.dumps(biz_event_to_send), encoding="utf-8")
    )

    # Retrieve response status from Dynatrace
    biz_event_resp_json = json.loads(biz_event_resp.data.decode('utf-8'))

    return {
        'statusCode': biz_event_resp.status,
        'body': json.dumps('bizevent ingestion fired')
    }

Step 4: Assign Correct Permissions to Lambda

Lambda needs some additional permissions to access AWS Secrets Manager.

1. Go to the configuration tab in lambda and click the hyperlink of your execution role name.
2. Under Permissions policies click Attach policies
3. Search for secretsmanager. Select the checkbox next to SecretsManagerReadWrite and click Add permissions

Your Lambda function now has read/write permissions to retrieve the secrets stored in AWS Secrets Manager.

Step 5: Build a Dashboard in Dynatrace

Business event data is now being ingested into the Dynatrace platform. It is time to visualise and use the data.

1. Go to the platform and choose the Dashboards application.
2. Click New dashboard and click the + and choose Add data

3. Give the tile a name. The first step (pageName) in our tutorials is called intro.md so we’ll query that first. Add a title, type the following Dynatrace Query Language (DQL) query and experiment with the visualisation options.

fetch bizevents
| filter matchesValue(type, "com.tutorial.progress"
| parse data, "JSON:json_data"
| filter matchesValue(json_data[tutorial], "OpenFeatureDemo")
| filter matchesValue(json_data[pageName], "intro.md")
| summarize count(), by: timestamp

DQL Explainer

The DQL above probably needs some explanation. Remember that the incoming business event (in the form of a CloudEvent) looks like this. You can retrieve the raw list of bizevents by commenting out every line except fetch bizevents and change the visualisation to a table.

Dynatrace has automatically enriched the event by adding the following fields: timestamp, event.id, event.kind, event.provider and event.type.

DQL is sequential, so each line processes, adds or removes data that the following lines then have access to.

So:

1. fetch all bizevents
2. Leave only those that have the type field equal to com.tutorial.progress
3. Take the data field and parse it as JSON. Store it as a new field called json_data
4. Filter the event data again and leave only those fields where data.tutorial equals OpenFeatureDemo
5. Filter the event data further and leave only those fields where data.pageName equals
6. Summarize the resulting data by the count of records and bucket them by timestamp (the by clause is useful when you want to put results on a timeline / chart).

Of course, commenting out the lines with // makes them inactive.

Let’s represent that data differently.

Hover over the tile and click Duplicate

Modify the DQL to remove the by clause:

fetch bizevents
// existing lines
| summarize count()

Change the visualisation to the recommended Single Value

Then Customize the visualization. Choose Show label and type people started the tutorial. Untick the Use value formatter to remove the decimal points.

Shaping Up Well!

A Few Copy, Pastes and Edits of DQL Later…

That’s a good looking dashboard!

Curious About the Tutorials?

Want to know more about the open source projects mentioned above (Keptn and OpenFeature)? Want to do these tutorials?

Keptn Demos

• Keptn v1 Demo
• Keptn Kubernetes GitOps Lifecycle Toolkit (demo coming soon)

OpenFeature Demos

1. OpenFeature Demo: Start here. This is for “non-technical users” or those who have never encountered feature flagging.
   It will show a demo application and allow you to experience the “power” and “why” of feature flagging.
2. OpenFeature flagd Demo
   This one is for those looking to create their own in-house feature flag backend system. Before you start coding your own, follow this tutorial.
   flagd is a prebuilt feature flag backend that can shortcut your time to a feature flag backend.
   Think: flagd start --port=8013 --uri=flags.json for a fully-featured system. Cool!
3. Five Minutes to Feature Flags Demo
   Once your feature flag "backend" is running, you'll want to integrate it to your application. This tutorial demonstrates how to do so in an OpenFeature compliant way.
   Instrumenting your application in this way will avoid any vendor lock-in and it's a one-liner to change provider.
   Think: openFeature.setProvider(flagd) changing to openFeature.setProvider(VendorX) and you're back in business.
4. Kubernetes-native Feature Flag Operator
   If you are purely Kubernetes native, the OpenFeature Operator allows pure GitOps management and storage of feature flags as Custom Resources, made available to your pods via annotations.

Capture Business Grade Data with AWS Lambda, Python and Dynatrace Business Events was originally published in Dynatrace Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.