This OneAgent Extension allows you to collect metrics from your embedded Ganglia instance, the Apache Spark APIs, and/or the Databricks API on your Databricks Cluster.

NOTE: Databricks Runtime v13+ no longer supports Ganglia, please use the Spark and Databricks API Options within the configuration.

This is intended for users who:

• Have Databricks cluster(s) they would like to monitor job status' and other important job and cluster level metrics

• Look to analyze uptime and autoscaling issues of your Databricks Cluster(s)

This enables you to:

• Monitor both job, cluster and infrastructure metrics
• Detect long upscaling times
• Detect and filter Driver and Worker types

1. Define in the configuration which metrics you'd like to collect from your Databricks Clusters

2. Set up a global init script on your Databricks Cluster to download the Dynatrace OneAgent

3. Restart your Databricks Cluster to enable the Dynatrace OneAgent and this extension.

1. Ensure the EEC is enabled on each host. This can be done globally from
   • Settings -> Preferences -> Extension Execution Controller (/ui/settings/builtin:eec.local)

and turning on the first two options.

1. Create Databricks API Token from inside your Databricks Cluster

   • User Settings -> Create API Token

2. Copy your Databricks URL

3. Copy Linux OA Installation wget command from

   • Deploy Dynatrace -> Start Installation button -> Linux button -> Enter or Create Paas Token (#install/agentlinux;gf=all)

**NOTE: Databricks Clusters can go up and down quickly causing multiple HOST entities within Dynatrace. Databricks reuses IP addresses so if you'd like to have the same HOST entities for your clusters you can add this flag --set-host-id-source="ip-addresses" to the OneAgent installation command in your Global Init Script. For Example :

/bin/sh Dynatrace-OneAgent-Linux.sh --set-infra-only=true --set-app-log-content-access=true --set-host-id-source="ip-addresses"

Configuration for Apache Spark & Databricks API Metrics (Recommended)

1. Set up Global Init Script on Databricks Cluster

   • Change Dynatrace Tenant & API Token values
   • NOTE: If your Databricks Cluster does not have network access to your Dynatrace Cluster or ActiveGate, the Dynatrace-OneAgent-Linux.sh file can be manually uploaded to you Databricks DBFS and the script below can be modified to use those locations instead of using the wget command.

   #!/usr/bin/env bash
   
   wget  -O Dynatrace-OneAgent-Linux.sh "https://<TENANT>.live.dynatrace.com/api/v1/deployment/installer/agent/unix/default/latest?arch=x86&flavor=default" --header="Authorization: Api-Token <Installer API-TOKEN>"
   /bin/sh Dynatrace-OneAgent-Linux.sh --set-infra-only=true --set-app-log-content-access=true --set-host-id-source="ip-addresses"
   
   

2. Configure OneAgent Extension in Dynatrace Cluster from

   • Extensions -> Databricks (/ui/hub/ext/com.dynatrace.databricks) -> Add Monitoring Configuration -> Select Databricks Hosts ->
   • Enable Call Spark API Slider
   • Enable Call Databricks API Slider
     • Enter in your Databricks URL and User Token

3. Select which Feature Sets of Metrics you'd like to Capture

4. Restart your Databricks Clusters

5. Verify Metrics show up on the HOST screen of your Databricks Cluster's Driver Node. All the metrics will be attached to that HOST entity.

Configuration for Ganglia (Legacy)

1. Create Dynatrace API with ReadConfig Permissions

2. Set up Global Init Script on Databricks Cluster

   • Change Dynatrace Tenant & API Token values
   • Change DB_WS_URL & DB_WS_TOKEN Values (from steps above)
   • NOTE: If your Databricks Cluster does not have network access to your Dynatrace Cluster or ActiveGate, the OneAgent.sh and extension zip file can be manually uploaded to you Databricks DBFS and the script below can be modified to use those locations instead of using the wget commands.

   #!/usr/bin/env bash
   
   wget  -O Dynatrace-OneAgent-Linux.sh "https://<TENANT>.live.dynatrace.com/api/v1/deployment/installer/agent/unix/default/latest?arch=x86&flavor=default" --header="Authorization: Api-Token <Installer API-TOKEN>"
   /bin/sh Dynatrace-OneAgent-Linux.sh --set-infra-only=true --set-app-log-content-access=true --set-host-id-source="ip-addresses"
   
   # token with 'ReadConfig' permissions
   wget -O custom_python_databricks_ganglia.zip "https://<TENANT>.live.dynatrace.com/api/config/v1/extensions/custom.python.databricks_ganglia/binary" --header="Authorization: Api-Token <ReadConfig API-TOKEN>"
   unzip custom_python_databricks_ganglia.zip -d /opt/dynatrace/oneagent/plugin_deployment/
   
   # Add Databricks Workspace URL Environment Variable
   cat <<EOF | sudo tee /etc/databricks_env
   DB_WS_URL=https://adb-XXXXXXXXX.XX.azuredatabricks.net
   DB_WS_TOKEN=dapiXXXXXXXXXXXXXXXXXXXXXXXXXXX
   EOF
   

3. Create Dynatrace API Token with entities.read & entitie.write permissions.

4. Configure OneAgent Extension in Dynatrace Cluster from

   • Extensions -> Databricks -> Add Monitoring Configuration -> Select Databricks Hosts -> Enable Call Ganglia API Slider (/ui/hub/ext/com.dynatrace.databricks)
   1. Enter in Dynatrace Tenant URL and API Token
   2. Select which metrics you'd like to capture from Ganglia

5. Restart Databricks Cluster(s)

6. Verify Metrics are showing up on the included Dashboard