Monitoring Docker container environments

If you’re responsible for running and monitoring Docker environments, you likely have two main concerns: (1) are the applications and services deployed on your clusters performing well, and (2) are your clusters healthy and properly sized for running your apps with the required workload?

Monitoring dockerized applications, not just containers

The metrics you get from your orchestration tool (e.g., Kubernetes, Openshift, or Mesos/Marathon) or by Google’s cAdvisor can help you to get a basic understanding of your cluster and container performance. However, you can’t determine if the applications and services deployed within your containers are working properly by analyzing only process-, container-, and host metrics. The information you need to understand the health of your containerized applications and services must be collected and instrumented from the applications themselves (for example, from the byte-code executed within the JVM).

In microservices environments, the complexity of deployments typically increases because orchestration tools dynamically start and stop containers on cluster nodes based on the health checks and thresholds that have been defined for each microservices container. A container can virtually “hop” from one host to another host, and the number of containers per app or microservice can change dynamically over time. This makes it difficult to draw conclusions about an application’s or service’s health. One solution for tackling this problem is analyzing and combining the monitoring data from the infrastructure level (which includes information about containers, processes, and hosts), and the application level, from within an application or service’s execution path and data.

Dynatrace addresses this problem with a full-stack monitoring approach in which all data from the infrastructure and application/service levels are integrated, enabling you to quickly get to the actual root cause of detected application and service problems.

OneAgent for container and application monitoring

To monitor Docker environments with Dynatrace, you only need to run Dynatrace OneAgent on your Docker hosts—either as a separate container or by installing Dynatrace OneAgent directly on the hosts. You don’t need to embed OneAgent into any of your application images or have it inherited from a special base image.

To run Dynatrace OneAgent as a Docker container, issue the following docker run command on all your Docker hosts. Be sure to replace the REPLACE_WITH placeholders in this command with the respective information explained above:

docker run -d --privileged=true --pid=host --net=host --ipc=host -v /:/mnt/root dynatrace/oneagent TENANT=REPLACE_WITH_ENVIRONMENT_ID TENANT_TOKEN=REPLACE_WITH_TOKEN SERVER=REPLACE_WITH_CONNECTION_ENDPOINT

Note that the connection endpoint in REPLACE_WITH_CONNECTION_ENDPOINT is https://REPLACE_WITH_ENVIRONMENT_ID.live.dynatrace.com . If you’re using Dynatrace Managed the connection endpoint is https://<YourManagedServerURL>/e/REPLACE_WITH_ENVIRONMENT_ID

Monitoring Mesos/Marathon

To use Dynatrace to monitor apps running in Mesos clusters, it’s recommended that you (1) deploy Dynatrace OneAgent on all Mesos agent nodes (formally known as “Mesos slaves”) by means of a Marathon app deployment, and (2) install Dynatrace OneAgent on all Mesos master nodes, as explained below.

If you’re using DC/OS to manage your Mesos clusters, you can take advantage of the Dynatrace package that we’ve made available within the DC/OS Universe. The DC/OS Universe package automatically deploys Dynatrace to all your Mesos agent nodes.

If you’re not using DC/OS, you can run Dynatrace OneAgent as a Marathon app. See the following example:

cat <<- EOF > dynatrace-oneagent.json
{
  "id": "dynatrace-oneagent",
  "cpus": 0.1,
  "mem": 256,
  "instances": REPLACE_WITH_NUMER_OF_NODES,
  "constraints": [["hostname", "UNIQUE"], ["hostname", "GROUP_BY"]],
  "container": {
    "type": "DOCKER",
    "volumes": [
      {
        "containerPath": "/mnt/root",
        "hostPath": "/",
        "mode": "RW"
      }
    ],
    "docker": {
      "image": "dynatrace/oneagent",
      "forcePullImage": true,
      "network": "HOST",
      "privileged": true,
      "parameters": [
        { "key": "pid", "value": "host" },
        { "key": "ipc", "value": "host" }
      ]
    }
  },
  "args": [
    "TENANT=REPLACE_WITH_ENVIRONMENT_ID",
    "TENANT_TOKEN=REPLACE_WITH_TOKEN",
    "SERVER=REPLACE_WITH_CONNECTION_ENDPOINT"
  ]
}
EOF

Be sure to replace all REPLACE_WITH placeholders in the dynatrace-oneagent.json file with the respective information explained above. The connection endpoint in REPLACE_WITH_CONNECTION_ENDPOINT is https://REPLACE_WITH_ENVIRONMENT_ID.live.dynatrace.com. If you’re using Dynatrace Managed the connection endpoint is https://<YourManagedServerURL>/e/REPLACE_WITH_ENVIRONMENT_ID

Now, send an HTTP POST request to the Mesos master “leader” to deploy the Marathon app with Dynatrace OneAgent.

curl -X POST -H "Content-Type: application/json" http://your-mesos-master:8080/v2/apps -d@dynatrace-oneagent.json

Deploy Dynatrace OneAgent on Mesos master nodes

Marathon doesn’t allow you to deploy apps to master nodes (except nodes that are tagged as both master and agent). This is why you need to manually install Dynatrace OneAgent on all Mesos master nodes that haven’t additionally been configured as Mesos agents. Please use the default Linux installer for this.

Monitoring Kubernetes

To use Dynatrace to monitor apps that run within Kubernetes clusters, it’s recommended that you (1) deploy Dynatrace OneAgent on all Kubernetes nodes (previously known as “minions”) by means of a DaemonSet, and (2) install Dynatrace OneAgent on the Kubernetes master nodes. See the example below:

cat <<- EOF > dynatrace-oneagent.yaml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: dynatrace-oneagent
spec:
  template:
    metadata:
      labels:
        run: dynatrace-oneagent
    spec:
      hostPID: true
      hostIPC: true
      hostNetwork: true
      volumes:
      - name: host-root
        hostPath:
          path: /
      containers:
      - name: dynatrace-oneagent
        image: dynatrace/oneagent
        args:
        - "TENANT=REPLACE_WITH_ENVIRONMENT_ID"
        - "TENANT_TOKEN=REPLACE_WITH_TOKEN"
        - "SERVER=REPLACE_WITH_CONNECTION_ENDPOINT"
        volumeMounts:
        - mountPath: /mnt/root
          name: host-root
        securityContext:
          privileged: true
EOF

Be sure to replace all REPLACE_WITH placeholders in the dynatrace-oneagent.yaml file with the respective information explained above. The connection endpoint in REPLACE_WITH_CONNECTION_ENDPOINT is https://REPLACE_WITH_ENVIRONMENT_ID.live.dynatrace.com. If you’re using Dynatrace Managed the connection endpoint is https://<YourManagedServerURL>/e/REPLACE_WITH_ENVIRONMENT_ID

Now, run OneAgent by creating a DaemonSet using the Kubernetes CLI.

kubectl create -f dynatrace-oneagent.yaml

The command above runs Dynatrace OneAgent on all Kubernetes workers. You may also want to run Dynatrace OneAgent on the Kubernetes masters. Please use the default Linux installer for this.

Ready to get started?

Learn what full-stack monitoring for your dockerized applications in Kubernetes and Mesos/Marathon environments means for yourself. Simply run the OneAgent Docker image as illustrated above and sit back.