Docker log analytics has arrived!

With the introduction of OneAgent version 107 it’s now possible to access Docker container log files related to specific applications (i.e., process groups).

Different approaches to gathering log data

The Application Performance Management market has matured over the past few years. There are now a number of intelligent, automated approaches available for managing application lifecycle. Still, most applications continue to rely on logging as a foundation for diagnostics and tracing. While this may not be bad practice, it does create some challenges. For example, writing logs in dynamic, microservices-based environments involves the problem of diagnostics data persistence. Physical disks often aren’t available, containers are volatile, and saving logs as local disk files isn’t an option because such files disappear when the containers they reside in are shut down.

Different infrastructure and platform vendors approach this challenge from different angles. Some vendors advise that you should export logs to external storage. Others gather logs within proprietary frameworks, making them available and persistent long after the services they relate to have stopped.

Docker solves this dilemma by gathering logging information from all containers’ standard input/output errors and saving the data to a central repository that’s governed by a so-called logging driver. By default, the json-file logging driver is used and log messages are made available via the docker logs command. This is inconvenient, however. The driver only works for individual containers (you must provide an ID as a command line parameter), filtering is problematic, and output handling is time-consuming. A better option may be to send log information to external sources using other available drivers. Although this approach requires separate external (often expensive) tools to store and manage the data. To learn more about the issues you may encounter when confronting the challenges of Docker logging, have a look at this great blog series by Yoanis Gil Delgado.

The Dynatrace approach

Here at Dynatrace, we’ve been working hard to help you tackle the management challenges involved in Docker monitoring. Our goal has been to provide an easy-to-use solution for analyzing, filtering, and parsing all the log information that applications generate, regardless of the number and stability of your Docker containers. We’ve also strived to save you the burden of setting up expensive storage solutions for your log data.

With OneAgent version 107 it’s now possible to access all Docker container log data related to specific applications (i.e., process groups). The workflow is simple and there’s no need for you to buy additional tools or pay for storage.

To analyze log data for a specific application

1. Select Technologies from the navigation menu.
2. To filter the Process group list at the bottom of the page, select the technology type that your application’s process group runs on  (you may need to scroll down to view the list).
3. Select the process group you want to analyze.
   The process group list entry then expands, revealing an overview chart and the process group’s Process group details button.
4. Click the Process group details button to view this process group’s details page (see example below).

On the example Process group details page above, note the high dynamics of this process group, expressed in the highly variable number of detected processes (i.e., Docker containers). Because log files for this process group have been detected by Dynatrace, a Log files tab is provided. The log files for this process group are available for download from the Log files tab (see example below). Note that because this monitored process uses the json-file logging driver provided by Docker to generate its logs, a Docker container logs log file is available for download. Docker container log files can also be accessed via the Log viewer.

Easy access to log diagnostics data

Best of all, with Dynatrace Docker log analytics, you don’t need to know on which container images a process group runs on. There’s no need to know the container names, IDs, or even the names of the hosts where the images are running. As long as the log data still exists in Docker’s host logging-driver history, you can find the data with the Log viewer.

Each log entry includes information about the corresponding container image and ID that logged the message and the type of output that was used. You can use this information for query filtering and thereby focus your analysis only on relevant containers and images.

Note that container IDs change with each new line in the log message stream (see example below). Even though this application (i.e., process group) is distributed across multiple hosts, processes, and containers, all log message associated with this application are analyzed and presented as a single virtual log file.

We hope that this addition to Dynatrace log analytics functionality saves you time in quickly accessing application-specific log diagnostics data and in triaging application performance issues. With this enhancement, you can now navigate from a problem details page that indicates a faulty application directly to the log files of the application’s supporting processes, in just a few clicks—regardless of how dynamic your underlying infrastructure is.