How Dynatrace uses Dynatrace to combat the Log4j vulnerability (Log4Shell)

On December 9, 2021, the first indicators of the Log4j vulnerability (Log4Shell) began to reverberate across the world. As organizations started learning about Log4Shell from news feeds, blogs, and social media, the Dynatrace security team—and Dynatrace Application Security—kicked into action. With its continuous surveillance of Dynatrace’s own production environments, Dynatrace Application Security enabled our security team to detect the Log4j vulnerabilities in real-time and implement immediate remediation at scale.

Log4Shell is a software vulnerability in Apache Log4j 2, a popular Java library for logging information in applications. The vulnerability enables a remote attacker to execute arbitrary code on a service on the internet if the service runs certain versions of Log4j 2.

With the highest-level CVSS severity score of 10 out of 10, Log4Shell (CVE-2021-44228) has a wide-ranging effect given the ease with which attackers can exploit Log4j, and the impact it could have if exploited. Because logging with Log4j 2 is so common across platforms and services and attacks on the Log4shell vulnerability are easy to perform, remediating its effects can be complex and potentially time-consuming.

Dynatrace immediately identified Log4Shell—and the first attempted attack patterns

As our team discussed attack detection and mitigation strategies for Log4Shell, we received the first real-time updates in our vulnerability feed from Dynatrace Application Security.

Dynatrace Application Security automatically found our risk exposure to Log4Shell in production in real-time. With the Dynatrace platform’s real-time topology map and intelligence about system states, locations, and dependencies, we were able to instantly understand all affected entities and the systems they depend on.

Combined with Davis® AI’s automatic prioritization of the most critical apps affected, we were able to immediately kick off remediation workflows across multiple engineering teams. This ability to instantly prioritize was critical for our rapid remediation response.

Within a few hours, we had fixed all publicly available instances potentially affected by the vulnerability and had planned mitigation measures for all other affected software components.

The Dynatrace Log4j remediation strategy: Why Dynatrace Application Security is crucial

When facing a situation like Log4Shell, it is important to focus on three topics:

• Vulnerability detection and mitigation
• Incident detection and response
• Coordination and communication

Vulnerability detection and mitigation

Dynatrace Application Security automatically provided us with a prioritized list of potentially affected systems, their exposure, and purpose, which enabled our team to direct remediation.

Using the Application Security remediation tracking screen for each vulnerability, we could see exactly whether each affected process still had the vulnerability loaded. As soon as we resolved all instances, Dynatrace automatically closed the vulnerability report. Dynatrace automatically re-opens the vulnerability report if it detects a new instance with the same problem.

For those systems where an upgrade was not possible, we used data from Dynatrace’s automatic vulnerability reports to recommend mitigation actions. Dynatrace’s automatic integration with our ITSM tools enabled our teams to operationalize these remediation workflows. Each engineering team had dedicated security team members as points of contact in case of questions, necessary advice, and status updates.

As the additional Log4j vulnerabilities (CVE-2021-45046, CVE-2021-45105, CVE-2021-44832) and their patches (Log4j 2.16.0, 2.17.0, 2.17.1) rolled out, Dynatrace Application Security provided real-time information about resolved vulnerabilities so we could verify successful deployments.

Incident detection and response

We used Dynatrace capabilities to set up Log4Shell-specific attack monitoring and incident detection. We rapidly identified Log4shell-specific log patterns and, using Dynatrace’s log analytics and alerting capabilities in addition to the Dynatrace Application Security Module, we configured alerting mechanisms for attacks on our environments.

Because the vulnerability is triggered by calls to the lookup function of the class org.apache.logging.log4j.core.lookup.JndiLookup, we also configured a custom service in Dynatrace. With the metrics and alerting also bound to this service, we used Dynatrace’s visibility into the underlying code to quickly set up a dedicated alerting mechanism for any potential successful attacks on this critical vulnerability.

Using details provided by Dynatrace on suspicious events, we performed an extensive analysis of logs for the past weeks to ensure there were no successful attacks on this vulnerability. Dynatrace AI automatically analyzes and prioritizes alerts and eliminates false positives, which was essential in classifying potential alerts as false alerts. These immediate, prioritized answers enabled us to remediate our systems very quickly and confidently.

Coordination and communication

Starting on December 10, the office of the CISO, the security team, engineering teams, and our customer support teams had multiple status updates every day until we had patched all systems against CVE-2021-44228, CVE-2021-45046, and CVE-2021-45105. This enabled us to coordinate and swiftly mitigate potential risks to our environments and communicate clearly to our customers.

As the Log4Shell landscape develops, Dynatrace adapts

For many organizations, the Log4Shell vulnerability may take months or years to resolve. In contrast, using Dynatrace Application Security enabled our security team to automatically gain insight into runtime vulnerabilities across our own complex, distributed multicloud environment.

The Dynatrace platform also helped our team address the most critical systems first. With a real-time-updated and automatically prioritized view of all affected systems, we could easily tier our remediation efforts. As additional Log4j vulnerabilities and patches have emerged, Dynatrace Application Security enables us to stay agile and roll out each update strategically and efficiently.

Beyond this event, security teams need to prepare for other vulnerabilities and attacks. The Dynatrace security team maintains our strategy to secure Dynatrace systems and those of our customers against Log4Shell. The Dynatrace platform gives us the full-stack observability, intelligence, and agility we need to address Log4Shell’s cascading effects. With Dynatrace, we are prepared for the unknown unknowns and whatever comes next.