Log auditing and log forensics benefit from converging observability and security data

Log auditing and log forensics are essential practices for securing apps and infrastructure. But the complexity of cloud-native environments requires a new approach to keep investigations real-time and relevant. Converging observability and security data gives security teams end-to-end visibility into application security issues for real-time answers at scale.

The growing complexity of multicloud environments and ever-increasing number of application vulnerabilities have made it harder than ever to protect against attacks. Log auditing—and its investigative partner, log forensics—are becoming essential practices for securing cloud-native applications and infrastructure.

Many organizations don’t know for months (or years) after a security attack when, why, or how it happened. This represents a significant risk, with the same attack vector repeatedly exploited because the vulnerability wasn’t detected on time. The massive volumes of log data associated with a breach have made cybersecurity forensics a complicated, costly problem to solve.

As organizations adopt more cloud-native technologies, observability data—telemetry from applications and infrastructure, including logs, metrics, and traces—and security data are converging. This alignment provides security teams with the opportunity to track application security issues through the ever-increasing volume and variety of log data. Together, this data makes teams more effective in identifying and responding to critical security incidents as quickly as possible. Overall, this results in a better security posture. Let’s explore how a log auditing and log forensics program can benefit from the convergence of observability and security data.

What is log auditing?

Log auditing is a cybersecurity practice that involves examining logs generated by various applications, computer systems, and network devices to identify and analyze security-related events. Logs can include information about user activities, system events, network traffic, and other various activities that can help to detect and respond to critical security incidents.

Log auditing is a crucial part of building a comprehensive security program. Log auditing helps ensure that teams are following security policies and procedures and that they are identifying and addressing any anomalies or suspicious activities in a timely manner.

What is log forensics?

Log forensics is a practice that involves collecting, analyzing, and preserving log data to identify the time a security incident was initiated, who initiated the incident, the sequence of actions they took, and the impact it had on an organization. It also helps to identify the data that has been affected by an attack and to identify the attack pattern.

Traditionally, log forensics has been based primarily on logs that can help teams to identify the source of a cyberattack, the surface area of damage, and any other relevant details surrounding the nature of the attack. Forensics is crucial for incident response and post-incident analysis. Forensics allows organizations to learn from critical security incidents and take proactive steps to prevent similar recurrences in the future.

Together, log auditing and log forensics are critically important components of security best practices, as they help organizations detect, respond, and recover from security incidents. However, cloud-native technologies have introduced a level of complexity that make a logs-only approach to auditing and forensics limiting. t’s also now critical for organizations to have detailed observability data to improve the quality and context of security investigations.

Cloud complexity introduces new challenges to security audit and forensics

Log auditing and forensics of cloud infrastructure requires a different approach and capabilities compared with traditional on-premises environments. It requires an understanding of cloud architecture and distributed systems, with the goal of automating processes.

Organizations face many challenges when it comes to log audit and forensics, including the following:

• The large volume of data. Cloud infrastructure and applications scale dynamically, which generates a large volume of logs at high This can make it challenging to process, store, and analyze them in real time.
• Distributed and complex topologies. In the cloud, infrastructure components are often distributed across multiple regions, availability zones, and even multiple cloud providers. This can make it difficult to understand the relationships between different entities, identify root cause, and determine resolution.
• Incomplete. Siloed data, incomplete traces, lack of context, and insufficient instrumentation and metrics are all factors that lead to organizations needing more trustworthy, automatable answers in critical security investigations.
• Skills and expertise. Efficient and effective log audit and forensics practices can require specialized understanding of cloud environments, applications, and log formats. This expertise may exist in teams that may not have the bandwidth to provide them for security incident response.
• Time and resources. Typically, the process of analyzing log data includes data rehydration, reloading, reindexing, or re-ingesting, which takes time and resources. Through this lengthy process and due to the pressure of an ongoing security investigation, teams are often expected to provide prompt answers to these questions, which conflicts with the additional time needed to conduct a precise analysis of all security incidents.

Organizations need to be aware of these challenges and take steps to address them to ensure their log audit and forensics programs are effective in detecting and responding to critical security incidents. An observability approach, one that covers logs comprehensively along with metrics, end-to-end traces, and real-time context, can enable teams to keep pace with the complexity.

Need security answers now? Observability context can help provide them quickly

Let’s consider an organization that is conducting an investigation of a current or suspected security incident. The company has applications that produce a high volume of logs per day, and a new wave of attacks has targeted this company’s sector. In the aftermath of a critical zero-day vulnerability, such as Log4Shell, it’s vital for teams to determine whether the vulnerability affects them and to identify signs of compromise. Since there is a possibility of the vulnerability existing for weeks or even months prior to discovery, an organization’s investigation should be thorough and span all logs, including a historical span.

Teams can quickly answer questions such as the following by querying not only logs but also observability data (including traces and metrics) and topology context.

• Were there attack attempts? (for example, query web server logs from the past year for specific attack strings containing ).
• Are there any indicators of compromise? (for example, query topology to cross reference entity information to narrow down attacked services to those that are running Java)
• To what extent could we be compromised? (for example, collate which and how many Java applications were attacked)
• Did we lose any critical data? (for example, query logs for all attacked java applications to find if there were there any suspicious outgoing connections)
• What data did we lose? (for example, what was the payload on these outgoing connections?)
• How can we protect ourselves against future attacks? (for example, query application traces to understand how an attack progressed through the application)

How to boost log auditing and log forensics with observability data from Dynatrace

Log auditing and log forensics can be intimidating and complex. But with a platform approach to log analytics based on observability at a cloud-native scale, organizations can accomplish much more.

Dynatrace Grail alleviates the burden of identifying security risks in multicloud and hybrid cloud infrastructures. Grail magnifies Dynatrace Application Security capabilities by enabling teams to make boundless queries of all observability data types. Dynatrace Query Language (DQL) offers a superior approach to query data and includes a unique high-performance Dynatrace Pattern Language (DPL) for easier and faster parsing and data matching. With Grail, Dynatrace customers can now leverage a unified point of governance for their DevSecOps strategies by doing the following:

• Automatically contextualizing security data with Dynatrace observability insights.
• Performing collaborative analysis through Dynatrace Notebooks, a method of context-based data sharing.
• Automating workflows for repetitive tasks and utilizing customized Dynatrace data analysis apps with the new AppEngine.
• Operationalize data findings in one unified platform, connecting data from development and runtime environments.
• Eliminate data silos between DevSecOps teams and increase data sharing and analysis capabilities.

With these capabilities, Grail enables customers to extend the existing OneAgent-focused Application Security solution with an end-to-end data analytics-driven approach, and unique convergence of observability and security data in context. Dynatrace Grail can help organizations overcome cloud complexity through instant, cost-efficient, AI-powered analytics for observability, security, and business data at any scale.