The state of observability in 2024: AI, analytics, and automation

In 2024, digital transformation continues to dominate as a key priority, where organizations seek to build and maintain a competitive edge.

To unlock the agility to drive this innovation, organizations are embracing multicloud environments and Agile delivery practices.

But these environments are, by nature, difficult to manage. They create an explosion of data that is extremely challenging to manually capture, analyze, and act on. As a result, it will become increasingly difficult for IT and security teams to optimize user experiences and improve the resilience of their applications.

The latest Dynatrace report, “The state of observability 2024: Overcoming complexity through AI-driven analytics and automation,” explores these challenges and highlights how IT, business, and security teams can overcome them with a mature AI, analytics, and automation strategy.

Fragmented monitoring and analytics can’t keep up

The continued reliance on fragmented monitoring tools and manual analytics strategies is a particular pain point for IT and security teams.

According to the report, the average multicloud environment spans 12 different platforms and services. On average, organizations use 10 different observability or monitoring tools to manage applications, infrastructure, and user experience across these environments.

This fragmented approach leaves teams struggling to access the answers they need to accelerate innovation and optimize digital services effectively. Some 85% of technology leaders say the number of tools, platforms, dashboards, and applications they rely on adds to the complexity of managing a multicloud environment.

Manual monitoring processes are also too time-consuming, which distracts teams from tasks that create new value for customers and the business. In fact, 81% of technology leaders say the effort their teams invest in maintaining monitoring tools and preparing data for analysis steals time from innovation.

Kubernetes adds to the complexity of technology stacks

Alongside the challenges of managing multicloud environments, IT and security teams struggle to maintain visibility into cloud-native architectures as Kubernetes continues to become the platform of choice for modern applications.

Kubernetes architectures make it easier to quickly scale services to new users and drive efficiency gains through dynamic resource provisioning. However, due to the dynamic nature of Kubernetes, 76% of technology leaders say it’s more difficult to maintain visibility into this architecture compared with traditional technology stacks.

More specifically, organizations have highlighted attack detection and blocking (48%), observability (45%), and cloud cost management (38%) as key challenges as they continue to shift toward Kubernetes.

Organizations are drowning in data

“The state of observability 2024” report also found that the rise of dynamic cloud-native technology stacks has unleashed a flood of data that ITOps and security teams are struggling to contain.

The data has clear value in providing the insights that modern digital enterprises need to drive smarter decisions and better business outcomes. However, cloud environments generate data at such volume and velocity that 86% of technology leaders consider it impossible for teams to cost-effectively capture and analyze it using outdated practices and fragmented monitoring tools.

The report shows that log analytics are a particular challenge, as the long-term storage cost of all this data has begun to overshadow the value organizations can unlock from querying it. As they continue in this way, teams are forced to decide which logs to retain for real-time analytics and which to discard or archive in a lower-cost, less-accessible storage.

To address this, 79% of organizations are currently using or planning to adopt a unified platform for observability and security data within the next 12 months.

A mature AI, analytics, and automation strategy is essential

Organizations are increasingly turning to advanced AI, analytics, and automation capabilities to reduce the complexity of managing their multicloud environments.

Nearly three-quarters (72%) of organizations have adopted AIOps (or AI for IT operations) solutions to tackle the complexity of their multicloud environments. However, 97% of technology leaders say probabilistic machine learning approaches limit the value that AIOps tools deliver given the manual effort required to gain reliable insights.

To unlock all the benefits of AI-driven approaches, organizations need a unified observability platform that combines multiple AI techniques–including causal, predictive, and generative AI. These capabilities enable security and IT teams to power custom applications and analytics use cases to drive smarter decision making and more efficient ways of working.