OpenPipeline: Simplify access to critical business data

Effective business analytics relies on frictionless access to business data—wherever it exists. Dynatrace OpenPipeline™ makes it easy to extract business data from log files, expanding opportunities for business reporting and business process monitoring use cases.

There’s a goldmine of business data traversing your IT systems, yet most of it remains untapped. To unlock business value, the data must be:

• Accessible from anywhere. Data has value only when you can access it, no matter where it lies.
• Easy to access. Simplicity accelerates time-to-value and reduces implementation and maintenance costs.
• Real time. Agile business decisions rely on fresh data.
• Precise. Accuracy provides the confidence needed for business automation.
• Contextualized. Metadata enrichment improves collaboration and increases analytic value.

The Dynatrace® platform continues to increase the value of your data—broadening and simplifying real-time access, enriching context, and delivering insightful, AI-augmented analytics. Our Business Analytics solution is a prominent beneficiary of this commitment.

Business events: Delivering the best data

It’s been two years since we introduced business events, a special class of events designed to support even the most demanding business use cases. Since then, many of our customers have embraced the opportunity to explore and adopt new business analytics use cases. Most of these leverage the unique capability of Dynatrace OneAgent® to extract business data from in-flight application payloads—without writing any code. Other data sources, including APIs and log files—are used to expand access, often to external or proprietary systems. Enhancing access to business data from log files is an important priority, and OpenPipeline makes this a reality.

Dynatrace OpenPipeline is a new stream processing technology that ingests and contextualizes data from any source. You can now use OpenPipeline to extract business events from log files, complementing OneAgent as a primary source of business data. This is especially important when legacy or proprietary systems don’t meet OneAgent’s environmental prerequisites; in these cases, log files are usually the preferred source of business data.

In fact, it’s likely that some of your critical business systems already write business data to log files. For years, logs have been the dominant approach many observability vendors have taken to report business metrics on dashboards. You might still be using one of these tools despite the drawbacks, which include a lack of IT context, constraints on data privacy and data retention, and, as the only convenient source of business data, a limiting lack of breadth. OpenPipeline makes migrating these use cases to Dynatrace easy, helping you overcome these limitations to realize greater value.

Use case categories

It’s helpful to categorize the nearly unlimited use cases covered by Dynatrace Business Analytics. Two of the most important categories are:

• Business reporting, analytics, and automation. Track business metrics, key performance indicators (KPIs), and service level objectives (SLOs)—automatically and in context with IT infrastructure and services—to promote collaboration between business and IT teams.
• Business process monitoring and optimization. Monitor and optimize business processes with real-time visibility into process KPIs and detailed analytics for each step to improve customer satisfaction, increase operational efficiency, and reduce cost.

Most of the use cases in these two broad categories benefit from the flexibility that comes from multiple available sources of business data. For example:

• An airline’s reporting and analytics dashboard includes data showing flights, passengers, available seats, passenger load, revenue per passenger, flight crew staffing, arrival delays, and customer satisfaction metrics. The data may come from a mix of systems, including a departure control system (DCS), an airline reservation system (ARS), a legacy inventory control system, and a SaaS-based Voice of the Customer (VoC) solution.
• A financial institution’s loan origination business process includes a series of process milestones, including loan application, credit scoring, application review, contract generation, CRM, and loan funding. All of these steps are critical components of the process, likely to be implemented using different systems. Furthermore, to understand process health, individual steps must be viewed in the context of the entire process. For this, we use Business Flow to track, analyze, and optimize complex business processes, treating the process as an observable IT and business asset.

Use OpenPipeline to extract business events from logs

Using OpenPipeline, you can ingest log data into Dynatrace from a wide range of sources, including OneAgent, Extensions, the Log ingest API, and OpenTelemetry. The pipeline includes a configurable business event processor as part of the data extraction stage; this processor is used to extract and convert relevant business data into business events.

There are many benefits to extracting business events from log files. These include:

• Improved data privacy. Sensitive business data is separated from IT observability data.
• Improved data management. Fine-grained permission and retention policies can be tailored to individual business use cases.
• Reduced storage and query overhead for business use cases. Business events are a small, often negligible subset of log data.
• Simplified and enhanced analytics efficiency. Business events from log files are unified with business events from OneAgent, RUM, and API.

Extracting business events from logs: Configuring OpenPipeline

The OpenPipeline app guides you through the three configuration steps required to extract business events from log files:

1. Identify the source of the log file.
2. Define OpenPipeline’s dynamic routing matching criteria used to route the log lines of interest to a second pipeline.
   
3. Create the target pipeline. Use the Data extraction tab to define the matching condition that creates the business event. You need to include static or dynamic Event type and Event provider fields.
   

Within the target pipeline, you can also define processing rules, extract metrics, set the security context, and define retention periods. Log data is then processed accordingly, stored in Dynatrace Grail™ causational data lakehouse, and available for your Business Analytics use cases.

The value is in the data

OpenPipeline broadens Dynatrace’s unified access to the best data to deliver the best value. Now’s the time to see how it can benefit your organization.

For more details about OpenPipeline read this Dynatrace OpenPipeline blog post.