Managing observability and business-data storage is essential to getting data-driven answers and setting up automation workflows. Traditionally, these efforts have led to compromises in cost, business requirements, and compliance with applicable regulations. And relying on an archive-and-retrieve solution isn’t an option because it’s slow and expensive to get value from your data. Thankfully, the new custom buckets in the Dynatrace Grail™ data lakehouse keep you in control of your data, make your data available at all times, and abolish data management overhead.
Optimize cost and availability while staying compliant
Observability data like logs and metrics provide automated answers, root cause detection, and security issues.
Customer decisions about data retention are often determined by important security, privacy, and legal issues. Customers must comply with internal and external policies and regulations that might demand them to keep specific data stored for a minimum period of time (for example, audit logs). However, the opposite is also true—in some cases data must be deleted after a certain period of time. This is the case when a company no longer has legal grounds to retain its customer data, as outlined in privacy protection regulations.
Often customers make business decisions about data retention based on the value they get from keeping historical data and the associated data retention costs. This means compromising between keeping data available as long as possible for analysis while juggling the costs and overhead of storage, archiving, and retrieval. For example, suppose data has to be retained for a longer period because of legal or business reasons. In such a case, the data is archived in cold storage where it can only be accessed for analysis following a delay, re-ingestion into a log analysis tool, and reindexing to prepare the data for analysis.
Ultimately this leads to a lose-lose situation for customers—they have to pay for and maintain data storage but they can’t get answers from their data quickly and effortlessly when needed.
Grail gives you control and the answers from data
With Grail, Dynatrace provides control over data retention and access policies for granular portions of data called “buckets.” This allows you to design data management and retention policies based on individual requirements, starting from days of retention up to a decade.
By introducing control over data retention, Dynatrace doesn’t impose any additional complexities. Even with the flexibility of buckets, there is no additional overhead of data storage management, no archiving, no retrieval from archives, and no performance degradations when using retained data for answers.
The price of data retention is always transparent and uniform, based on the number of days the data is retained, with no hidden fees for managing data. The same applies to querying data with transparent pricing based on read-data volume, with no extra costs for querying older data.
Use buckets for any use case in a secure way
When using Log Management and Analytics or Business Analytics with Grail, you can create custom buckets with specified data-retention periods. For example, you can route incoming log data to a specific bucket so selected team members can access it.
App developers might need to read logs from their environment for debugging purposes, but only for a specific timeframe. With Grail, it’s easy to create a bucket with ten days of retention time and provide all developers access to the data.
Infrastructure teams may need to work with host logs from recent months or quarters. To do this, infrastructure logs can be routed to a bucket with a retention period of three months to a year.
Local regulation often requires that security or audit logs be retained for 7 to 10 years. Such logs can be collected in a bucket with the required retention period, with only the security operations team having access to the logs.
A bucket can be wiped if, at any point in time, there is a need to delete the data stored in it. The reasons for this can vary from a changing business justification to data-privacy regulations. It’s also possible to extend or shorten a bucket’s retention period, which impacts how long existing data in a bucket is stored.
To support configuration-as-code for enterprise environments, creating, updating, and deleting data buckets in Grail is available through an API endpoint. This allows you to create new buckets, change the retention period of existing buckets, or delete buckets via an API call.
Bucket management follows a strict permission policy approach, where only users with corresponding permissions can create, update, or delete buckets. Every API call is saved in audit logs to document the complete picture of activities in your environments.
Get value from your data with the Dynatrace Grail today
- Sign up for a free trial account to start using Dynatrace today.
- Custom bucket support is available beginning with Dynatrace version 1.265; it will be rolled out gradually over the coming months to new and existing customer environments with Grail, and to free trial user accounts.
- Learn how Grail data lakehouse combines logs, metrics, traces, and Smartscape to power the Dynatrace platform and unlock boundless analytics
- Perform any analysis any time with Dynatrace Log Management and Analytics
Looking for answers?
Start a new discussion or ask for help in our Q&A forum.
Go to forum