Privacy spotlight: Ensure compliance by hard deleting individual records in Grail

Dynatrace introduces record-level hard deletion to more effectively comply with end-user deletion requests in line with privacy laws. By ensuring only relevant data is efficiently and permanently removed, deletion requests help drive innovation and don’t negatively impact your data quality.

Data deletion might seem simple, but it’s critical in data management and privacy laws. It’s not just about hitting the Delete button; it’s about securely meeting compliance requirements while ensuring data integrity. Dynatrace Grail™ is a data lakehouse optimized for high performance, automated data collection and processing, and queries of petabytes of data in real time. A data lakehouse is schema-on-read and indexless, making deletion operations complex. Adding to the technical challenges, effective deletion involves a combination of policies, procedures, and technologies to ensure data is appropriately managed throughout its lifecycle.

Strategically handle end-to-end data deletion

Two key elements form the backbone of an effective deletion strategy in Dynatrace SaaS data management: retention-based and on-demand deletion. Retention-based deletion is governed by a policy outlining the duration for which data is stored in the database before it’s deleted automatically. The retention period can vary based on the nature of the data, regulatory requirements, and the business’s specific needs. With Dynatrace Grail™, you can customize data retention periods with day-level granularity for up to 10 years.

On-demand deletions are initiated in response to specific events or requests. For instance, if data is mistakenly ingested into the database, it may need to be deleted to prevent inaccuracies or sensitive data from being stored. Another consideration is compliance with end-user privacy rights to delete personal data processed about them in line with data protection laws like GDPR and CCPA.

Hard deletion on the record level is the gold standard

On-demand data deletion in a Dynatrace SaaS environment relies on two core characteristics: granularity and hard deletion. Granularity, specifically at the record level, allows for precise control over what data is deleted. This means that individual records can be targeted for deletion without affecting the rest of the dataset. Hard deletion refers to the permanent and secure erasure of data. These characteristics are crucial in ensuring data deletion is effective, secure, and compliant with data privacy regulations.

Many other SaaS vendors mistakenly assume that a soft delete—merely hiding or marking data as deleted while retaining it elsewhere—is sufficient. However, this approach leaves room for accidental exposure, unauthorized access, or data breaches, jeopardizing compliance and customer trust. Industry standards refer to hard deletion by overwriting data as the appropriate data sanitization method.^[1]

In addition, as with some other solutions, data deletion is often limited to a timeframe or requires removing all data in an index. This approach lacks flexibility and results in losing large volumes of important business data.

_{[1] NIST Special Publication 800-88 Revision 1, Guidelines for Media Sanitization}

Keep valuable data in Grail with targeted deletion

With record-level hard deletion, Dynatrace gives you control and transparency over your data. It enables you to effectively identify and promptly delete targeted data in a thorough and irreversible manner. So, while the relevant data is completely erased, all other valuable data remains intact. This ensures that your business operations continue smoothly, and you can focus on extracting value from Dynatrace.

In addition to the possibility of deleting data in buckets in Grail, record deletion in Grail now enables you to identify and select the records to be removed by leveraging DQL. You can use the Grail Storage Record Deletion API to trigger a deletion request.

Here are some tips to consider when deleting records in Grail:

• Leverage Notebooks: Use Notebooks to create a DQL query to filter all the records you intend to delete. With Notebooks, you can easily query data from Grail and visualize the results. This step also helps you determine the timeframe within which the relevant records are stored. To delete the records, use the Storage Record Deletion API. You might need to invoke the API multiple times if your data spans multiple days. Be aware that only one deletion process can be executed at a time.
• Verify your permissions: Before proceeding with any data deletion tasks, ensure that you have the necessary permissions, specifically storage:records:delete.
• Pause data ingestion: To avoid mistakenly deleting any recent data, stop ingesting new data that contains information you plan to delete. Only data older than 4 hours can be deleted.
• Keep track of deletion: Use the status command to gain insights into the status of the current deletion process. If necessary, use the cancel command to cancel a running process.

An audit trail is maintained to provide a clear and transparent record of what data was deleted, when, and by whom. By following these best practices, you can ensure efficient and safe data management, allowing you to focus on extracting value from Dynatrace while maintaining smooth and compliant business operations.

Get started

• New to Dynatrace? Open a free trial environment right now and see Dynatrace in action.
• Learn more about our commitment to providing you with control and transparency over your customers’ personal data in the Dynatrace Trust Center.
• Share your feedback, suggestions, and ideas with us in the Dynatrace Community.

What’s next

As we continue to innovate and enhance our privacy features, we remain committed to providing you with the industry’s most robust and transparent data protection solutions. We’re working on making data deletion even easier by onboarding log deletion in Grail to the Privacy Rights app. Check our Privacy Rights documentation to stay tuned to our continuous improvements.