Kafka process monitoring out of the box

Dynatrace automatically recognizes Kafka processes and instantly gathers Kafka metrics on the process and cluster levels.

Apache Kafka is an open-source, distributed publish-subscribe message bus designed to be fast, scalable, and durable. Dynatrace is proud to announce out-of-the-box support for Kafka process monitoring.

With the release of Dynatrace SaaS/Managed version 1.155, Dynatrace now automatically recognizes Kafka processes and instantly gathers Kafka metrics on the process and cluster levels.

Cluster components

Dynatrace groups all Kafka processes into a single process group, so you can see the health of all Kafka components on a single page.

If you have more than one Kafka cluster, use an environment variable for proper cluster detection.

Kafka cluster metrics

Partition and controller charts present your cluster details so you can make sure, for example, that you have one controller per cluster and no offline or under-replicated partitions. You don’t necessarily have to keep an eye on all those metrics, however. Dynatrace informs you of any detected Kafka-related performance problems. You can customize anomaly detection for Kafka using the Plugin events settings.

Kafka broker metrics

Kafka latency and throughput depend on the rate of log flush to disk. Even a slight change can drastically impact Kafka performance. Under-replicated partitions indicate that replication is ongoing, consumers aren’t getting data, and latency is growing. Use the Kafka broker tab to keep an eye out for such problems. For other interesting Kafka metrics, such as details of broker metrics and Kafka request details, check the Further details tab.

Kafka consumer, producer, and connect components

Kafka monitoring is fully integrated with Dynatrace, enabling OneAgent to monitor all Kafka components. For the full list of Kafka metrics, prerequisites, and more see How Dynatrace monitors Kafka applications.

Your feedback is most welcome. Just add a comment below or post your feedback to Dynatrace Open Q&A.

And stay tuned for more Kafka developments…