Skip to technology filters Skip to main content
Dynatrace Hub

Extend the platform,
empower your team.

Popular searches:
Home hero bg
Apache KafkaApache Kafka
Apache Kafka

Apache Kafka

Automatic and intelligent observability with trace and metric insights.

Extension
Free trialDocumentation
Easily navigate your Kafka environment and its most important metrics with the out of the box dashboard, included with the activation of the extension.Dedicated screen for your Kafka broker with all its metrics, along with a navigation card for the rest of your Kafka environment.Dedicated screens also for your Kafka producers, consumers and topics.List all your Kafka topics to get a clear overview of your generated entities, with filtering and ordering for the most relevant metrics.Lists also available for the rest of the generated entities.
  • Product information
  • Release notes

Overview

With Dynatrace, you can get observability for Kafka without touching any code, thanks to automatic monitoring. Seamless end-to-end traces for connected producer and consumer clients allow you to diagnose anomalies and pinpoint the root cause of the broken code before your customers are affected. Comprehensive metrics give you insight into your Kafka servers with health and performance metrics for brokers, topics, producers, and consumers. Events point you to critical anomalies, reducing the mean repair time.

Use cases

  • Capture every message across tiers without blind spots.
  • Improve the performance of your producer and consumer services end-to-end.
  • Troubleshoot asynchronous service problems across your stack proactively.
  • Prevent message processing anomalies to reduce the mean time to repair.
  • Monitor the health and performance of all your brokers and topics.
  • Understand your consumer lag.

Get started

Messaging clients (applications)

To get trace insight:

  1. Install OneAgent on the virtual machine or server of your messaging clients (applications).
  2. Set up Dynatrace on Kubernetes or OpenShift for your messaging client (application) workloads.
  3. Activate the following OneAgent features:
    • Java Kafka
    • Java Kafka Streams
    • Spring for Apache Kafka
    • Node.js KafkaJs
    • .NET Messaging Apache Kafka

To get log insight:

  • Activate log monitoring.

Messaging servers (brokers)

Prerequisites:

  • Kafka broker, producer and consumer running on a supported Linux or Windows operating system.
  • OneAgent version 1.270+
  • Dynatrace version 1.270+

To get metric insight:

  1. Install OneAgent on the virtual machine or server of your Kafka broker process.
  2. Ensure your Kafka broker, producer and consumer processes are monitored.
  3. Activate the OneAgent feature Java Metric Extensions 2.0 (JMX).
  4. Select in Dynatrace Add to environment to configure the extension.
  5. Open the Apache Kafka Overview dashboard.

Extension

With the Kafka extension, you can get additional insight into your Kafka server with metrics for brokers, topics, producers, consumers, and more. The extension also provides alerts for the most critical metrics. It creates a custom topology and entities for brokers, topics, producers, and consumers. It provides a dashboard for easier access and configuration of the extension and its entities.

The extension will gather different metrics depending on if it's monitoring a Kafka broker, producer or consumer. Make sure to activate the extension on all of them to get all metrics.

Requirements

  • OneAgent version 1.270+
  • Dynatrace version 1.270+
  • OneAgent feature Java Metric Extensions 2.0 (JMX) activated

Select Add to environment to get started.

Dynatrace
Documentation
By Dynatrace
Dynatrace support center
Subscribe to new releases
Copy to clipboard

Extension content

Content typeNumber of items included
alerts
4
metric metadata
74
list screen layout
4
screen message cards
4
screen layout
4
screen injections
6
screen chart groups
8
screen actions
8
screen metric tables
4
screen entities lists
11
dashboards
1
generic relationship
8
generic type
4
screen properties
3

Feature sets

Below is a complete list of the feature sets provided in this version. To ensure a good fit for your needs, individual feature sets can be activated and deactivated by your administrator during configuration.

Feature setsNumber of metrics included
Metric nameMetric keyDescriptionUnit
Kafka Consumer - Requestskafka.consumer.consumer-metrics.request-rateThe average number of requests sent per second for a node.PerSecond
Kafka Consumer - Request sizekafka.consumer.consumer-metrics.request-size-avgThe average size of all requests in the window.Byte
Kafka Consumer - Incoming byte ratekafka.consumer.consumer-metrics.incoming-byte-rateBytes/second read off all sockets.BytePerSecond
Kafka Consumer - Outgoing byte ratekafka.consumer.consumer-metrics.outgoing-byte-rateThe average number of outgoing bytes sent per second to all servers.BytePerSecond
Kafka Consumer - Request latencykafka.consumer.consumer-metrics.request-latency-avgThe average request latency in ms for a node.MilliSecond
Kafka Consumer - Messages consumed ratekafka.consumer.consumer-metrics.records-consumed-rateThe average number of records consumed per second.PerSecond
Kafka Consumer - Bytes consumed ratekafka.consumer.consumer-metrics.bytes-consumed-rateThe average number of bytes consumed per second for a topic.PerSecond
Kafka Consumer - Fetch latencykafka.consumer.consumer-metrics.fetch-latency-avgThe average time taken for a fetch request.MilliSecond
Kafka Consumer - Consumer lagkafka.consumer.consumer-metrics.records-lagThe latest lag of the partition.Count
Kafka Consumer - Consumer lag averagekafka.consumer.consumer-metrics.records-lag-avgThe average lag of the partition.Count
Kafka Consumer - Consumer lag maximumkafka.consumer.consumer-metrics.records-lag-maxThe max lag of the partition.Count
Metric nameMetric keyDescriptionUnit
Kafka Server - Purgatory Produce Sizekafka.server.purgatory.produce-delay-sizeRequests waiting in the produce purgatory.Count
Kafka Server - Purgatory Fetch Sizekafka.server.purgatory.fetch-delay-sizeRequests waiting in the fetch purgatory.Count
Metric nameMetric keyDescriptionUnit
Kafka Producer - Incoming byte ratekafka.producer.producer-metrics.incoming-byte-rateThe average number of responses received per second for a node.BytePerSecond
Kafka Producer - Outgoing byte ratekafka.producer.producer-metrics.outgoing-byte-rateThe average number of outgoing bytes sent per second to all servers.BytePerSecond
Kafka Producer - I/O Wait timekafka.producer.producer-metrics.io-wait-time-ns-avgThe average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds.NanoSecond
Kafka Producer - Response ratekafka.producer.producer-metrics.response-rateResponses received sent per second.PerSecond
Kafka Producer - Request latencykafka.producer.producer-metrics.request-latency-avgThe average request latency in ms.MilliSecond
Kafka Producer - Compression ratekafka.producer.producer-metrics.compression-rate-avgThe average compression rate of record batches, defined as the average ratio of the compressed batch size over the uncompressed size.PerSecond
Kafka Producer - Request sizekafka.producer.producer-metrics.request-size-avgThe average size of all requests in the window.Byte
Kafka Producer - Requestskafka.producer.producer-metrics.request-rateThe average number of requests sent per second.PerSecond
Kafka Producer - Byte ratekafka.producer.producer-topic-metrics.byte-rateThe average number of bytes sent per second for a topic.BytePerSecond
Kafka Producer - Compression ratekafka.producer.producer-topic-metrics.compression-rateThe average compression rate of record batches for a topic, defined as the average ratio of the compressed batch size over the uncompressed size.BytePerSecond
Kafka Producer - Failed Requests Ratekafka.producer.producer-topic-metrics.record-error-rateThe average per-second number of record sends that resulted in errors for a topic.PerSecond
Kafka Producer - Requests Sent Ratekafka.producer.producer-topic-metrics.record-send-rateThe average number of records sent per second for a topic.PerSecond
Metric nameMetric keyDescriptionUnit
Kafka Connector - Statuskafka.connector.statusEquals 1 if the status is running, 0 otherwise.Count
Kafka Connector - Task statuskafka.connector.task.statusEquals 1 if the status is running, 0 otherwise.Count
Kafka Connector - Task pause ratiokafka.connector.task.pause-ratioThe fraction of time this task has spent in the pause state.Count
Kafka Connector - Task running ratiokafka.connector.task.running-ratioThe fraction of time this task has spent in the running state.Count
Kafka Connector - Task success ratiokafka.connector.task.offset-commit-success-percentageThe average percentage of this task's offset commit attempts that succeeded.Percent
Kafka Connector - Task commit time (max)kafka.connector.task.offset-commit-max-time-msThe maximum time in milliseconds taken by this task to commit offsets.MilliSecond
Kafka Connector - Task failure ratiokafka.connector.task.offset-commit-failure-percentageThe maximum time in milliseconds taken by this task to commit offsets.Percent
Kafka Connector - Task commit time (avg)kafka.connector.task.offset-commit-avg-time-msThe average time in milliseconds taken by this task to commit offsets.MilliSecond
Kafka Connector - Task batch size (max)kafka.connector.task.batch-size-maxThe maximum size of the batches processed by the connector.Byte
Kafka Connector - Task batch size (avg)kafka.connector.task.batch-size-avgThe average size of the batches processed by the connector.Byte
Metric nameMetric keyDescriptionUnit
Kafka Controller - Offline partitionskafka.controller.KafkaController.OfflinePartitionsCountThe number of partitions that don't have an active leader and are therefore not writable or readable.Count
Kafka Controller - Active cluster controllerskafka.controller.KafkaController.ActiveControllerCount.ValueIndicates whether the broker is the controller broker.Count
Metric nameMetric keyDescriptionUnit
Kafka Server - Handler Pool Idle Percent Ratekafka.server.handler.average-idle-percent.rateThe average fraction of time the request handler threads are idle. Values are between 0 meaning all resources are used and 1 meaning all resources are available.PerSecond
Metric nameMetric keyDescriptionUnit
Kafka Server - Disk Read Ratekafka.server.disk.read-bytesThe total number of bytes read by the broker process, including reads from all disks. The total doesn't include reads from page cache. Available only on Linux-based systems.PerSecond
Kafka Server - Disk Write Ratekafka.server.disk.write-bytesThe total number of bytes written by the broker process, including writes from all disks. Available only on Linux-based systems.PerSecond
Metric nameMetric keyDescriptionUnit
Kafka Broker - Incoming byte ratekafka.server.BrokerTopicMetrics.BytesInPerSec.OneMinuteRateThe rate at which data sent from producers is consumed by the broker.BytePerSecond
Kafka Broker - Outgoing byte ratekafka.server.BrokerTopicMetrics.BytesOutPerSec.OneMinuteRateThe rate at which data sent from other brokers is consumed by the follower broker.BytePerSecond
Kafka Broker - Messages in ratekafka.server.BrokerTopicMetrics.MessagesInPerSec.OneMinuteRateThe rate at which individual messages are consumed by the broker.PerSecond
Kafka Broker - Follower fetch requests ratekafka.server.BrokerTopicMetrics.TotalFollowerFetchRequestsPerSec.OneMinuteRateThe follower fetch request rate for the broker.PerSecond
Kafka Broker - Produce message conversions ratekafka.server.BrokerTopicMetrics.ProduceMessageConversionsPerSec.OneMinuteRateThe rate at which produce messages are converted, by topic.PerSecond
Kafka Broker - Partitionskafka.server.ReplicaManager.PartitionCountThe number of partitions in the broker.Count
Kafka Broker - Under replicated partitionskafka.server.ReplicaManager.UnderReplicatedPartitionsThe number of partitions that have not been fully replicated in the follower replicas.Count
Kafka Broker - Produce request ratekafka.server.BrokerTopicMetrics.TotalProduceRequestsPerSec.OneMinuteRateThe produce request rate per second.PerSecond
Kafka Broker - Fetch request ratekafka.server.BrokerTopicMetrics.TotalFetchRequestsPerSec.OneMinuteRateThe fetch request rate per second.PerSecond
Kafka Broker - Failed produce requestskafka.server.BrokerTopicMetrics.FailedProduceRequestsPerSec.OneMinuteRateThe produce request rate for requests that failed.PerSecond
Kafka Broker - Failed fetch requestskafka.server.BrokerTopicMetrics.FailedFetchRequestsPerSec.OneMinuteRateThe fetch request rate for requests that failed.PerSecond
Kafka Server - Max follower lagkafka.server.ReplicaFetcherManager.MaxLag.Replica.ValueThe maximum lag between the time that messages are received by the leader replica and by the follower replicas.Count
Kafka Server - Current follower lagkafka.server.FetcherLagMetrics.ConsumerLag.ValueThe lag in number of messages per follower replica.Count
Kafka Server - Fetch Conversions Ratekafka.server.FetchConversionsRate.OneMinuteRate-Count
Kafka Server - Produce Conversions Ratekafka.server.ProduceConversionsRate.OneMinuteRate-Count
Metric nameMetric keyDescriptionUnit
Kafka Controller - Leader election ratekafka.controller.ControllerStats.LeaderElectionRateAndTimeMs.OneMinuteRateThe broker leader election rate and latency in milliseconds. This is non-zero when there are broker failures.MilliSecond
Kafka Controller - Unclean election ratekafka.controller.ControllerStats.UncleanLeaderElectionsPerSec.OneMinuteRateThe unclean broker leader election rate. Should be 0.PerSecond
Kafka Server - Leader countkafka.server.ReplicaManager.LeaderCount.ValueThe number of replicas for which this broker is the leader.Count
Metric nameMetric keyDescriptionUnit
Kafka Network - Produce requests per secondkafka.network.RequestMetrics.RequestsPerSec.Produce.OneMinuteRateThe total number of requests made for produce per secondPerSecond
Kafka Network - FetchConsumer requests per secondkafka.network.RequestMetrics.RequestsPerSec.FetchConsumer.OneMinuteRateThe total number of requests made for fetch consumer per secondPerSecond
Kafka Network - FetchFollower requests per secondkafka.network.RequestMetrics.RequestsPerSec.FetchFollower.OneMinuteRateThe total number of requests made for fetch follower per secondPerSecond
Kafka Network - Total time per Produce requestkafka.network.RequestMetrics.TotalTimeMs.Produce.CountTotal time, in milliseconds, spent processing requests, for produce.MilliSecond
Kafka Network - Total time per FetchConsumer requestkafka.network.RequestMetrics.TotalTimeMs.FetchConsumer.CountTotal time, in milliseconds, spent processing requests, for fetch consumer.MilliSecond
Kafka Network - Total time per FetchFollower requestkafka.network.RequestMetrics.TotalTimeMs.FetchFollower.CountTotal time, in milliseconds, spent processing requests, for pfetch follower.MilliSecond
Kafka Network - Request queue sizekafka.network.RequestChannel.RequestQueueSize.ValueSize of the request queue.Count
Metric nameMetric keyDescriptionUnit
Kafka Server - ZooKeeper disconnectskafka.server.SessionExpireListener.ZooKeeperDisconnectsPerSec.OneMinuteRateA meter that provides the number of recent ZooKeeper client disconnects.PerSecond
Kafka Server - ZooKeeper expireskafka.server.SessionExpireListener.ZooKeeperExpiresPerSec.OneMinuteRateThe number of ZooKeeper sessions that have expired.PerSecond
Kafka Server - Zookeeper Active Connectionskafka.server.active-connectionsThe number of currently open connections to the broker.Count
Metric nameMetric keyDescriptionUnit
Kafka Connect - Requestskafka.connect.connect-metrics.request-rateThe average number of requests sent per second.PerSecond
Kafka Connect - Outgoing byte ratekafka.connect.connect-metrics.outgoing-byte-rateThe average number of outgoing bytes sent per second to all servers.BytePerSecond
Kafka Connect - Request sizekafka.connect.connect-metrics.request-size-avgThe average size of all requests in the window.Byte
Kafka Connect - Incoming byte ratekafka.connect.connect-metrics.incoming-byte-rateBytes/second read off all sockets.BytePerSecond
Metric nameMetric keyDescriptionUnit
Kafka Log - Log flush 95th percentilekafka.log.LogFlushStats.LogFlushRateAndTimeMs.Percentile95thLog flush rate and time in milliseconds.MilliSecond
Kafka Log - Log flush mean timekafka.log.LogFlushStats.LogFlushRateAndTimeMs.MeanLog flush rate and time in milliseconds.MilliSecond

Full version history

To have more information on how to install the downloaded package, please follow the instructions on this page.
ReleaseDate

Full version history

🚀 Improved in this version:

  • [🔴 Breaking change] Refactored entity creation for consumers and producers. The dimension client-id is no longer used to generate these entities but only the process_group_instanceand the host will be considered. This will generate new entity IDs for all your existing consumers and producers, which will break any configuration (metric events, dashboards, etc.) that use those individual IDs. The name of these entities will also change, as it was based on the client-idpreviously.
  • Charts in consumers and producers have been changed to show metrics per client-id, instead of per entity.
  • Added descriptions to all extension metrics

✨ New in this version:

  • Added 8 new metrics related to the Kafka Connector.

🐛 Fixed in this version:

  • As client-id is no longer used for generating consumer and producer entities, we no longer run into the entity dimension limitation that prevented new data from being ingested.

Full version history

  • Broker now uses the hostname instead of the host ID, when available.
  • Entity lists for broker, producer and consumer now show the host and the process they are related to, as well as allowing you to filter by those entities.
  • Removed any reference to zookeeper metrics or entities. To get insight into Zookeeper, activate the recently released standalone extension.

Full version history

  • Changed the name of the out-of-the-box alerts to include the word Kafka in the title for increased accessibility.
  • Added the following metrics under the zookeeper-metrics feature set:
    • kafka.zookeeper.server.maxRequestLatency
    • kafka.zookeeper.server.minRequestLatency
    • kafka.zookeeper.server.avgRequestLatency
    • kafka.zookeeper.server.aliveConnections
    • kafka.zookeeper.server.packetsReceived.count
    • kafka.zookeeper.server.packetsSent.count
    • kafka.zookeeper.server.outstandingRequests
  • Added custom topology for the zookeeper
  • Added UA screens for the zookeeper
  • Modified the out-of-the-box dashboard to include data about the zookeeper

Full version history

  • Added a new metric kafka.consumer.consumer-metrics.records-lag to show latest consumer lag instead of the already available consumer lag average and maximum.
  • Removed the status dimension from the kafka.connector.status and kafka.connector.task.status as the framework is not ready to report such dimensions. The metrics now report 1 when the status is running or 0 otherwise.

Full version history

  • Fixed a bug where consumer-fetch-manager-metrics where not being captured correctly.
  • Added connector task status to metrics.
  • Replace entity lists for Host and Process Group Instance with properties in broker, consumer and producer screens.

Full version history

  • Fixed a bug where metrics for kafka.consumer:type=consumer-fetch-manager-metrics where not being populated:
    • kafka.consumer.consumer-metrics.records-lag-avg
    • kafka.consumer.consumer-metrics.records-lag-max
    • kafka.consumer.consumer-metrics.records-consumed-rate
    • kafka.consumer.consumer-metrics.bytes-consumed-rate
    • kafka.consumer.consumer-metrics.fetch-latency-avg

Full version history

  • Add topic dimension to existing metrics:
    • kafka.server.BrokerTopicMetrics.BytesInPerSec.OneMinuteRate
    • kafka.server.BrokerTopicMetrics.BytesOutPerSec.OneMinuteRate
    • kafka.server.BrokerTopicMetrics.MessagesInPerSec.OneMinuteRate
    • kafka.server.BrokerTopicMetrics.TotalProduceRequestsPerSec.OneMinuteRate
  • Add new metrics:
    • kafka.server.BrokerTopicMetrics.TotalFollowerFetchRequestsPerSec.OneMinuteRate
    • kafka.server.BrokerTopicMetrics.ProduceMessageConversionsPerSec.OneMinuteRate
  • Modified UA screens to include the metrics and dimensions listed above:
    • Kafka Broker screen
    • Kafka Topic screen
    • Host injected tile
    • Process Group Instance injected tile

Full version history

  • Kafka Server - Current follower lag: Fixed typo in metric key, changed from kafka.server.FetcherLagMetrics.CosumerLag.Value to kafka.server.FetcherLagMetrics.ConsumerLag.Value
    • This is a breaking change if you're using this metric in dashboards, metric events or other configurations. You will need to reference the new key.
  • Added connector status metric
  • Added primary entity for most metrics
  • Added navigation card to all screens

Full version history

No release notes

Dynatrace Hub
Hub HomeGet data into DynatraceBuild your own app
Log Management and AnalyticsKubernetesAI/ML ObservabilityInfrastructure ObservabilitySoftware DeliveryApplication ObservabilityApplication SecurityDigital ExperienceBusiness Analytics
Filter
Type
Built and maintained by
Deployment model
SaaS
  • SaaS
  • Managed
Partner FinderBecome a partnerDynatrace Developer

All

2 Results filtered by:

How fast is your app? logo

How fast is your app?

Measure your native mobile app performance and availability on real Android and iPhone devices, with no instrumentation.

OutSystems Cloud Metrics logo

OutSystems Cloud Metrics

OutSystems is a leading low-code application platform. With this integration, their Cloud API data becomes available inside Dynatrace.

Reach out to certified Dynatrace partners to solve your unique use-case

Spica Solutions logo

Spica Solutions

Certified individuals: 30Endorsements: Services Endorsed Partner
Authorized Sales Partner
Alanata logo

Alanata

Certified individuals: 30Endorsements: Services Endorsed Partner
Premier Sales Partner
avodaq AG logo

avodaq AG

Certified individuals: 31Endorsements: Services Endorsed Partner
Advanced Sales Partner
Moviri logo

Moviri

Certified individuals: 14
Advanced Sales Partner
AskMe Solutions & Consultants Co Ltd logo

AskMe Solutions & Consultants Co Ltd

Certified individuals: 30Endorsements: Services Endorsed Partner
Authorized Sales Partner
Eviden logo

Eviden

Certified individuals: 79Endorsements: Services Endorsed Partner
Premier Sales Partner
Omnilogy logo

Omnilogy

Certified individuals: 38Endorsements: Services Endorsed Partner
Premier Sales Partner
Matrix logo

Matrix

Certified individuals: 14
Premier Sales Partner
Phenisys logo

Phenisys

Certified individuals: 32Endorsements: Services Endorsed Partner
Premier Sales Partner
Spindox logo

Spindox

Certified individuals: 11
Authorized Sales Partner