Skip to technology filters Skip to main content
Dynatrace Hub

Extend the platform,
empower your team.

Popular searches:
Home hero bg
Apache SparkApache Spark
Apache Spark

Apache Spark

Enhanced insights for Spark Components

Extension
Free trial
Overview dashboardConfiguration feature setsMetrics Explorer - Spark tagProcess group Unified Analysis Screen updatedSpark Process Unified Analysis Screen
  • Product information
  • Release notes

Overview

This extension collects JMX metrics to provide insights into resource usage, job and application status, and performance of your spark components.

Apache Spark metrics are presented alongside other infrastructure measurements, enabling in-depth cluster performance analysis of both current and historical data.

Use cases

The extension enables insights into the overall health of Spark component instances

Get started

Dynatrace

Activate this extension in your Dynatrace environment from the in-product Hub and simply select which OneAgents to enable this on.

Spark

You must configure the required component metrics to be reported to the JMX sink.

  • The metrics system is configured via a configuration file that Spark expects to be present at $SPARK_HOME/conf/metrics.properties

For example, you can enable metric collection to the JmxSink for the master, worker, driver and executor components with a command such as:

cat << EOF > $SPARK_HOME/conf/metrics.properties
*.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink
master.source.jvm.class=org.apache.spark.metrics.source.JvmSource
worker.source.jvm.class=org.apache.spark.metrics.source.JvmSource
driver.source.jvm.class=org.apache.spark.metrics.source.JvmSource
executor.source.jvm.class=org.apache.spark.metrics.source.JvmSource
EOF

Refer to the Spark documentation for more details

Details

Details: This extension can query and collect almost all component instance and namespace metrics as defined in Spark Metric Providers

  • Driver
  • Executor
  • applicationMaster (YARN)
  • mesos_cluster
  • master
  • ApplicationSource (Spark standalone)
  • worker
  • shuffleservice

Compatibility information

  • Dynatrace OneAgent 1.275+
  • Linux or Windows OS
  • Spark version 3.x
  • Spark JMX metrics are enabled
Dynatrace
By Dynatrace
Dynatrace support center
Subscribe to new releases
Copy to clipboard

Extension content

Content typeNumber of items included
screen injections
1
generic relationship
1
metric metadata
80
dashboards
1
generic type
1
screen entities lists
1

Feature sets

Below is a complete list of the feature sets provided in this version. To ensure a good fit for your needs, individual feature sets can be activated and deactivated by your administrator during configuration.

Feature setsNumber of metrics included
Metric nameMetric keyDescriptionUnit
streaming.inputRate-totalspark.streaming.inputRate-total-Count
streaming.latencyspark.streaming.latency-MilliSecond
streaming.processingRate-totalspark.streaming.processingRate-total-Count
streaming.states-rowsTotalspark.streaming.states-rowsTotal-Count
streaming.states-usedBytesspark.streaming.states-usedBytes-Byte
Metric nameMetric keyDescriptionUnit
shuffleService.numActiveConnections.countspark.shuffleService.numActiveConnections.count-Count
shuffleService.numRegisteredConnections.countspark.shuffleService.numRegisteredConnections.count-Count
shuffleService.numCaughtExceptions.countspark.shuffleService.numCaughtExceptions.count-Count
shuffleService.registeredExecutorsSizespark.shuffleService.registeredExecutorsSize-Count
Metric nameMetric keyDescriptionUnit
appStatus.stages.failedStages.countspark.appStatus.stages.failedStages.count-Count
appStatus.stages.skippedStages.countspark.appStatus.stages.skippedStages.count-Count
appStatus.stages.completedStages.countspark.appStatus.stages.completedStages.count-Count
appStatus.tasks.excludedExecutors.countspark.appStatus.tasks.excludedExecutors.count-Count
appStatus.tasks.completedTasks.countspark.appStatus.tasks.completedTasks.count-Count
appStatus.tasks.failedTasks.countspark.appStatus.tasks.failedTasks.count-Count
appStatus.tasks.killedTasks.countspark.appStatus.tasks.killedTasks.count-Count
appStatus.tasks.skippedTasks.countspark.appStatus.tasks.skippedTasks.count-Count
spark.appStatus.tasks.unexcludedExecutors.countspark.appStatus.tasks.unexcludedExecutors.count-Count
appStatus.jobs.succeededJobs.countspark.appStatus.jobs.succeededJobs.count-Count
appStatus.jobs.failedJobs.countspark.appStatus.jobs.failedJobs.count-Count
appStatus.jobs.jobDurationspark.appStatus.jobs.jobDuration-MilliSecond
Metric nameMetric keyDescriptionUnit
HiveExternalCatalog.fileCacheHits.countspark.HiveExternalCatalog.fileCacheHits.count-Count
HiveExternalCatalog.filesDiscovered.countspark.HiveExternalCatalog.filesDiscovered.count-Count
HiveExternalCatalog.hiveClientCalls.countspark.HiveExternalCatalog.hiveClientCalls.count-Count
HiveExternalCatalog.parallelListingJobCount.countspark.HiveExternalCatalog.parallelListingJobCount.count-Count
HiveExternalCatalog.partitionsFetched.countspark.HiveExternalCatalog.partitionsFetched.count-Count
Metric nameMetric keyDescriptionUnit
ApplicationSource.statusspark.ApplicationSource.status-Count
ApplicationSource.runtime_msspark.ApplicationSource.runtime_ms-Count
ApplicationSource.coresspark.ApplicationSource.cores-Count
Metric nameMetric keyDescriptionUnit
BlockManager.disk.diskSpaceUsed_MBspark.BlockManager.disk.diskSpaceUsed_MB-MegaByte
BlockManager.memory.maxMem_MBspark.BlockManager.memory.maxMem_MB-MegaByte
BlockManager.memory.maxOffHeapMem_MBspark.BlockManager.memory.maxOffHeapMem_MB-MegaByte
BlockManager.memory.maxOnHeapMem_MBspark.BlockManager.memory.maxOnHeapMem_MB-MegaByte
BlockManager.memory.memUsed_MBspark.BlockManager.memory.memUsed_MB-MegaByte
BlockManager.memory.offHeapMemUsed_MBspark.BlockManager.memory.offHeapMemUsed_MB-MegaByte
BlockManager.memory.onHeapMemUsed_MBspark.BlockManager.memory.onHeapMemUsed_MB-MegaByte
BlockManager.memory.remainingMem_MBspark.BlockManager.memory.remainingMem_MB-MegaByte
BlockManager.memory.remainingOffHeapMem_MBspark.BlockManager.memory.remainingOffHeapMem_MB-MegaByte
BlockManager.memory.remainingOnHeapMem_MBspark.BlockManager.memory.remainingOnHeapMem_MB-MegaByte
Metric nameMetric keyDescriptionUnit
LiveListenerBus.numEventsPosted.countspark.LiveListenerBus.numEventsPosted.count-Count
LiveListenerBus.queue.appStatus.numDroppedEvents.countspark.LiveListenerBus.queue.appStatus.numDroppedEvents.count-Count
LiveListenerBus.queue.appStatus.sizespark.LiveListenerBus.queue.appStatus.size-Count
LiveListenerBus.queue.eventLog.numDroppedEvents.countspark.LiveListenerBus.queue.eventLog.numDroppedEvents.count-Count
LiveListenerBus.queue.eventLog.sizespark.LiveListenerBus.queue.eventLog.size-Count
Metric nameMetric keyDescriptionUnit
DAGScheduler.job.activeJobsspark.DAGScheduler.job.activeJobs-Count
DAGScheduler.job.allJobsspark.DAGScheduler.job.allJobs-Count
DAGScheduler.messageProcessingTime.countspark.DAGScheduler.messageProcessingTime.count-Count
DAGScheduler.messageProcessingTime.oneminuteratespark.DAGScheduler.messageProcessingTime.oneminuterate-PerMinute
DAGScheduler.messageProcessingTime.meanspark.DAGScheduler.messageProcessingTime.mean-MilliSecond
DAGScheduler.stage.failedStagesspark.DAGScheduler.stage.failedStages-Count
spark.DAGScheduler.stage.runningStagesspark.DAGScheduler.stage.runningStages-Count
DAGScheduler.stage.waitingStagesspark.DAGScheduler.stage.waitingStages-Count
Metric nameMetric keyDescriptionUnit
master.workersspark.master.workers-Count
master.aliveWorkersspark.master.aliveWorkers-Count
master.appsspark.master.apps-Count
master.waitingAppsspark.master.waitingApps-Count
Metric nameMetric keyDescriptionUnit
worker.executorsspark.worker.executors-Count
spark.worker.coresUsedspark.worker.coresUsed-Count
spark.worker.memUsed_MBspark.worker.memUsed_MB-MegaByte
spark.worker.coresFreespark.worker.coresFree-Count
spark.worker.memFree_MBspark.worker.memFree_MB-MegaByte
Metric nameMetric keyDescriptionUnit
mesos_cluster.waitingDriversspark.mesos_cluster.waitingDrivers-Count
mesos_cluster.launchedDriversspark.mesos_cluster.launchedDrivers-Count
mesos_cluster.retryDriversspark.mesos_cluster.retryDrivers-Count
Metric nameMetric keyDescriptionUnit
executor.bytesRead.countspark.executor.bytesRead.count-Byte
executor.bytesWritten.countspark.executor.bytesWritten.count-Byte
executor.cpuTime.countspark.executor.cpuTime.count-Count
executor.filesystem.file.largeRead_opsspark.executor.filesystem.file.largeRead_ops-Count
spark.executor.filesystem.file.read_bytesspark.executor.filesystem.file.read_bytes-Byte
spark.executor.filesystem.file.read_opsspark.executor.filesystem.file.read_ops-Count
spark.executor.filesystem.file.write_bytesspark.executor.filesystem.file.write_bytes-Byte
executor.filesystem.file.write_opsspark.executor.filesystem.file.write_ops-Count
executor.recordsRead.countspark.executor.recordsRead.count-Count
executor.recordsWritten.countspark.executor.recordsWritten.count-Count
executor.succeededTasks.countspark.executor.succeededTasks.count-Count
Metric nameMetric keyDescriptionUnit
applicationMaster.numContainersPendingAllocatespark.applicationMaster.numContainersPendingAllocate-Count
spark.applicationMaster.numExecutorsFailedspark.applicationMaster.numExecutorsFailed-Count
applicationMaster.numExecutorsRunningspark.applicationMaster.numExecutorsRunning-Count
applicationMaster.numLocalityAwareTasksspark.applicationMaster.numLocalityAwareTasks-Count
applicationMaster.numReleasedContainersspark.applicationMaster.numReleasedContainers-Count

Full version history

To have more information on how to install the downloaded package, please follow the instructions on this page.
ReleaseDate

Full version history

  • Add support for MBeans which have a UUID prefix

Full version history

  • JMX metrics collection for Spark component metric providers
  • Metadata - metrics, topology
Dynatrace Hub
Hub HomeGet data into DynatraceBuild your own app
Log Management and AnalyticsKubernetesAI/ML ObservabilityInfrastructure ObservabilitySoftware DeliveryApplication ObservabilityApplication SecurityDigital ExperienceBusiness Analytics
Filter
Type
Built and maintained by
Deployment model
SaaS
  • SaaS
  • Managed
Partner FinderBecome a partnerDynatrace Developer

0 Results

Found in 'All'

We couldn't find any results

You can search all listings, or try a different spelling or keyword. Still nothing? Dynatrace makes it easy to create custom apps.

Find a partnerLearn to build apps