HadoopHadoop monitoring in Dynatrace provides a high-level overview of the main Hadoop components within your cluster. These enhanced insights provides additional metrics directly from the HDFS and YARN services.
The extension enables insights into the overall health of Hadoop HDFS and YARN services
Activate this extension in your Dynatrace environment from the in-product Hub and simply select which OneAgents to enable this on.
This extension provides extra Hadoop metrics through the use of JMX queries.
Below is a complete list of the feature sets provided in this version. To ensure a good fit for your needs, individual feature sets can be activated and deactivated by your administrator during configuration.
| Metric name | Metric key | Description | Unit |
|---|---|---|---|
| FilesAppended | hadoop.hdfs.namenode.FilesAppended.count | Total number of files appended | Count |
| FilesCreated | hadoop.hdfs.namenode.FilesCreated.count | Total number of files and directories created by create or mkdir operations | Count |
| FilesDeleted | hadoop.hdfs.namenode.FilesDeleted.count | Total number of files and directories deleted by delete or rename operations | Count |
| FilesRenamed | hadoop.hdfs.namenode.FilesRenamed.count | Total number of rename operations (NOT number of files/dirs renamed) | Count |
| Metric name | Metric key | Description | Unit |
|---|---|---|---|
| NodeManager AllocatedContainers | hadoop.yarn.nodemanager.AllocatedContainers | NodeManager Current number of allocated containers | Count |
| AllocatedGB | hadoop.yarn.nodemanager.AllocatedGB | Current allocated memory in GB | GigaByte |
| AvailableGB | hadoop.yarn.nodemanager.AvailableGB | Current available memory in GB | GigaByte |
| ContainersCompleted | hadoop.yarn.nodemanager.ContainersCompleted.count | Total number of successfully completed containers | Count |
| ContainersFailed | hadoop.yarn.nodemanager.ContainersFailed.count | Total number of failed containers | Count |
| ContainersIniting | hadoop.yarn.nodemanager.ContainersIniting.count | Current number of initializing containers | Count |
| ContainersKilled | hadoop.yarn.nodemanager.ContainersKilled.count | Total number of killed containers | Count |
| ContainersLaunched | hadoop.yarn.nodemanager.ContainersLaunched.count | Total number of launched containers | Count |
| ContainersRunning | hadoop.yarn.nodemanager.ContainersRunning.count | Current number of running containers | Count |
| Metric name | Metric key | Description | Unit |
|---|---|---|---|
| DataNode CacheCapacity | hadoop.hdfs.datanode.CacheCapacity | The cache capacity of the DataNode | Byte |
| DataNode CacheUsed | hadoop.hdfs.datanode.CacheUsed | The cache used the DataNode | Byte |
| DataNode Capacity | hadoop.hdfs.datanode.Capacity | Current raw capacity of the DataNodes in bytes | Byte |
| DataNode DfsUsed | hadoop.hdfs.datanode.DfsUsed | The storage space that has been used up by HDFS. | Byte |
| DataNode NumBlocksCached | hadoop.hdfs.datanode.NumBlocksCached | The number of blocks cached on the DataNode | Count |
| DataNode NumBlocksFailedToCache | hadoop.hdfs.datanode.NumBlocksFailedToCache | The number of blocks that failed to cache on the DataNode | Count |
| DataNode NumBlocksFailedToUncache | hadoop.hdfs.datanode.NumBlocksFailedToUncache | The number of failed blocks to remove from cache. | Count |
| DataNode NumFailedVolumes | hadoop.hdfs.datanode.NumFailedVolumes | Number of failed volumes. | Count |
| DataNode Remaining | hadoop.hdfs.datanode.Remaining | The remaining DataNode disk space left in Percent | Percent |
| Metric name | Metric key | Description | Unit |
|---|---|---|---|
| DataNode BlocksRead | hadoop.hdfs.datanode.BlocksRead.count | Total number of blocks read from DataNode | Count |
| DataNode BlocksRemoved | hadoop.hdfs.datanode.BlocksRemoved.count | Total number of blocks removed from DataNode | Count |
| DataNode BlocksReplicated | hadoop.hdfs.datanode.BlocksReplicated.count | Total number of blocks replicated | Count |
| DataNode BlocksVerified | hadoop.hdfs.datanode.BlocksVerified.count | Total number of blocks verified | Count |
| DataNode BlocksWritten | hadoop.hdfs.datanode.BlocksWritten.count | Total number of blocks written to DataNode | Count |
| DataNode BytesRead | hadoop.hdfs.datanode.BytesRead.count | Total number of bytes read from DataNode | Byte |
| DataNode BytesWritten | hadoop.hdfs.datanode.BytesWritten.count | Total number of bytes written to DataNode | Byte |
| Metric name | Metric key | Description | Unit |
|---|---|---|---|
| ShuffleConnections | hadoop.yarn.nodemanager.ShuffleConnections.count | Number of Shuffle connections | Count |
| ShuffleOutputBytes | hadoop.yarn.nodemanager.ShuffleOutputBytes.count | Total shuffle output bytes | Byte |
| ShuffleOutputsFailed | hadoop.yarn.nodemanager.ShuffleOutputsFailed.count | Number of failed shuffle outputs | Count |
| ShuffleOutputsOK | hadoop.yarn.nodemanager.ShuffleOutputsOK.count | Number of OK shuffle outputs | Count |
| Metric name | Metric key | Description | Unit |
|---|---|---|---|
| CapacityTotal | hadoop.hdfs.namenode.CapacityTotal | Current raw capacity of DataNodes in bytes | Byte |
| CapacityUsed | hadoop.hdfs.namenode.CapacityUsed | Current used capacity across all DataNodes in bytes | Byte |
| CapacityRemaining | hadoop.hdfs.namenode.CapacityRemaining | Current remaining capacity in bytes | Byte |
| TotalLoad | hadoop.hdfs.namenode.TotalLoad | Current number of connections | Count |
| BlocksTotal | hadoop.hdfs.namenode.BlocksTotal | Current number of allocated blocks in the system | Count |
| PendingDeletionBlocks | hadoop.hdfs.namenode.PendingDeletionBlocks | Current number of blocks pending deletion | Count |
| CorruptBlocks | hadoop.hdfs.namenode.CorruptBlocks | Current number of blocks with corrupt replicas. | Count |
| CapacityUsedNonDFS | hadoop.hdfs.namenode.CapacityUsedNonDFS | Current space used by DataNodes for non DFS purposes in bytes | Count |
| Metric name | Metric key | Description | Unit |
|---|---|---|---|
| JobsCompleted | hadoop.yarn.mrappmaster.JobsCompleted | Number of completed jobs | Count |
| JobsFailed | hadoop.yarn.mrappmaster.JobsFailed | Number of failed jobs | Count |
| JobsKilled | hadoop.yarn.mrappmaster.JobsKilled | Number of killed jobs | Count |
| JobsPreparing | hadoop.yarn.mrappmaster.JobsPreparing | Number of preparing jobs | Count |
| JobsRunning | hadoop.yarn.mrappmaster.JobsRunning | Number of running jobs | Count |
| MapsCompleted | hadoop.yarn.mrappmaster.MapsCompleted | Number of maps completed | Count |
| MapsFailed | hadoop.yarn.mrappmaster.MapsFailed | Number of maps failed | Count |
| MapsKilled | hadoop.yarn.mrappmaster.MapsKilled | Number of maps killed | Count |
| MapsRunning | hadoop.yarn.mrappmaster.MapsRunning | Number of maps running | Count |
| MapsWaiting | hadoop.yarn.mrappmaster.MapsWaiting | Number of maps waiting | Count |
| ReducesCompleted | hadoop.yarn.mrappmaster.ReducesCompleted | Number of completed reduces | Count |
| ReducesFailed | hadoop.yarn.mrappmaster.ReducesFailed | Number of failed reduces | Count |
| ReducesKilled | hadoop.yarn.mrappmaster.ReducesKilled | Number of killed reduces | Count |
| ReducesRunning | hadoop.yarn.mrappmaster.ReducesRunning | Number of running reduces | Count |
| ReducesWaiting | hadoop.yarn.mrappmaster.ReducesWaiting | Number of waiting reduces | Count |
| Metric name | Metric key | Description | Unit |
|---|---|---|---|
| FilesTotal | hadoop.hdfs.namenode.FilesTotal | Current number of files and directories | Count |
| PendingReplicationBlocks | hadoop.hdfs.namenode.PendingReplicationBlocks | Current number of blocks pending to be replicated | Count |
| UnderReplicatedBlocks | hadoop.hdfs.namenode.UnderReplicatedBlocks | Current number of blocks under replicated | Count |
| ScheduledReplicationBlocks | hadoop.hdfs.namenode.ScheduledReplicationBlocks | Current number of blocks scheduled for replications | Count |
| NumLiveDataNodes | hadoop.hdfs.namenode.NumLiveDataNodes | Number of datanodes which are currently live | Count |
| NumDeadDataNodes | hadoop.hdfs.namenode.NumDeadDataNodes | Number of datanodes which are currently dead | Count |
| NumDecomLiveDataNodes | hadoop.hdfs.namenode.NumDecomLiveDataNodes | Number of datanodes which have been decommissioned and are now live | Count |
| NumDecomDeadDataNodes | hadoop.hdfs.namenode.NumDecomDeadDataNodes | Number of datanodes which have been decommissioned and are now dead | Count |
| VolumeFailuresTotal | hadoop.hdfs.namenode.VolumeFailuresTotal | Total number of volume failures across all Datanodes | Count |
| EstimatedCapacityLostTotal | hadoop.hdfs.namenode.EstimatedCapacityLostTotal | An estimate of the total capacity lost due to volume failures | Count |
| NumDecommissioningDataNodes | hadoop.hdfs.namenode.NumDecommissioningDataNodes | Number of datanodes in decommissioning state | Count |
| NumStaleDataNodes | hadoop.hdfs.namenode.NumStaleDataNodes | Number of datanodes marked stale due to delayed hearbeat. | Count |
| Metric name | Metric key | Description | Unit |
|---|---|---|---|
| NumActiveNMs | hadoop.yarn.resourcemanager.NumActiveNMs | Current number of active NodeManagers | Count |
| NumDecommissioningNMs | hadoop.yarn.resourcemanager.NumDecommissionedNMs | Current number of NodeManagers being decommissioned | Count |
| NumLostNMs | hadoop.yarn.resourcemanager.NumLostNMs | Current number of lost NodeManagers for not sending heartbeats | Count |
| NumRebootedNMs | hadoop.yarn.resourcemanager.NumRebootedNMs | Current number of rebooted NodeManagers | Count |
| NumUnhealthyNMs | hadoop.yarn.resourcemanager.NumUnhealthyNMs | Current number of unhealthy NodeManagers | Count |
| Metric name | Metric key | Description | Unit |
|---|---|---|---|
| NumberOfMissingBlocks | hadoop.hdfs.namenode.NumberOfMissingBlocks | - | Count |
| CacheCapacity | hadoop.hdfs.namenode.CacheCapacity | The total cache capacity of all DataNodes | Byte |
| CacheUsed | hadoop.hdfs.namenode.CacheUsed | The total cache used by all DataNodes | Byte |
| Metric name | Metric key | Description | Unit |
|---|---|---|---|
| AllocatedContainers | hadoop.yarn.resourcemanager.AllocatedContainers | Queue Current number of allocated containers | Count |
| AllocatedMB | hadoop.yarn.resourcemanager.AllocatedMB | Current allocated memory in MB | MegaByte |
| AllocatedVCores | hadoop.yarn.resourcemanager.AllocatedVCores | Current allocated CPU in virtual cores | Count |
| AppsCompleted | hadoop.yarn.resourcemanager.AppsCompleted.count | Total number of completed applications | Count |
| AppsFailed | hadoop.yarn.resourcemanager.AppsFailed.count | Total number of failed applications | Count |
| AppsKilled | hadoop.yarn.resourcemanager.AppsKilled.count | Total number of killed applications | Count |
| AppsPending | hadoop.yarn.resourcemanager.AppsPending.count | Current number of applications that have not yet been assigned by any containers | Count |
| AppsRunning | hadoop.yarn.resourcemanager.AppsRunning.count | Current number of running applications | Count |
| AppsSubmitted | hadoop.yarn.resourcemanager.AppsSubmitted.count | Total number of submitted applications | Count |
| AvailableMB | hadoop.yarn.resourcemanager.AvailableMB | Current available memory in MB | Count |
| AvailableVCores | hadoop.yarn.resourcemanager.AvailableVCores | Current available CPU in virtual cores | Count |
| PendingMB | hadoop.yarn.resourcemanager.PendingMB | Current memory requests in MB that are pending to be fulfilled by the scheduler | MegaByte |
| PendingVCores | hadoop.yarn.resourcemanager.PendingVCores | Current CPU requests in virtual cores that are pending to be fulfilled by the scheduler | Count |
| ReservedMB | hadoop.yarn.resourcemanager.ReservedMB | Current reserved memory in MB | MegaByte |
| ReservedVCores | hadoop.yarn.resourcemanager.ReservedVCores | Current reserved CPU in virtual cores | Count |
No release notes
