Skip to technology filters Skip to main content
Dynatrace Hub

Extend the platform,
empower your team.

Popular searches:
Home hero bg
Google DataflowGoogle Dataflow
Google Dataflow

Google Dataflow

Get insights into Google Dataflow service metrics collected from the Google Operations API to ensure health of your cloud infrastructure.

Extension
Free trialDocumentation
  • Product information
  • Release notes

Overview

This Dynatrace extension leverages data collected from the Google Operations API to constantly monitor health and performance of Google Dataflow services. This extension combines all relevant data into pre-configured dashboards and provides alerting and event tracking.

This is intended for users, who want to:

  • enrich their monitoring data with metrics and logs from Google Operations API

This enables you to:

  • View and analyze 23 metrics that are specific to Google Dataflow, like Billable shuffle data processed, Current number of vCPUs in use, Current shuffle slots in use, and more.

  • Build custom dashboards for your cloud infrastructure.

  • Analyze Google Dataflow logs.

  • Set custom alerts that trigger remediation workflows.

Google Dataflow metric and log ingestion requires advanced GCP integration.

Compatibility requirements

  • Dynatrace GCP integration version 1.0.0+. To enable metric and log ingestion, see Dynatrace Documentation.
  • Dynatrace version 1.230+

Details

This extension package contains:

  • Configuration for Google Dataflow metric ingest based on your selected feature set (see full list of feature sets and metrics)

To provide correlation and causation analysis all ingested metrics and logs are analyzed by the Dynatrace Davis AI engine, which consumes DDUs.

Get started

To add this extension to your environment:

  • Follow the instructions detailed in Dynatrace Documentation. (Ignore the “Add to environment” button below.)
  • Ensure that you have GCP integration running in your environment and that Google Dataflow service is configured.

Following GCP integration and Google Dataflow configuration:

  • The first data points will be ingested by Dynatrace Davis within ~5 minutes.
  • Explore Google Dataflow metrics in Data Explorer and create custom charts.
  • Set custom events for alerting
Dynatrace
Documentation
By Dynatrace
Dynatrace support center
Subscribe to new releases
Copy to clipboard

Extension content

Content typeNumber of items included
metric metadata
44
generic type
1

Feature sets

Below is a complete list of the feature sets provided in this version. To ensure a good fit for your needs, individual feature sets can be activated and deactivated by your administrator during configuration.

Feature setsNumber of metrics included
Metric nameMetric keyDescriptionUnit
Per-stage backlog in bytescloud.gcp.dataflow_googleapis_com.job.backlog_bytes-Byte
Per-stage backlog in elementscloud.gcp.dataflow_googleapis_com.job.backlog_elements-Count
BigQueryIO.Write Requestscloud.gcp.dataflow_googleapis_com.job.bigquery.write_count-Count
Billable shuffle data processedcloud.gcp.dataflow_googleapis_com.job.billable_shuffle_data_processed-Byte
Bundle user processing latenciescloud.gcp.dataflow_googleapis_com.job.bundle_user_processing_latencies-MilliSecond
Current number of vCPUs in usecloud.gcp.dataflow_googleapis_com.job.current_num_vcpus-Count
Current shuffle slots in usecloud.gcp.dataflow_googleapis_com.job.current_shuffle_slots-Count
Data watermark lagcloud.gcp.dataflow_googleapis_com.job.data_watermark_age-Second
Disk Space Capacitycloud.gcp.dataflow_googleapis_com.job.disk_space_capacity-Byte
Duplicate message count per stagecloud.gcp.dataflow_googleapis_com.job.duplicates_filtered_out_count-Count
Elapsed timecloud.gcp.dataflow_googleapis_com.job.elapsed_time-Second
Element countcloud.gcp.dataflow_googleapis_com.job.element_count.gauge-Count
Elements Producedcloud.gcp.dataflow_googleapis_com.job.elements_produced_count-Count
Estimated byte countcloud.gcp.dataflow_googleapis_com.job.estimated_byte_count.gauge-Byte
Estimated Bytes Producedcloud.gcp.dataflow_googleapis_com.job.estimated_bytes_produced_count-Count
Failedcloud.gcp.dataflow_googleapis_com.job.is_failed-Count
Memory Capacitycloud.gcp.dataflow_googleapis_com.job.memory_capacity-Byte
Per-stage data watermark lagcloud.gcp.dataflow_googleapis_com.job.per_stage_data_watermark_age-Second
Per-stage system lagcloud.gcp.dataflow_googleapis_com.job.per_stage_system_lag-Second
The approximate number of parallel processing keyscloud.gcp.dataflow_googleapis_com.job.processing_parallelism_keys-Count
Pub/Sub Late Messages Countcloud.gcp.dataflow_googleapis_com.job.pubsub.late_messages_count-Count
Pub/Sub Published Messages Countcloud.gcp.dataflow_googleapis_com.job.pubsub.published_messages_count-Count
Pub/Sub Pulled Message Agescloud.gcp.dataflow_googleapis_com.job.pubsub.pulled_message_ages-MilliSecond
PubsubIO.Read requests from Dataflow jobscloud.gcp.dataflow_googleapis_com.job.pubsub.read_count-Count
Pub/Sub Pull Request Latenciescloud.gcp.dataflow_googleapis_com.job.pubsub.read_latencies-MilliSecond
Percentage of active/terminated Streaming Pull connectionscloud.gcp.dataflow_googleapis_com.job.pubsub.streaming_pull_connection_status-Percent
Pub/Sub Publish Requestscloud.gcp.dataflow_googleapis_com.job.pubsub.write_count-Count
Pub/Sub Publish Request Latenciescloud.gcp.dataflow_googleapis_com.job.pubsub.write_latencies-MilliSecond
Statuscloud.gcp.dataflow_googleapis_com.job.status-Unspecified
Storage bytes readcloud.gcp.dataflow_googleapis_com.job.streaming_engine.persistent_state.read_bytes_count-Count
Storage bytes writtencloud.gcp.dataflow_googleapis_com.job.streaming_engine.persistent_state.write_bytes_count-Count
Storage write latenciescloud.gcp.dataflow_googleapis_com.job.streaming_engine.persistent_state.write_latencies-MilliSecond
Per stage end to end latencies.cloud.gcp.dataflow_googleapis_com.job.streaming_engine.stage_end_to_end_latencies-MilliSecond
System lagcloud.gcp.dataflow_googleapis_com.job.system_lag-Second
Thread Timecloud.gcp.dataflow_googleapis_com.job.thread_time-MilliSecond
Timers pending count per stagecloud.gcp.dataflow_googleapis_com.job.timers_pending_count-Count
Timers processed count per stagecloud.gcp.dataflow_googleapis_com.job.timers_processed_count-Count
Total memory usage timecloud.gcp.dataflow_googleapis_com.job.total_memory_usage_time-GigaByte
Total PD usage timecloud.gcp.dataflow_googleapis_com.job.total_pd_usage_time-GigaByte
Total SECU usagecloud.gcp.dataflow_googleapis_com.job.total_secu_usage-Unspecified
Total shuffle data processedcloud.gcp.dataflow_googleapis_com.job.total_shuffle_data_processed-Byte
Total streaming data processedcloud.gcp.dataflow_googleapis_com.job.total_streaming_data_processed-Byte
Total vCPU timecloud.gcp.dataflow_googleapis_com.job.total_vcpu_time-Second
User Countercloud.gcp.dataflow_googleapis_com.job.user_counter-Count

Full version history

To have more information on how to install the downloaded package, please follow the instructions on this page.
ReleaseDate

Full version history

Version 1.0.5

Updated

  • Updated set of supported metrics

Full version history

Version 1.0.4

Updated

  • Feature sets & metrics section in HUB Product Information page

Full version history

Version 1.0.3

Added

  • Release notes to published extension
  • Auto publishing extensions to Hub

Full version history

No release notes

Full version history

No release notes

Dynatrace Hub
Hub HomeGet data into DynatraceBuild your own app
Log Management and AnalyticsKubernetesAI/ML ObservabilityInfrastructure ObservabilitySoftware DeliveryApplication ObservabilityApplication SecurityDigital ExperienceBusiness Analytics
Filter
Type
Built and maintained by
Deployment model
SaaS
  • SaaS
  • Managed
Partner FinderBecome a partnerDynatrace Developer

All

2 Results filtered by:

How fast is your app? logo

How fast is your app?

Measure your native mobile app performance and availability on real Android and iPhone devices, with no instrumentation.

OutSystems Cloud Metrics logo

OutSystems Cloud Metrics

OutSystems is a leading low-code application platform. With this integration, their Cloud API data becomes available inside Dynatrace.

Reach out to certified Dynatrace partners to solve your unique use-case

Spica Solutions logo

Spica Solutions

Certified individuals: 30Endorsements: Services Endorsed Partner
Authorized Sales Partner
Alanata logo

Alanata

Certified individuals: 30Endorsements: Services Endorsed Partner
Premier Sales Partner
avodaq AG logo

avodaq AG

Certified individuals: 31Endorsements: Services Endorsed Partner
Advanced Sales Partner
Moviri logo

Moviri

Certified individuals: 14
Advanced Sales Partner
AskMe Solutions & Consultants Co Ltd logo

AskMe Solutions & Consultants Co Ltd

Certified individuals: 30Endorsements: Services Endorsed Partner
Authorized Sales Partner
Eviden logo

Eviden

Certified individuals: 79Endorsements: Services Endorsed Partner
Premier Sales Partner
Omnilogy logo

Omnilogy

Certified individuals: 38Endorsements: Services Endorsed Partner
Premier Sales Partner
Matrix logo

Matrix

Certified individuals: 14
Premier Sales Partner
Phenisys logo

Phenisys

Certified individuals: 32Endorsements: Services Endorsed Partner
Premier Sales Partner
Spindox logo

Spindox

Certified individuals: 11
Authorized Sales Partner