Get automatic code-level insights into your Go applications and cloud platform components

Go is the fourth most popular programming language, used by nearly two million developers (based on 2018 GitHub usage). Cloud platforms like Cloud Foundry, Kubernetes, and OpenShift also heavily rely on Go for implementing platform components (like Cloud Foundry’s Gorouter, or Kubernetes’s Kubelet).

Within Dynatrace alone, we saw the Go usage growing constantly throughout the last year and a half. Today, the code module for Go monitors more than 18,000 processes in approximately 3,000 process groups running on more than 6,300 hosts. More than 10,000 of these processes are related to Cloud Foundry, OpenShift, and Kubernetes.

Traditional monitoring requires manual instrumentation to provide Go code-level insights

One of the great things about cloud platforms like Cloud Foundry, Kubernetes, or OpenShift, which are typically used to run web-scale and highly dynamic microservices architectures, is their ability to automatically scale applications as well as platform components. For instance, you can add more application instances to handle increased traffic and demand.

However, with application instances spun up and down automatically you also need to ensure that monitoring is enabled automatically as manual approaches can’t keep up with the speed and scale.

Additionally, manual instrumentation makes it impossible to monitor 3rd party Go-based applications without changing code.

Get automatic code-level insights into your own and 3rd party Go applications without code changes

Dynatrace aims to “boldly Go where no one has monitored before“. No code changes are required to your Go applications when using Dynatrace. The unique OneAgent code module for Go monitoring technology enables you to capture data that is well beyond the capabilities of traditional monitoring solutions. Dynatrace can extract internal Go runtime information that is inaccessible with public Go runtime APIs.

This means that with Dynatrace Go monitoring support, you get code-level insight not only into your own Go applications but also into 3rd party Go-based applications that your applications may rely on. Examples of such cloud platform components (or 3rd party Go-based applications) are Cloud Foundry’s Gorouter components, which route incoming traffic to appropriate components.

This improved visibility enables you to:

• Monitor web-scale and highly dynamic microservices architectures including Go-based applications and platform components.
• Optimize Go applications based on real-time data from production environments.
• Accelerate problem resolution and troubleshooting by up to 90% with our automatic root-cause analysis.
• Improve cloud platform management with deeper insights into platform metrics.

What you get with Dynatrace code-level monitoring for Go applications

The OneAgent code module for Go blends well into the feature set of other Dynatrace code modules and provides:

• Go-specific metrics
• Service identification and service-related data acquisition
• Full code-level visibility into the service/transaction and background processing level
• Cloud Foundry Gorouter metrics

Go-specific metrics

The Go runtime system is unique with respect to scheduling and handling of massively parallel workloads. Detailed insight into key execution metrics with real-world load profiles is, therefore, key to optimizing application execution and to pinning down processing bottlenecks (for example, goroutine leaks).

The OneAgent code module for Go collects a large set of metric data—from high-level garbage collection and heap statistics to expert level key performance indicators of Go runtime internal entities.

Check out this blog post for a detailed breakdown of Go metrics.

Services

Service identification and service-related data acquisition are key features of the OneAgent code module for Go. The code module fully automatically supports end-to-end request monitoring. Learn more about how to gain full insight into incoming and outgoing HTTP requests in this blog post.

Custom services add service-level visibility to application-specific, proprietary components. Once defined, these custom service response times and errors become part of Dynatrace AI anomaly detection and root cause identification.

Full code-level visibility

The OneAgent code module for Go provides full code-level visibility into the service/transaction and background processing. Hotspot analysis shows the function call hierarchy of specific requests. The CPU profiler shows functions called in background (non-service related) processing.

Read full details about code-level visibility in this blog post.

Cloud Foundry Gorouter metrics

With Dynatrace, you get insights into the overall traffic flow of each Cloud Foundry foundation based on the number of total requests aggregated across all Gorouters. Furthermore, Dynatrace indicates if Gorouter processes have a negative impact on your overall application responsiveness based on response latency. Repeatedly crashing applications or response issues from applications typically result in an increasing number of HTTP 5xx responses, which Dynatrace OneAgent also detects.

Please find all details around Cloud Foundry Gorouter metrics in this blog post.

How to enable code-level monitoring of Go processes in Dynatrace

The Dynatrace Go team is excited to announce the code module for Go being promoted from beta to general availability status with OneAgent version 1.157 and Dynatrace SaaS and managed version 1.160. Go technology is enabled by default for new environments. For existing environments, the current state of Go technology monitoring is preserved. In such cases, you can enable Go monitoring within Dynatrace settings at Settings > Monitoring > Monitored technologies.

As Go is a general-purpose programming language used for a broad range of applications, not all Go processes are automatically monitored by Dynatrace. Like .NET technology, monitoring of Go processes is opt-in. Nonetheless, monitoring of processes can be easily controlled with monitoring rules—please refer to this blog post for more details.

Dynatrace enables monitoring of well-known Go applications with a set of built-in monitoring rules.

The list of built-in rules can be viewed in Dynatrace settings at Settings > Process groups > Process group monitoring > Built-in process monitoring rules.

#devlife: the journey of the Go team in the last year and a half

Since the initial announcement of the OneAgent code module for Go at the end of August 2017, the Go team has continuously extended code module functionality.

The comparably small feature set of the initial release triggered some outside skepticism about whether the Dynatrace Go monitoring approach would be sustainable and capable to evolve towards code-level visibility. Knowing that much of the core functionality was hidden in this initial release, the Dynatrace Go team kept its cool. Our unique Go monitoring approach had yet to prove itself in the field and therefore the decision was to release early to gather experience and collect feedback. To the team’s delight, the initial release went smoothly—the code module was widely enabled and its operation was extremely stable. Intensive Dynatrace internal-testing and design for robustness fully paid off.

Encouraged by this smooth launch, the next big feature increment was released five months later in March 2018. End-to-end request monitoring was the central, predominant feature of this milestone release. For the first time ever Go processes made their contributions to Dynatrace PurePath! The release also contained many less visible refinements—many of them originating from valuable customer feedback.

In the same time frame the first and, so far, the only serious code module issue surfaced in production. The issue was unrelated to the recently released functionality but caused an operational issue in combination with specific versions of OpenShift processes. The OpenShift binaries had been built with a proprietary Go SDK version which included cherry-picked features from later Go releases.

The issue is worth mentioning for two reasons. First, it is part of the code module’s biography. Even acting with the greatest care, that specific use case had not been foreseen. Second, root cause analysis and provisioning of an initial fix was extremely quick. A more sophisticated detection of “home brewed” Go toolchains was then added to the code module.

The best time to adopt the Go code module to new Go releases is a recurring question. Beside feature development, this is an important perpetual Dynatrace Go team task. The team permanently monitors the evolution of the Go programming language and changes to the Go SDK. Experience over the last 1.5 years shows that support for a new Go version takes about 1-3 weeks in engineering. Regression testing and passing product acceptance typically adds another 4-8 weeks. This schedule applies to all (even patch level) Go releases. A detailed Go version support matrix can be found in Dynatrace Help.

July 2018 finally delivered full code-level visibility. It’s my personal favorite pick among all Go code module features—technically challenging and highly visible (and real fun to blog about). The feature is an integration of numerous smaller functionalities in the code module. The team performed a lot of Go standard runtime research and design meetings. Many smaller obstacles had to be overcome, and assumptions had to be adjusted. Nonetheless, all the small parts integrated nicely and the whole system started working right from the start.

The recently released custom services feature was the final large feature increment planned for the general availability milestone.

The Go code module’s matured feature set plus proven stability and robustness finally allowed it to pass the quality gates to reach general availability status.

What’s coming next

As already hinted at in our custom services blog post, GRPC and databases are the next big things for the Go code module. Many more, smaller-scale features are already in the pipeline to be released. For example, Go 1.11 modules are an enabler for reporting a Go application’s packages and their version numbers.

Credits

• Gopher side and Gopher front graphics published under Creative Commons 3.0 Attributions license.
• Crowbar graphics published under GNU Free Documentation License