Full code-level visibility now available for Go-application monitoring

The Dynatrace Go team is proud to announce full code-level visibility into the performance of your Golang-based applications, without requiring any changes to your code or your application (requires OneAgent v1.145 or above).

Jelly on the belly – Ultrasound for our Golang baby

Every parent has indelible memories of the first ultrasound images they saw of their offspring while their offspring continued to develop in the womb. Ultrasound images invite parents to zoom in on areas of interest, explore, categorize (will the baby’s room be pink or blue?), and to observe responses to stimuli. While the Dynatrace OneAgent for Go team can’t claim the same joy as a new parent, we had a similar surge of joy when OneAgent v1.145 was first injected into a Cloud Foundry Gorouter process and achieved full code-level monitoring visibility with Dynatrace. And, we did zoom in, explore, categorize, and happily tamper with the process inputs without any interruption from hospital staff.

It’s Go code-level visibility time

The analogy to ultrasound examinations is quite striking. Dynatrace OneAgent interferes as little as possible with the applications it monitors while exploring the performance of those applications at the deepest possible level. While our top priority is preserving the integrity of the applications we monitor, OneAgent has a largely diametrically opposed purpose—to reveal all details of application processing.

Go applications are unique in that they compile to binary executables and are executed on bare metal by the CPU. Go lacks the abstracting layer of a virtual machine with defined monitoring hooks, like Java JVM or .NET CLR. Therefore, Dynatrace OneAgent needs to operate on the CPU instruction level to intercept Go function calls. Go parallel processing capabilities are quite unique and OneAgent is therefore carefully crafted so that it doesn’t introduce implicit synchronization points between Goroutines by capturing monitoring data.

All this required a deep and thorough understanding of Go application execution and made it tricky to implement a fully auto-instrumenting OneAgent for Go technology. This is what makes the Go team most proud of our Golang offspring—knowing that Dynatrace OneAgent for Go is unique in its capabilities. It’s the only solution in existence for monitoring Go applications without source code changes or even application recompilation.

The Cloud Foundry BBS showcase

We’ve selected Cloud Foundry Diego’s BBS (Bulletin Board System) to showcase the new OneAgent capabilities. BBS is the central data store and orchestrator of a Diego cluster that uses protocol buffer transporting HTTP requests to communicate with other Cloud Foundry instances.

The process view example below represents a single BBS service running in a Cloud Foundry deployment. Note the new HTTP metrics (for details, see our recent Gorouter metrics blog post).

The default web request service summary page is shown below.

Within the default web request service execution, Dynatrace has identified two hotspots. Let’s first drill down to the high service resource consumption hotspot.

Hotspots view unfolds the critical code execution path for entry point /v1/actual_lrps/start to the executing Go handler function StartActualLRP. The acronym “LRP” stands for Long Running Process. Drilling further down into the StartActualLRP execution path reveals requests to the ActualLRP MySQL database.

Next, we investigate the service response time hotspot that Dynatrace has identified.

Entry point /v1/desired_lrp/desire.r2 triggers an outgoing request to the Cloud Foundry auctioneer component. This client web request is contributing most of the request response time.

Expanding the stack reveals the composition of the LRP auction request towards auctioneer. The various wrap and record functions indicate usage of a frequently used pattern for wrapping request handlers in Lambda functions to execute code before and after invoking the handler function.

The final stage of our journey into the processing details of BBS features the Dynatrace CPU profiler. The CPU Profiler enables you to investigate code that’s executed outside the context of a web request (for example, background activities and schedulers).

Dynatrace OneAgent understands how much of your code is executed and the amount of CPU that’s consumed by each web request. Background thread CPU and Service request CPU are tracked separately. For example, the Go net/http package (see below) spawns a new Goroutine for each incoming web request (more precisely, each new connection). Thus, the Goroutine executing the web request handler function is tagged as service-related. Other Goroutines, which are started in advance of processing the web request, are also considered to be part of the service execution and are therefore also tagged. On the other hand, processing done by Goroutines that don’t have the service related tag are accounted for as background activity.

The result of this categorization can be seen in the background activity Hotspots view below.

The function names of most of the call trees listed indicate that they do indeed execute background tasks. We can identify the call tree of the BBS Converger process, a worker pool that efficiently executes work items in a set of pooled Goroutines, the socket level connection part of web request processing, and a call tree of ifrit, a process management package hosted on github.com.

What’s next

The Dynatrace OneAgent for Go team is already working on the next big things to come. I’ve overheard discussions mentioning “custom services” and I’ve noticed some unusual behavior in some Cloud Foundry processes. So, stay tuned.

Credits

Kudos must be given to the marvelous team that drives OneAgent for Go development. Michael Obermueller and Peter Feichtinger do an awesome job advancing the development of this challenging technology support.

Special thanks for the ultrasound Gopher illustration go to Karen Bangoyan.