Introducing custom services for Go applications

Service to be or not to be… is now up to you

After giving birth to full code level visibility, the Dynatrace Go team relentlessly pushed for the next one and only, world exclusive Go application monitoring feature. Exceeding initial expectations, they once again proved the skeptics wrong about what can be achieved.

Dynatrace is proud to announce custom services for Go applications. Associate services with arbitrary functions in any Go application—as always, without source code modifications.

Custom services at a glance

Dynatrace OneAgent already detects web applications written in Go automatically. It also provides code-level visibility into method hotspots for requests and background tasks. With this in mind, we’ve looked at support for proprietary application-specific services. What we wanted to do was to give our customers the ability to monitor any Go application service, without code changes, even if we have no specific support for it. Starting with OneAgent version 1.155, custom services will allow this. A custom service can be used to monitor functions in any Go application in minutes.

Custom services serve multiple purposes.

• Add service level visibility to application-specific components.
• Response time baselining. This enables the Dynatrace AI to automatically detect changes in response time and their root cause.
• Failure detection and baselining. Not only does this detect if a custom service failed due to an error, the AI will also be able to detect the root cause and determine if the failure impacts end-user experience.
• Deepen visibility into your application; this will give you request level end to end traces and method hotspots.

Introducing GoCron

To illustrate how to define Go custom services we’ve implemented a simple cron-like job scheduler called GoCron. It accepts job registration requests on port 8000 and will execute the application specified in the job definition periodically. The console output of finished jobs is sent as plain text to an HTTP server listening on port 8001.
GoCron uses github.com/robfig/cron, and to keep things short and crispy we boldly skip error handling. Nevertheless, you can find a more elaborate version of GoCron on GitHub.

Note that there is no Dynatrace-specific code in there. With a quick go build GoCron.go, our setup is ready to go. Dynatrace identifies GoCron as a Go process and starts monitoring it automatically.

As expected, the incoming job registration requests are detected by Dynatrace as an HTTP web request service automatically. The scheduled cron jobs are executed outside of the registration request context and are therefore seen by Dynatrace as background activity.

Let’s get started

To elevate the background cron job execution to a first-class service citizen, we will define a custom service as follows.

1. Go to Settings > Server-side monitoring > Custom service detection.
2. Click the Go services tab.
3. Click Define Go services and type a meaningful name for the service (e.g., Cron Job Service).
   
4. Select the GoCron process group that contains the entry points.
5. Now you need to find the scope of the function which runs the cron jobs.
   
   In our example the scope is main.(*CronJob), because the function we are looking for is defined as a method of type CronJob in package main.
6. Select the function Run() as an entry point for the custom service.
   
7. Click Save Changes to enable the new custom Go service.
   
8. Lastly, you need to restart the GoCron process to allow Dynatrace OneAgent to instrument all entry points defined in custom Go services.

Let’s send a job registration request to GoCron:

curl -X POST "http://localhost:8000/register?command=PrintTime.sh&schedule=@every%201s"

This schedules the shell script PrintTime.sh to run periodically every second, which simply prints the current time to the console.

Consequently, our newly defined service will appear in Smartscape, Service flows, and on its own standalone service overview page:

Impressive, right? So far, we know how long and how often each cron job is executed on average, which brings quite good visibility into our code. But what about the underlying third party cron package?

Digging deeper

We especially want to see:

1. When a new job is added
2. When cron runs our jobs.

The first point is simple, we need to instrument the function AddJob() which is well known from the public API.  The second one requires knowledge about the implementation of cron. The call trees in Dynatrace CPU Profiler come in handy to find good entry point candidates within third party code. In case of GoCron we can easily identify runWithRecovery() which is responsible for invoking the installed application handler function.

Let’s deepen the visibility of our existing PurePaths by defining a second custom Go service. It takes exactly the same steps as described before, but instead of main.(*CronJob).Run(), we use AddJob() and runWithRecovery() defined in scope github.com/robfig.cron(*Cron) as entry points:

Register job services:

Run job services:

This shows that we now have deep visibility, both into our own code as well as third-party code.

To summarize, custom services are an essential instrument in your Dynatrace monitoring toolbox to highlight and survey application-specific processes. You’ll love them!

Remarks on closures in Go

Closures are anonymous functions for which the Go compiler assigns special names. The generated closure names defined in a scope are numbered in order of their appearance: func1, func2, and so on. For example, the closure in function main in the snippet above is assigned the name func1:

What’s next

There are many great new features ahead for Go application full-stack monitoring with Dynatrace. Next, we will add deep monitoring for database and GRPC calls performed by your application. So, stay tuned!
In the meantime, let us know how you are using custom services to bring deep visibility into your Go applications. Please share your feedback with us at Dynatrace Community.

Attention

• Go custom service support requires OneAgent version 1.155 or later.
• Go custom service is available in Dynatrace SaaS.
• Go custom service is available in Dynatrace managed v156 or later.
• Go support is still in beta. To enable it, go to Settings > Monitoring > Monitored technologies and set the Go switch to the On position.