OpenTelemetry basics: Trace ingest to Dynatrace
Get started with OpenTelemetry and Dynatrace with our tutorial.
OpenTelemetry is powerful but can be challenging to get started with. You've probably already read the documentation and spun up a demo system.
But a lot of what you've seen is "magic." A lot happens behind the scenes. Sometimes, it is helpful to strip it back and start from the basics to understand what is truly happening.
TL;DR Experiment with the In-Browser Live Demo Here
A basic OpenTelemetry setup
The most basic OpenTelemetry setup has three components:
- An OpenTelemetry trace generator. Usually, an application that emits OTEL data (metrics, logs, traces, etc.).
- An OpenTelemetry collector. The aggregation point for all that data.
- A storage back end. That's where you want to send the data. In this demo, the back end will be Dynatrace.
Of course, you can have multiple components, but the above is the most basic setup.
What will be built
In this demo, tracegen generates trace data that will be sent to an OpenTelemetry collector.
The collector will be configured to pass the trace data to Dynatrace.
In real life, tracegen would be replaced by your application(s). Of course, if you choose to install the OneAgent on your hosts, the OpenTelemetry collector becomes an optional component.
Create a new Docker network
By default, Docker starts containers on the bridge network. In this mode, containers can only address each other via their IPs.
That's a bit clunky so let's make our lives easier and create a new network called tracegen-demo-net just for this demo. That way, containers can address each other via their name instead — which is easier for us humans and easier to automate.
docker network create tracegen-demo-net
You can see all Docker networks with the following command.
docker network ls
As a result, you should see four networks, one of which is tracegen-demo-net.
Run the collector
Run the collector on the tracegen-demo-net network.
Give the container a name: otelcol, which means other containers can find it via the name instead of a unique ID or IP (which will change at some point).
docker run \
--net tracegen-demo-net \
--name otelcol \
-v $(pwd)/config.yaml:/etc/otelcol-contrib/config.yaml \
ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.63.0
Generate trace data
Open a second terminal window. Run the trace generator utility.
Again, give the container a name (just for consistency). Also, set the network, so the container runs in the same network. The following command does all that for you.
Notice we're passing the endpoint as otelcol:4317 which means "send trace data to the otelcol container on port 4317" (the default OpenTelemetry gRPC port).
The -otlp-insecure flag is needed because we don't have a proper TLS cert configured on the demo system.
docker run \
--net tracegen-demo-net \
--name tracegenerator \
ghcr.io/open-telemetry/opentelemetry-collector-contrib/tracegen:v0.63.0 -traces 1 -otlp-insecure -otlp-endpoint otelcol:4317
If it works, you should see lots of output that ends with: stopping the exporter.
View traces in Dynatrace
Open Dynatrace, go to Distributed Traces and then toggle over to Ingested traces.
In a few seconds, you should see a trace called lets-go (that's what tracegen names the traces it sends).
Generate more trace data by re-running:
docker run \
--net tracegen-demo-net \
--name tracegenerator \
ghcr.io/open-telemetry/opentelemetry-collector-contrib/tracegen:v0.63.0 -traces 1 -otlp-insecure -otlp-endpoint otelcol:4317
Of course, suppose you have the OneAgent deployed on the host. In that case, a lot of this complexity vanishes: You don't necessarily need to instrument your applications (although if you do, OneAgent will respect those traces and ingest them), and you probably don't need an OpenTelemetry collector either.
OpenTelemetry basics: Trace ingest to Dynatrace was originally published in Dynatrace Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.
Written by