Organize Kubernetes/OpenShift deployments by tags

Dynatrace automatically derives tags from your Kubernetes/OpenShift labels. This enables you to automatically organize and filter all your monitored Kubernetes/OpenShift application components.

Recommendations

We recommend that you define additional metadata at the deployed system. For Kubernetes-based applications, you can simply use Kubernetes annotations. Dynatrace automatically detects and retrieves all Kubernetes and OpenShift annotations for pods that are monitored with a OneAgent code module. This enables you to use automated tagging rules, based on existing or custom metadata, to define your filter sets for charts, alerting, and more. These tags and rules can be changed and adapted any time and will apply almost immediately without any change to the monitored environment or applications.

In Dynatrace, you can specify entity ownership for different Kubernetes objects such as Deployments, Pods, Services, or namespaces. We recommend providing ownership information via Kubernetes labels or annotations (you can use either labels or annotations to attach metadata to Kubernetes objects). This ensures that Kubernetes objects have adequate ownership coverage, which is especially important for short-lived entities like Pods. See Assign ownership teams to monitored entities for more on the correct format and examples of providing ownership information in key-value pairs in the deployment specification file.

We recommend defining ownership for the Deployment and all other objects for which you want ownership coverage. See also Best practices for entity ownership. You can assign more than one team to a Kubernetes object, provided that the keys in the key-value pairs are unique.

While you can also use tags (manual, automated, and via API) to apply ownership information to Kubernetes objects, this approach has its limitations—read more in Best practices for entity ownership.

Automatic detection of Kubernetes properties and annotations

Dynatrace detects Kubernetes properties and annotations. Such properties and annotations can be used when specifying automated rule-based tags.

Additionally Dynatrace detects the following properties that can be used for automated rule-based tags and property-based process group detection rules.

  • Kubernetes base pod name: User-provided name of the pod the container belongs to.
  • Kubernetes container: Name of the container that runs the process.
  • Kubernetes full pod name: Full name of the pod the container belongs to.
  • Kubernetes namespace: Namespace to which the containerized process is assigned.
  • Kubernetes pod UID: Unique ID of the related pod.

Leverage Kubernetes labels in Dynatrace

Kubernetes-based tags are searchable via Dynatrace search. This allows you to easily find and inspect the monitoring results of related processes running in your Kubernetes or OpenShift environment. You can also leverage Kubernetes tags to set up fine-grained problem alerting profiles. Kubernetes tags also integrate perfectly with Dynatrace filters.

Import your labels and annotations

Requirements

For OneAgent to detect Kubernetes annotations and properties, make sure that

  • Pods are monitored with a code module
  • automountServiceAccountToken: false isn't set in your pod's spec

You can specify Kubernetes labels in the deployment definition of your application or you can update the labels of your Kubernetes resources using the command kubectl label.

Dynatrace automatically detects all labels attached to pods at application deployment time. All you have to do is grant sufficient privileges to the pods that allow for reading the metadata from the Kubernetes REST API endpoint. This way, the OneAgent code modules can read these labels directly from the pod.

Grant viewer role to service accounts

In Kubernetes, every pod is associated with a service account which is used to authenticate the pod's requests to the Kubernetes API. If not otherwise specified the pod uses the default service account of its namespace.

Every namespace has its own set of service accounts and thus also its own namespace-scoped default service account. The labels of each pod for which the service account has view permissions will be imported into Dynatrace automatically.

The following steps show you how to add view privileges to the default service account in the namespace1 namespace. You need to repeat these steps for all service accounts and namespaces you want to enable for Dynatrace.

Create the following Role and RoleBinding, which allow the default service account to view the necessary metadata about your namespace namespace1 via the Kubernetes REST API:

# dynatrace-oneagent-metadata-viewer.yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: namespace1
  name: dynatrace-oneagent-metadata-viewer
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: dynatrace-oneagent-metadata-viewer-binding
  namespace: namespace1
subjects:
- kind: ServiceAccount
  name: default
  apiGroup: ""
roleRef:
  kind: Role
  name: dynatrace-oneagent-metadata-viewer
  apiGroup: ""
kubectl -n namespace1 create -f dynatrace-oneagent-metadata-viewer.yaml

As a result, Kubernetes processes monitored in your Dynatrace environment will have Kubernetes labels attached as Kubernetes tags. For namespaces, pods, and workloads, Kubernetes tags are not evaluated.

Related topics