Enrich logs, traces, events, and metrics with pod labels & annotations

groundcover allows you to add custom labels and annotations from Kubernetes pods and Docker containers to the traces, logs, events, and metrics generated by those pods or containers.

Specifying what to collect

To apply custom labels and annotations collection, you need to update your groundcover deployment values.yaml file.

To collect a pod label named app.kubernetes.io/name and an annotation named app.kubernetes.io/other-name you will need to add the following configuration:

agent:
  sensor:
    customEntityTags:
      labels:
        keys: ["app.kubernetes.io/name"]
        collectAll: false
        promoteToMetrics:
          keys: []
          all: true
      annotations:
        keys: ["app.kubernetes.io/other-name"]
        collectAll: false
        promoteToMetrics:
          keys: []
          all: true

Enriching metrics

You can promote a subset of your collected labels and annotations to metrics using the promoteToMetrics option under labels and/or annotations. Only the labels/annotations specified here are added to metrics; all collected labels and annotations are still enriched to logs, traces, and events. Use keys to specify which labels/annotations to promote, or set all: true to promote all collected labels/annotations to metrics.

To collect all pod labels and/or annotations, use the following configuration:

agent:
  sensor:
    customEntityTags:
      labels:
        keys: []
        collectAll: false
      annotations:
        keys: []
        collectAll: true

We prefix the labels/annotations with the following (according to pod/docker):

• k8s.pod.label

• k8s.pod.annotation

• docker.container.label

• docker.container.annotation

pod labels in a span attributes

Searching labels and annotations

Once you've setup the configuration to the sensor, you will be able to view and search for specific labels and annotations like any other attribute. For example, to search for traces, logs, or events that have the label app.kubernetes.io/part-of = groundcover you can do the following search: k8s.pod.label.app.kubernetes.io/part-of:groundcover in the search bar.

auto-complete label values from traces