Controlling the eBPF sampling mechanism

This page addresses the sampling of eBPF traces generated by the groundcover sensors. For sampling of traces generated by external libraries, see OpenTelemetry & DataDog. Can't find what you're looking for? Let us know over Slack!

groundcover utilizes smart sampling in order to only store the traces which we believe are the most important for monitoring your environment. However, some more advanced use cases might require adjusting the sampling strategy, making sure you get the exact coverage you need.

This page details the way in which sampling can be configured.

Force Sampling

Some cases might require 100% visibility into traces of specific services or APIs. groundcover allows forcing sampling of transactions by these methods:

Force sampling lifts the sampling limits. It does not change which entities are traced, so a namespace, workload or container removed by traces filtering rules stays untraced.

By request header (HTTP/gRPC only)

HTTP/gRPC requests which include the header below will be force sampled. Any value counts, including false — the header force samples by being present.

x-groundcover-force-sample: true

By pod label

Pods carrying this label have all their traces sampled, in every container in the pod:

groundcover/force-sample: "true"

Apply on a running pod:

kubectl label pod <pod_name> groundcover/force-sample=true --overwrite

Persist the label in the Deployment's pod template so new pods keep it. This triggers a rollout:

kubectl patch deployment <name> --type merge \
  -p '{"spec":{"template":{"metadata":{"labels":{"groundcover/force-sample":"true"}}}}}'

By namespace label

The same key on a Namespace force samples the pods in it, including pods created later, with no pod restart:

kubectl label namespace <namespace> groundcover/force-sample=true --overwrite

Pods take precedence over namespaces, so you can force a whole namespace and still opt one noisy pod out:

kubectl label pod <pod_name> groundcover/force-sample=false --overwrite

Labels tell you what is configured, not what is in effect — a pod force sampled through its namespace carries no label of its own. To see what was actually force sampled, and by which level, filter traces on the span attribute tracing.gc.forced_sample_source:

tracing.gc.forced_sample_source: pod
tracing.gc.forced_sample_source: namespace
tracing.gc.forced_sample_source: header.groundcover

Accepted label values

true force samples. false does not, and goes back to default sampling — overriding a namespace set to true. Removing the key leaves the namespace to decide.

Remove the key with the trailing - form:

kubectl label namespace <namespace> groundcover/force-sample-

The namespace watcher

Namespace force sampling relies on the sensor watching namespaces, which is on by default. Turning that watch off makes namespace labels stop having any effect:

agent:
  sensor:
    k8sEntitiesWatch:
      namespace:
        enabled: false

Rate Limiting

The sampling mechanism can be rate limited to reduce the total amount of traces produced. Note that traces marked as issues will not be affected by this configuration.

Rate is specified by number of traces allowed per 100 milliseconds, using the following values override:

agent:
  sensor:
    env:
      - name: GC_GLOBALLIMITER_MAXQUEUE
        value: 10 # will limit to a rate of 100 traces / second