Controlling the eBPF sampling mechanism

This page addresses the sampling of eBPF traces generated by the groundcover sensors. For sampling of traces generated by external libraries, see OpenTelemetry & DataDog. Can't find what you're looking for? Let us know over Slack!

groundcover utilizes smart sampling in order to only store the traces which we believe are the most important for monitoring your environment. However, some more advanced use cases might require adjusting the sampling strategy, making sure you get the exact coverage you need.

This page details the way in which sampling can be configured.

Force Sampling

Some cases might require 100% visibility into traces of specific services or APIs. groundcover allows forcing sampling of transactions by these methods:

By request header (HTTP/gRPC only)

HTTP/gRPC requests which include the header below will be force sampled.

x-groundcover-force-sample: true

By pod labels / annotations

Pods which have the following value as either a label and/or annotation will have all traces sampled:

groundcover/force-sample: "true"

Rate Limiting

The sampling mechanism can be rate limited to reduce the total amount of traces produced. Note that traces marked as issues will not be affected by this configuration.

Rate is specified by number of traces allowed per 100 milliseconds, using the following values override:

agent:
  sensor:
    env:
      - name: GC_GLOBALLIMITER_MAXQUEUE
        value: 10 # will limit to a rate of 100 traces / second