> For the complete documentation index, see [llms.txt](https://docs.groundcover.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.groundcover.com/operate/customize-usage/controlling-the-ebpf-sampling-mechanism.md).

# Controlling the eBPF sampling mechanism

{% hint style="info" %}
This page addresses the sampling of eBPF traces generated by the groundcover sensors.\
For sampling of traces generated by external libraries, see [OpenTelemetry](broken://pages/zj2qhGmf3BoJ3yOOPd4M#sampling-incoming-traces) & [DataDog](broken://pages/HYp4iENWF8Mcd05MAGGb#sampling-incoming-traces).\
\
Can't find what you're looking for? [Let us know over Slack!](https://www.groundcover.com/join-slack)
{% endhint %}

groundcover utilizes [smart sampling](/investigate/traces-and-apm/traces.md#sampling) in order to only store the traces which we believe are the most important for monitoring your environment. However, some more advanced use cases might require adjusting the sampling strategy, making sure you get the exact coverage you need.

This page details the way in which sampling can be configured.

### Force Sampling

Some cases might require 100% visibility into traces of specific services or APIs. groundcover allows forcing sampling of transactions by these methods:

Force sampling lifts the sampling limits. It does not change which entities are traced, so a namespace, workload or container removed by [traces filtering rules](/operate/customize-usage/filtering-kubernetes-entities.md) stays untraced.

#### By request header (HTTP/gRPC only)

HTTP/gRPC requests which include the header below will be force sampled. Any value counts, including `false` — the header force samples by being present.

```
x-groundcover-force-sample: true
```

#### By pod label

Pods carrying this label have all their traces sampled, in every container in the pod:

```
groundcover/force-sample: "true"
```

Apply on a running pod:

```bash
kubectl label pod <pod_name> groundcover/force-sample=true --overwrite
```

Persist the label in the Deployment's pod template so new pods keep it. This triggers a rollout:

```bash
kubectl patch deployment <name> --type merge \
  -p '{"spec":{"template":{"metadata":{"labels":{"groundcover/force-sample":"true"}}}}}'
```

#### By namespace label

The same key on a **Namespace** force samples the pods in it, including pods created later, with **no pod restart**:

```bash
kubectl label namespace <namespace> groundcover/force-sample=true --overwrite
```

Pods take precedence over namespaces, so you can force a whole namespace and still opt one noisy pod out:

```bash
kubectl label pod <pod_name> groundcover/force-sample=false --overwrite
```

Labels tell you what is *configured*, not what is in effect — a pod force sampled through its namespace carries no label of its own. To see what was actually force sampled, and by which level, filter traces on the span attribute `tracing.gc.forced_sample_source`:

```
tracing.gc.forced_sample_source: pod
tracing.gc.forced_sample_source: namespace
tracing.gc.forced_sample_source: header.groundcover
```

#### Accepted label values

`true` force samples. `false` does not, and goes back to default sampling — overriding a namespace set to `true`. Removing the key leaves the namespace to decide.

Remove the key with the trailing `-` form:

```bash
kubectl label namespace <namespace> groundcover/force-sample-
```

#### The namespace watcher

Namespace force sampling relies on the sensor watching namespaces, which is on by default. Turning that watch off makes namespace labels stop having any effect:

```yaml
agent:
  sensor:
    k8sEntitiesWatch:
      namespace:
        enabled: false
```

### Rate Limiting

The sampling mechanism can be rate limited to reduce the total amount of traces produced. Note that traces marked as issues will not be affected by this configuration.

Rate is specified by number of traces allowed per 100 milliseconds, using the following values override:

```
agent:
  sensor:
    env:
      - name: GC_GLOBALLIMITER_MAXQUEUE
        value: 10 # will limit to a rate of 100 traces / second
```


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.groundcover.com/operate/customize-usage/controlling-the-ebpf-sampling-mechanism.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.