> For the complete documentation index, see [llms.txt](https://docs.groundcover.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.groundcover.com/integrations/data-sources/opentelemetry/sending-from-kubernetes-pods.md).

# Sending from Kubernetes Pods

## Intro and requirements

OTLP signals generated from Kubernetes pods can be ingested directly by our DaemonSet `Sensor`.

The recommend method is is by exporting the telemetry data from the pods to the `groundcover-sensor` Service Endpoint as described below.

{% hint style="success" %}
The sensor will also enrich the received spans and logs with Kubernetes metadata.
{% endhint %}

{% hint style="warning" %}
groundcover does not instrument your services with OpenTelemetry for you. The following assumes the services have already been instrumented with OpenTelemetry.
{% endhint %}

## Configuring the Exporter

### Setting Up The Required Environment Variables

{% hint style="info" %}
Ingestion is supported for both OTLP/HTTP and OTLP/gRPC.\
Choose what your instrumentation is using - it's often part of the instrumentation itself.
{% endhint %}

Apply the environment variables below to your services in order to make them ship data to groundcover's ingestion endpoint.

#### OTLP/HTTP

```yaml
env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://groundcover-sensor.groundcover.svc.cluster.local:4318"
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: "http/protobuf"
```

#### OTLP/gRPC

```yaml
env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "groundcover-sensor.groundcover.svc.cluster.local:4317"
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: "grpc"
```

## Attribute Enrichment

#### Automatic Enrichment

groundcover will automatically enrich data ingested by the sensors with extra Kubernetes metadata, in order to provide as much context as possible.

{% hint style="success" %}
groundcover follows [OpenTelemetry's Semantic Conventions](https://opentelemetry.io/docs/concepts/semantic-conventions/) when naming the attributes.
{% endhint %}

{% hint style="success" %}
groundcover will replace the `service.name` attribute indicating the name of the service with the name of the Kubernetes Deployment which is the owner of the pod. Keep this in mind when looking for your traces and logs in the system!
{% endhint %}

**Top Level attributes:**

You can enrich your data with top-level attributes such as workload name, environment type, and environment name. These can be passed either through resource attributes or HTTP headers.

{% hint style="info" %}
For detailed information on enriching your data with headers and resource attributes, see our [Enriching Data with Headers](https://github.com/groundcover-com/docs/blob/main/integrations/data-sources/enriching-data-with-headers.md) guide.
{% endhint %}

**Pod Level attributes:**

* `k8s.namespace.name` - the namespace of the pod
* `k8s.node.name` - the node the pod is scheduled on
* `k8s.pod.name` - the name of the pod
* `k8s.pod.uid` - the UID of the pod
* `k8s.pod.ip` - the IP address of the pod at the time of the trace
* `k8s.cluster.name` - the Kubernetes cluster name

**Container level attributes:**

{% hint style="info" %}
The following enrichment requires `container.id` to be present on the span
{% endhint %}

* `container.name` - the name of the container
* `container.image.name` - the name of the container image
* `container.image.tag` - the tag of the container image

#### Manual Enrichment via Headers

It is also possible to enrich specific attributes using headers - read more [here](/integrations/data-sources/enriching-3rd-party-data.md).

See below for an example on adding `workload` and `env` via headers:

```yaml
env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://groundcover-sensor.groundcover.svc.cluster.local:4318"
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: "http/protobuf"
  - name: OTEL_EXPORTER_OTLP_HEADERS
    value: "x-groundcover-service-name=my-service,x-groundcover-env-name=my-env"
```

## Sampling

While some instrumentation libraries allow sampling of traces, it can be convenient to sample a ratio of the incoming traces directly in groundcover.

{% hint style="info" %}
The default sampling ratio for traces ingested by the sensor is 5%
{% endhint %}

{% hint style="warning" %}
groundcover sampling does not take into account sampling being done in earlier stages (e.g SDK or collectors). It's recommended to choose one point for sampling.
{% endhint %}

The `samplingRatio` field is a fraction in the range 0-1. For example, 0.1 means 10% of the incoming traces will be sampled and stored in groundcover.

To configure sampling, the relevant values can be used:

```yaml
agent:
  sensor:
    apmIngestor:
      otel:
        direct:
          samplingRatio: 0.05
```

#### Configuring 100% Sampling Ratio (No Sampling)

Use the values below to disable sampling and ingest 100% of the incoming traces.

```yaml
agent:
  sensor:
    apmIngestor:
      otel:
        direct:
          samplingRatio: 1
```

### Troubleshooting

#### Common Issues

**Connection Refused / Timeout**

* Verify the sensor endpoint URL
* Check application logs for OTLP exporter errors

**Data Not Appearing**

* Check application logs for OTLP exporter errors
* Verify the service name and resource attributes are set
* Allow 1-2 minutes for data to appear in groundcover UI
* For low-volume environments like development clusters, increase sampling ratio to 1 or increase the volume of requests sent to the service


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.groundcover.com/integrations/data-sources/opentelemetry/sending-from-kubernetes-pods.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.