Sending from Kubernetes Pods

Intro and requirements

OTLP signals generated from Kubernetes pods can be ingested directly by our DaemonSet Sensor.

The recommend method is is by exporting the telemetry data from the pods to the groundcover-sensor Service Endpoint as described below.

The sensor will also enrich the received spans and logs with Kubernetes metadata.

groundcover does not instrument your services with OpenTelemetry for you. The following assumes the services have already been instrumented with OpenTelemetry.

Configuring the Exporter

Finding the groundcover Sensor Service Endpoint

Use the instructions here to locate the endpoint for the Sensor service, referenced below as {GROUNDCOVER_SENSOR_ENDPOINT}.

Setting Up The Required Environment Variables

Ingestion is supported for both OTLP/HTTP and OTLP/gRPC. Choose what your instrumentation is using - it's often part of the instrumentation itself.

Apply the environment variables below to your services in order to make them ship data to groundcover's ingestion endpoint.

OTLP/HTTP

env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://{GROUNDCOVER_SENSOR_ENDPOINT}:4318"
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: "http/protobuf"

OTLP/gRPC

env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "{GROUNDCOVER_SENSOR_ENDPOINT}:4317"
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: "grpc"

Attribute Enrichment

Automatic Enrichment

groundcover will automatically enrich data ingested by the sensors with extra Kubernetes metadata, in order to provide as much context as possible.

groundcover follows OpenTelemetry's Semantic Conventions when naming the attributes.

groundcover will replace the service.name attribute indicating the name of the service with the name of the Kubernetes Deployment which is the owner of the pod. Keep this in mind when looking for your traces and logs in the system!

Top Level attributes:

You can enrich your data with top-level attributes such as workload name, environment type, and environment name. These can be passed either through resource attributes or HTTP headers.

For detailed information on enriching your data with headers and resource attributes, see our Enriching Data with Headers guide.

Pod Level attributes:

k8s.namespace.name - the namespace of the pod
k8s.node.name - the node the pod is scheduled on
k8s.pod.name - the name of the pod
k8s.pod.uid - the UID of the pod
k8s.pod.ip - the IP address of the pod at the time of the trace
k8s.cluster.name - the Kubernetes cluster name

Container level attributes:

The following enrichment requires container.id to be present on the span

container.name - the name of the container
container.image.name - the name of the container image
container.image.tag - the tag of the container image

Manual Enrichment via Headers

It is also possible to enrich specific attributes using headers - read more here.

See below for an example on adding workload and env via headers:

env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://{GROUNDCOVER_SENSOR_ENDPOINT}:4318"
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: "http/protobuf"
  - name: OTEL_EXPORTER_OTLP_HEADERS
    value: "x-groundcover-service-name=my-service,x-groundcover-env-name=my-env"

Sampling

While some instrumentation libraries allow sampling of traces, it can be convenient to sample a ratio of the incoming traces directly in groundcover.

The default sampling ratio for traces ingested by the sensor is 5%

groundcover sampling does not take into account sampling being done in earlier stages (e.g SDK or collectors). It's recommended to choose one point for sampling.

The samplingRatio field is a fraction in the range 0-1. For example, 0.1 means 10% of the incoming traces will be sampled and stored in groundcover.

To configure sampling, the relevant values can be used:

agent:
  sensor:
    apmIngestor:
      otel:
        direct:
          samplingRatio: 0.05

Configuring 100% Sampling Ratio (No Sampling)

Use the values below to disable sampling and ingest 100% of the incoming traces.

agent:
  sensor:
    apmIngestor:
      otel:
        direct:
          samplingRatio: 1

Troubleshooting

Common Issues

Connection Refused / Timeout

Verify the sensor endpoint URL
Check application logs for OTLP exporter errors

Data Not Appearing

Check application logs for OTLP exporter errors
Verify the service name and resource attributes are set
Allow 1-2 minutes for data to appear in groundcover UI
For low-volume environments like development clusters, increase sampling ratio to 1 or increase the volume of requests sent to the service

Last updated 9 days ago