Traces & Logs

Learn how to ingest OTEL traces & logs with groundcover

groundcover fully supports the ingestion of traces and logs in the Open Telemetry format, displaying them natively in our UI.

Ingestion From Kubernetes Pods

OTLP Traces and Logs generated from Kubernetes pods can be ingested directly by our DaemonSet Sensor. Ingestion is supported by changing the exporter endpoint to the Sensor Service Endpoint, which will also enrich the received spans and logs with Kubernetes metadata.

Finding the groundcover Sensor Service Endpoint

Use the instructions here to locate the endpoint for the Sensor service, referenced below as {GROUNDCOVER_SENSOR_ENDPOINT}.

Setting Up The Required Environment Variables

Change your deployments environments variables as detailed below. As mentioned above, in most cases it makes sense to only ship traces.

groundcover collects all logs inside your k8s cluster out of the box without the need for any additional setup, so in most cases ingesting OTLP logs from inside your Kubernetes cluster is not recommended as it will cause log duplications. Not sure? Contact us on Slack!

env:
  - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
    value: "http://{GROUNDCOVER_SENSOR_ENDPOINT}:4318/v1/traces"
  - name: OTEL_LOGS_EXPORTER
    value: "none"

env:
  - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
    value: "http://{GROUNDCOVER_SENSOR_ENDPOINT}:4318/v1/traces"
  - name: OTEL_EXPORTER_OTLP_LOGS_ENDPOINT
    value: "http://{GROUNDCOVER_SENSOR_ENDPOINT}:4318/v1/logs"

Automatic Attribute Enrichment

groundcover will automatically enrich traces and logs ingested with Kubernetes metadata, in order to provide as much context as possible.

groundcover follows OpenTelemetry's Semantic Conventions when naming the attributes.

groundcover will replace the service.name attribute indicating the name of the service with the name of the Kubernetes Deployment which is the owner of the pod. Keep this in mind when looking for your traces and logs in the system!

Pod Level attributes:

k8s.namespace.name - the namespace of the pod
k8s.node.name - the node the pod is scheduled on
k8s.pod.name - the name of the pod
k8s.pod.uid - the UID of the pod
k8s.pod.ip - the IP address of the pod at the time of the trace
k8s.cluster.name - the Kubernetes cluster name

Container level attributes:

If the container.id tag is provided with the container ID provided by the Container Runtime, the following tags will also be enriched:

container.name - the name of the container
container.image.name - the name of the container image
container.image.tag - the tag of the container image

Starting from version 1.8.216, groundcover will enrich container level attributes for pods with a single container without the need for providing the container.id tag.

Forwarding from an OpenTelemetry Collector Inside Kubernetes

Starting from version 1.8.216, the recommended method to ship traces & logs from an OpenTelemetry Collector is the same as other deployments - directly to the groundcover sensor endpoint.

Using an earlier version? Upgrade your installation or contact us on Slack!

Finding the groundcover Sensor Service Endpoint

Use the instructions here to locate the endpoint for the Sensor service, referenced below as {GROUNDCOVER_SENSOR_ENDPOINT}.

Defining the OTLP exporter and pipelines

exporters:
  otlp/groundcover:
    endpoint: {GROUNDCOVER_SENSOR_ENDPOINT}:4317

pipelines:
  traces:
    exporters:
    - otlp/groundcover

For logs forwarding, add the following to your logs pipeline. Keep in mind that groundcover collects your logs out of the box, so this is redundant in most cases.

pipelines:
  logs:
    exporters:
    - otlp/groundcover

Shipping Data from Outside of a Kubernetes Cluster

This feature is only supported for inCloud Managed installations as part of our Enterprise offering. See here for more details.

groundcover exposes an OpenTelemetry interface as part of our inCloud Managed endpoints, which can be used to ingest data in all standard OTLP protocols. Use the instructions here to locate the OpenTelemetry endpoint, referenced below as {GROUNDCOVER_OPENTELEMETRY_ENDPOINT}.

These endpoints require authentication using an {apikey} which can be fetched with the groundcover CLI using the following command:

groundcover auth print-api-key

Additional notes

Both gRPCs and HTTPs are supported.
The list of supported authentication methods can be found here.

Example - Forwarding Traces from an OpenTelemetry Collector outside of a cluster

exporters:
  otlp/groundcover:
    endpoint: {GROUNDCOVER_OPENTELEMETRY_ENDPOINT}:4317
    headers: 
      apikey: {apikey}
      
pipelines:
  traces:
    exporters:
    - otlp/groundcover

Sampling Incoming Traces

While some instrumentation libraries allow sampling of traces, it can be convenient to sample a ratio of the incoming traces directly in groundcover.

groundcover sampling does not take into account sampling being done in earlier stages (e.g SDK or collectors). It's recommended to choose one point for sampling.

To configure sampling, the relevant values can be used:

agent:
  sensor:
    apmIngestor:
      otel:
        direct:
          samplingRatio: 0.1

The samplingRatio field is a fraction in the range 0-1. For example, 0.1 means 10% of the incoming traces will be sampled and stored in groundcover.

As of December 1st, 2024, the default sampling rate is 5%.

Configuring 100% Sampling Ratio (No Sampling)

Use the values below to disable sampling and ingest 100% of the incoming traces.

agent:
  sensor:
    apmIngestor:
      otel:
        direct:
          samplingRatio: 1

Last updated 3 days ago

env: - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT value: "http://{GROUNDCOVER_SENSOR_ENDPOINT}:4318/v1/traces" - name: OTEL_EXPORTER_OTLP_LOGS_ENDPOINT value: "http://{GROUNDCOVER_SENSOR_ENDPOINT}:4318/v1/logs"