OpenTelemetry

Learn how to collect OTel data with groundcover

groundcover fully supports the ingestion of traces and logs in the Open Telemetry format, displaying them natively in our UI.

Traces collected via OpenTelemetry will have a matching source property

Ingesting OpenTelemetry Data From Kubernetes Pods

Traces and Logs in the OTLP format can be ingested directly by our DaemonSet Alligator. Ingestion is supported by changing the exporter endpoint to the Alligator Service Endpoint, which will also enrich the received spans and logs with Kubernetes metadata (see below).

groundcover collects all logs inside your k8s cluster out of the box without the need for any additional setup, so in most cases ingesting OTLP logs from inside your Kubernetes cluster is not recommended as it will cause log duplications. Not sure? Contact us on Slack!

Finding the groundcover Alligator Service Endpoint

Use the instructions here to locate the endpoint for the Alligator service, referenced below as {GROUNDCOVER_ALLIGATOR_ENDPOINT}.

Setting the OpenTelemetry Exporter Endpoint

Change the OTEL_EXPORTER_OTLP_ENDPOINT environment variable in your service to the endpoint found above. This will redirect all traces and logs generated by the service to groundcover.

env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://{GROUNDCOVER_ALLIGATOR_ENDPOINT}:4317"

See the OTLP Exporter Configuration page for more customization options, including selective signal exporting (e.g only traces or only logs), using HTTP instead of gRPC, etc.

Automatic Attribute Enrichment

groundcover will automatically enrich traces and logs ingested with Kubernetes metadata, in order to provide as much context as possible.

groundcover follows OpenTelemetry's Semantic Conventions when naming the attributes.

Pod Level attributes:

• k8s.namespace.name - the namespace of the pod

• k8s.node.name - the node the pod is scheduled on

• k8s.pod.name - the name of the pod

• k8s.pod.uid - the UID of the pod

• k8s.pod.ip - the IP address of the pod at the time of the trace

• k8s.cluster.name - the Kubernetes cluster name

Container level attributes:

If the container.id tag is provided with the container ID provided by the Container Runtime, the following tags will also be enriched:

• container.name - the name of the container

• container.image.name - the name of the container image

• container.image.tag - the tag of the container image

groundcover will replace the service.name attribute indicating the name of the service with the name of the Kubernetes Deployment which is the owner of the pod. Keep this in mind when looking for your traces and logs in the system!

Forwarding traces from an existing OpenTelemetry collector

groundcover ships with an OpenTelemetry collector out of the box. If your services are configured to use an already existing OpenTelemetry collector, you can forward the traces it collects to groundcover's collector using the OTLP exporter, specifying our collector as the endpoint.

Finding the groundcover OpenTelemetry Collector Endpoint

Use the instructions here to locate the endpoint for the Alligator service, referenced below as {GROUNDCOVER_OPENTELEMETRY_COLLECTOR_ENDPOINT}.

For default installations, groundcover's OpenTelemetry Collector communicates with TLS turned off, which is why TLS is disabled in the below configuration. For more complex environments with ingestion of sources from outside the cluster, contact us on Slack!

Defining the OTLP exporter and pipelines

Start by adding the needed exporter:

exporters:
  otlp/groundcover:
    endpoint: {GROUNDCOVER_OPENTELEMETRY_COLLECTOR_ENDPOINT}:4317
    tls:
      insecure: true

For traces forwarding, add the following to your traces pipeline:

pipelines:
  traces:
    exporters:
    - otlp/groundcover

For logs forwarding, add the following to your logs pipeline. Keep in mind that groundcover collects your logs out of the box, so this is redundant in most setups.

pipelines:
  logs:
    exporters:
    - otlp/groundcover