Traces & Logs
Learn how to ingest OTEL traces & logs with groundcover
Last updated
Learn how to ingest OTEL traces & logs with groundcover
Last updated
groundcover fully supports the ingestion of traces and logs in the Open Telemetry format, displaying them natively in our UI.
OTLP Traces and Logs generated from Kubernetes pods can be ingested directly by our DaemonSet Sensor
. Ingestion is supported by changing the exporter endpoint to the Sensor
Service Endpoint, which will also enrich the received spans and logs with Kubernetes metadata.
Use the instructions here to locate the endpoint for the Sensor service, referenced below as {GROUNDCOVER_SENSOR_ENDPOINT}
.
Ingestion is supported for both OTLP/HTTP and OTLP/gRPC
Apply the environment variables below to your services in order to make them ship data to groundcover's ingestion endpoint.
env:
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: "http://{GROUNDCOVER_SENSOR_ENDPOINT}:4318"
- name: OTEL_EXPORTER_OTLP_PROTOCOL
value: "http/protobuf"
groundcover will automatically enrich traces and logs ingested with Kubernetes metadata, in order to provide as much context as possible.
groundcover follows OpenTelemetry's Semantic Conventions when naming the attributes.
groundcover will replace the service.name
attribute indicating the name of the service with the name of the Kubernetes Deployment which is the owner of the pod. Keep this in mind when looking for your traces and logs in the system!
Pod Level attributes:
k8s.namespace.name
- the namespace of the pod
k8s.node.name
- the node the pod is scheduled on
k8s.pod.name
- the name of the pod
k8s.pod.uid
- the UID of the pod
k8s.pod.ip
- the IP address of the pod at the time of the trace
k8s.cluster.name
- the Kubernetes cluster name
Container level attributes:
If the container.id
tag is provided with the container ID provided by the Container Runtime, the following tags will also be enriched:
container.name
- the name of the container
container.image.name
- the name of the container image
container.image.tag
- the tag of the container image
Use the instructions here to locate the endpoint for the Sensor service, referenced below as {GROUNDCOVER_SENSOR_ENDPOINT}
.
Ingestion is supported for both OTLP/HTTP and OTLP/gRPC
exporters:
otlphttp/groundcover:
endpoint: http://{GROUNDCOVER_SENSOR_ENDPOINT}:4318
pipelines:
traces:
exporters:
- otlphttp/groundcover
logs:
exporters:
- otlphttp/groundcover
groundcover exposes an OpenTelemetry interface as part of our inCloud Managed endpoints, which can be used to ingest data in all standard OTLP protocols for workloads which are not running alongside sensors.
Use the instructions here to locate the endpoint, referenced below as {GROUNDCOVER_MANAGED_OPENTELEMETRY_ENDPOINT}.
These endpoints require authentication using an {INGESTION_KEY}
of type 3rd Party
(Settings -> Access -> Ingestion Keys)
Both gRPCs and HTTPs are supported.
The list of supported authentication methods can be found here.
Ingestion is supported for both OTLP/HTTP and OTLP/gRPC
exporters:
otlphttp/groundcover:
endpoint: http://{GROUNDCOVER_MANAGED_OPENTELEMETRY_ENDPOINT}:443
headers:
apikey: {INGESTION_KEY}
pipelines:
traces:
exporters:
- otlphttp/groundcover
logs:
exporters:
- otlphttp/groundcover
While some instrumentation libraries allow sampling of traces, it can be convenient to sample a ratio of the incoming traces directly in groundcover. groundcover employs sampling mechanisms on traces by default.
groundcover sampling does not take into account sampling being done in earlier stages (e.g SDK or collectors). It's recommended to choose one point for sampling.
To configure sampling, the relevant values can be used:
agent:
sensor:
apmIngestor:
otel:
direct:
samplingRatio: 0.05
The samplingRatio
field is a fraction in the range 0-1. For example, 0.1 means 10% of the incoming traces will be sampled and stored in groundcover.
Use the values below to disable sampling and ingest 100% of the incoming traces.
agent:
sensor:
apmIngestor:
otel:
direct:
samplingRatio: 1