LogoLogo
Log in|Playground
  • Welcome
    • Introduction
    • FAQ
  • Capabilities
    • Log Management
    • Infrastructure Monitoring
    • Application Performance Monitoring (APM)
      • Application Metrics
      • Traces
      • Supported Technologies
    • Real User Monitoring (RUM)
  • Getting Started
    • Requirements
      • Kubernetes requirements
      • Kernel requirements for eBPF sensor
      • CPU architectures
      • ClickHouse resources
    • Installation & updating
    • Connect Linux hosts
    • Connect RUM
    • 5 quick steps to get you started
    • groundcover MCP
      • Configure groundcover's MCP Server
      • Getting-started Prompts
      • Real-world Use Cases
  • Use groundcover
    • Monitors
      • Create a new Monitor
      • Issues page
      • Monitor List page
      • Silences page
      • Monitor Catalog page
      • Monitor YAML structure
      • Embedded Grafana Alerts
        • Create a Grafana alert
    • Dashboards
      • Create a dashboard
      • Embedded Grafana Dashboards
        • Create a Grafana dashboard
        • Build alerts & dashboards with Grafana Terraform provider
        • Using groundcover datasources in a Self-hosted Grafana
    • Insights
    • Explore & Monitors query builder
    • Workflows
      • Create a new Workflow
      • Workflow Examples
      • Alert Structure
    • Search & Filter
    • Issues
    • Role-Based Access Control (RBAC)
    • Service Accounts
    • API Keys
    • APIs
    • Log Patterns
    • Drilldown
    • Scraping custom metrics
      • Operator based metrics
      • kube-state-metrics
      • cadvisor metrics
    • Backup & Restore Metrics
    • Metrics & Labels
    • Add custom environment labels
    • Configuring Pipelines
      • Writing Remap Transforms
      • Logs Pipeline Examples
      • Traces Pipeline Examples
      • Logs to Events Pipeline Examples
      • Logs/Traces Sensitive Data Obfuscation
      • Sensitive Data Obfuscation using OTTL
      • Log Filtering using OTTL
    • Querying your groundcover data
      • Query your logs
        • Example queries
        • Logs alerting
      • Query your metrics
      • Querying you data using an API
      • Using KEDA autoscaler with groundcover
  • Log Parsing with OpenTelemetry Pipelines
  • Log and Trace Correlation
  • RUM
  • Customization
    • Customize deployment
      • Agents in host network mode
      • API Key Secret
      • Argo CD
      • On-premise deployment
      • Quay.io registry
      • Configuring sensor deployment coverage
      • Enabling SSL Tracing in Java Applications
    • Customize usage
      • Filtering Kubernetes entities
      • Custom data retention
      • Sensitive data obfuscation
      • Custom storage
      • Custom logs collection
      • Custom labels and annotations
        • Enrich logs and traces with pod labels & annotations
        • Enrich metrics with node labels
      • Disable tracing for specific protocols
      • Tuning resources
      • Controlling the eBPF sampling mechanism
  • Integrations
    • Overview
    • Workflow Integrations
      • Slack Webhook Integration
      • Opsgenie Integration
      • Webhook Integration
        • Incident.io
      • PagerDuty Integration
      • Jira Webhook Integration
      • Send groundcover Alerts to Email via Zapier
    • Data sources
      • OpenTelemetry
        • Traces & Logs
        • Metrics
      • Istio
      • AWS
        • Ingest CloudWatch Metrics
        • Ingest CloudWatch Logs
        • Ingest Logs Stored on S3
        • Integrate CloudWatch Grafana Datasource
      • GCP
        • Ingest Google Cloud Monitoring Metrics
        • Stream Logs using Pub/Sub
        • Integrate Google Cloud Monitoring Grafana Datasource
      • Azure
        • Ingest Azure Monitor Metrics
      • DataDog
        • Traces
        • Metrics
      • FluentBit
      • Fluentd
      • JSON Logs
    • 3rd-party metrics
      • ActiveMQ
      • Aerospike
      • Cassandra
      • CloudFlare
      • Consul
      • CoreDNS
      • Etcd
      • HAProxy
      • Harbor
      • JMeter
      • K6
      • Loki
      • Nginx
      • Pi-hole
      • Postfix
      • RabbitMQ
      • Redpanda
      • SNMP
      • Solr
      • Tomcat
      • Traefik
      • Varnish
      • Vertica
      • Zabbix
    • Source control (Gitlab/Github)
  • Architecture
    • Overview
    • inCloud Managed
      • Setup inCloud Managed with AWS
        • AWS PrivateLink Setup
        • EKS add-on
      • Setup inCloud Managed with GCP
      • Setup inCloud Managed with Azure
      • High Availability
      • Disaster Recovery
      • Ingestion Endpoints
      • Deploying in Sensor-Only mode
    • Security considerations
      • Okta SSO - onboarding
    • Service endpoints inside the cluster
  • Product Updates
    • What's new?
    • Earlier updates
      • 2025
        • Mar 2025
        • Feb 2025
        • Jan 2025
      • 2024
        • Dec 2024
        • Nov 2024
        • Oct 2024
        • Sep 2024
        • Aug 2024
        • July 2024
        • May 2024
        • Apr 2024
        • Mar 2024
        • Feb 2024
        • Jan 2024
      • 2023
        • Dec 2023
        • Nov 2023
        • Oct 2023
Powered by GitBook
On this page
  • Ingestion from Kubernetes Pods
  • Shipping from an OpenTelemetry Collector
  • Shipping Data to the sensors
  • Shipping Data directly to inCloud Managed endpoint
  • Sampling
Export as PDF
  1. Integrations
  2. Data sources
  3. OpenTelemetry

Traces & Logs

Learn how to ingest OTEL traces & logs with groundcover

Last updated 9 days ago

groundcover fully supports the ingestion of traces and logs in the Open Telemetry format, displaying them natively in our UI.

Ingestion from Kubernetes Pods

OTLP Traces and Logs generated from Kubernetes pods can be ingested directly by our DaemonSet Sensor. Ingestion is supported by changing the exporter endpoint to the Sensor Service Endpoint, which will also enrich the received spans and logs with Kubernetes metadata.

Finding the groundcover Sensor Service Endpoint

Setting Up The Required Environment Variables

Ingestion is supported for both OTLP/HTTP and OTLP/gRPC

Apply the environment variables below to your services in order to make them ship data to groundcover's ingestion endpoint.

env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://{GROUNDCOVER_SENSOR_ENDPOINT}:4318"
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: "http/protobuf"
env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "{GROUNDCOVER_SENSOR_ENDPOINT}:4317"

Automatic Attribute Enrichment

groundcover will automatically enrich traces and logs ingested with Kubernetes metadata, in order to provide as much context as possible.

groundcover will replace the service.name attribute indicating the name of the service with the name of the Kubernetes Deployment which is the owner of the pod. Keep this in mind when looking for your traces and logs in the system!

Pod Level attributes:

  • k8s.namespace.name - the namespace of the pod

  • k8s.node.name - the node the pod is scheduled on

  • k8s.pod.name - the name of the pod

  • k8s.pod.uid - the UID of the pod

  • k8s.pod.ip - the IP address of the pod at the time of the trace

  • k8s.cluster.name - the Kubernetes cluster name

Container level attributes:

If the container.id tag is provided with the container ID provided by the Container Runtime, the following tags will also be enriched:

  • container.name - the name of the container

  • container.image.name - the name of the container image

  • container.image.tag - the tag of the container image

Starting from version 1.8.216, groundcover will enrich container level attributes for pods with a single container without the need for providing the container.id tag.

Shipping from an OpenTelemetry Collector

Shipping Data to the sensors

Starting from version 1.8.216, the recommended method to ship traces & logs from an OpenTelemetry Collector is the same as other deployments - directly to the groundcover sensor endpoint.

Finding the groundcover Sensor Service Endpoint

Defining the OTLP exporter and pipelines

Ingestion is supported for both OTLP/HTTP and OTLP/gRPC

exporters:
  otlphttp/groundcover:
    endpoint: http://{GROUNDCOVER_SENSOR_ENDPOINT}:4318

pipelines:
  traces:
    exporters:
    - otlphttp/groundcover
  logs:
    exporters:
    - otlphttp/groundcover
exporters:
  otlp/groundcover:
    endpoint: {GROUNDCOVER_SENSOR_ENDPOINT}:4317

pipelines:
  traces:
    exporters:
    - otlp/groundcover
  logs:
    exporters:
    - otlp/groundcover

Shipping Data directly to inCloud Managed endpoint

groundcover exposes an OpenTelemetry interface as part of our inCloud Managed endpoints, which can be used to ingest data in all standard OTLP protocols for workloads which are not running alongside sensors.

These endpoints require authentication using an {apikey} which can be fetched with the groundcover CLI using the following command:

groundcover auth print-api-key

Additional notes

  1. Both gRPCs and HTTPs are supported.

Ingestion is supported for both OTLP/HTTP and OTLP/gRPC

exporters:
  otlphttp/groundcover:
    endpoint: http://{GROUNDCOVER_MANAGED_OPENTELEMETRY_ENDPOINT}:443
    headers: 
      apikey: {apikey}

pipelines:
  traces:
    exporters:
    - otlphttp/groundcover
  logs:
    exporters:
    - otlphttp/groundcover
exporters:
  otlp/groundcover:
    endpoint: {GROUNDCOVER_MANAGED_OPENTELEMETRY_ENDPOINT}:443
    headers: 
      apikey: {apikey}

pipelines:
  traces:
    exporters:
    - otlp/groundcover
  logs:
    exporters:
    - otlp/groundcover

Sampling

As of December 1st, 2024, the default sampling rate is 5%. See below on how to control this value.

While some instrumentation libraries allow sampling of traces, it can be convenient to sample a ratio of the incoming traces directly in groundcover. groundcover employs sampling mechanisms on traces by default.

groundcover sampling does not take into account sampling being done in earlier stages (e.g SDK or collectors). It's recommended to choose one point for sampling.

To configure sampling, the relevant values can be used:

agent:
  sensor:
    apmIngestor:
      otel:
        direct:
          samplingRatio: 0.05

The samplingRatio field is a fraction in the range 0-1. For example, 0.1 means 10% of the incoming traces will be sampled and stored in groundcover.

Configuring 100% Sampling Ratio (No Sampling)

Use the values below to disable sampling and ingest 100% of the incoming traces.

agent:
  sensor:
    apmIngestor:
      otel:
        direct:
          samplingRatio: 1

Use the instructions to locate the endpoint for the Sensor service, referenced below as {GROUNDCOVER_SENSOR_ENDPOINT}.

groundcover follows when naming the attributes.

Using an earlier version? Upgrade your installation or

Use the instructions to locate the endpoint for the Sensor service, referenced below as {GROUNDCOVER_SENSOR_ENDPOINT}.

This feature is only supported for inCloud Managed installations as part of our Enterprise offering. See for more details.

Use the instructions to locate the endpoint, referenced below as {GROUNDCOVER_MANAGED_OPENTELEMETRY_ENDPOINT}.

The list of supported authentication methods can be found .

OpenTelemetry's Semantic Conventions
contact us on Slack!
here
here
here
here
here