Traces

Our traces philosophy

Traces are a powerful observability pillar, providing granular insights into microservice interactions. Traditionally, they were hard to implement, requiring coordination of multiple teams and constant code changes, making this critical aspect very challenging to maintain.

groundcover's eBPF sensor disrupts the famous tradeoff, empowering developers to gain full visibility into their applications, effortlessly and without any code changes.

The platform supports two kinds of traces:

eBPF traces are automatically generated for every supported service in your stack. They are available out-of-the-box and within seconds of installation. These traces always include critical information such as:

• All services that took part in the interaction (both client and server)

• Accessed resource

• Full payloads, including:

  • All headers

  • All query parameters

  • All bodies - for both the request and response

3rd-party traces can be ingested into the platform, allowing to leverage already existing instrumentation to create a single pane of glass for all of your traces.

Traces are stored in groundcover's ClickHouse deployment, ensuring top notch performance on every scale.

For more details about ingesting 3rd party traces, see the integrations page.

Sampling

groundcover further disrupts the customary traces experience by reinventing the concept of sampling. This innovation differs between the different types of traces:

eBPF traces are generated by using 100% of the data, always processing every request being made, on every scale. However, the groundcover platform utilizes smart sampling to only store a fraction of the traces, while still generating an accurate picture. In general, sampling is performed according to these rules:

• Requests with unusually high or low latencies, measured per resource

• Requests which returned an error response (e.g 500 status code for HTTP)

• "Normal" requests which form the baseline for each resource

Lastly, stream processing is utilized to make the sampling decisions on the node itself, without having to send or save any redundant traces.

Certain aspects of our sampling algorithm are configurable - read more here.

3rd-party traces are always ingested fully - meaning, no sampling is applied to traces which have been generated elsewhere and ingested by the platform.

When integrating 3rd-party traces, it is often wise to configure some sampling mechanism according to the specific use case.

Additional Context

Each trace is enriched with additional information to give as much context as possible for the service which generated the trace. This includes:

• Container information - image, environment variables, pod name

• Logs generated by the service around the time of the trace

• Golden Signals of the resource around the time of the trace

• Kubernetes events relevant to the service

• CPU and Memory utilization of the service and the node it is scheduled on

Distributed Tracing

One of the advantages of ingesting 3rd-party traces is the ability to leverage their distributed tracing feature. When viewing a trace generated by a supported library, clicking the Full Trace button will open a view of the end-to-end associated trace in a new tab.

eBPF traces do not currently support distributed tracing. This is very much on our roadmap - stay tuned!

Controlling retention

groundcover allows full control over the retention of your traces. Read here to learn more.

Custom Configuration

Tracing can be customized in several ways:

• Configuring which protocols should be traced

• Configuring obfuscation for sensitive payload data

• Configuring the sampling mechanism