Traces
Our traces philosophy
Traces are a powerful observability pillar, providing granular insights into microservice interactions. Traditionally, they were hard to implement, requiring coordination of multiple teams and constant code changes, making this critical aspect very challenging to maintain.
groundcover's eBPF sensor disrupts the famous tradeoff, empowering developers to gain full visibility into their applications, effortlessly and without any code changes.
The platform supports two kinds of traces:
eBPF traces
These traces are automatically generated for every supported service in your stack. They are available out-of-the-box and within seconds of installation. These traces always include critical information such as:
All services that took part in the interaction (both client and server)
Accessed resource
Full payloads, including:
All headers
All query parameters
All bodies - for both the request and response
3rd-party traces
These can be ingested into the platform, allowing to leverage already existing instrumentation to create a single pane of glass for all of your traces.
Traces are stored in groundcover's ClickHouse deployment, ensuring top notch performance on every scale.
Sampling
groundcover further disrupts the customary traces experience by reinventing the concept of sampling. This innovation differs between the different types of traces:
eBPF traces
These are generated by using 100% of the data, always processing every request being made, on every scale. However, the groundcover platform utilizes smart sampling to only store a fraction of the traces, while still generating an accurate picture. In general, sampling is performed according to these rules:
Requests with unusually high or low latencies, measured per resource
Requests which returned an error response (e.g 500 status code for HTTP)
"Normal" requests which form the baseline for each resource
Lastly, stream processing is utilized to make the sampling decisions on the node itself, without having to send or save any redundant traces.
3rd-party traces
Various mechanisms control the sampling performed over 3rd party traces. Read more here:
When integrating 3rd-party traces, it is often wise to configure some sampling mechanism according to the specific use case.
Additional Context
Each trace is enriched with additional information to give as much context as possible for the service which generated the trace. This includes:
Container information - image, environment variables, pod name
Logs generated by the service around the time of the trace
Golden Signals of the resource around the time of the trace
Kubernetes events relevant to the service
CPU and Memory utilization of the service and the node it is scheduled on
Distributed Tracing
One of the advantages of ingesting 3rd-party traces is the ability to leverage their distributed tracing feature. groundcover natively displays the full trace for ingested traces in the Traces page.
Trace Attributes
Trace Attributes enable advanced filtering and search capabilities. groundcover support attributes across all trace types. This encompasses a diverse range of protocols such as HTTP, MongoDB, PostgreSQL, and others, as well as varied sources including eBPF or manual instrumentations (for example - OpenTelemetry).
groundcover enriches your original traces and generates meaningful metadata as key-value pairs. This metadata includes critical information, such as protocol type, http.path, db.statement, and similar attributes, aligning with OTel conventions. Furthermore, groundcover seamlessly incorporates this metadata from spans received through supported manual instrumentations. For an in-depth understanding of attributes in OTel, please refer to OTel Attributes Documentation (external link to OpenTelemtry website).
Each attribute can be effortlessly integrated into your filters and search queries. You can add them directly from the trace side-panel with a simple click or input them manually into the search bar.
Example: If you want to filter all HTTP traces that contain the path "/products". The query would be formatted as: @http.path:"/products". For a comprehensive guide on the query syntax, see Syntax table below.
Trace Tags
Trace Tags enable advanced filtering and search capabilities. groundcover support tags across all trace types. This encompasses a diverse range of protocols such as HTTP, MongoDB, PostgreSQL, and others, as well as varied sources including eBPF or manual instrumentations (for example - OpenTelemetry).
Tags are powerful metadata components, structured as key-value pairs. They offer insightful information about the resource generating the span, like: container.image.name ,host.name and more.
Tags include metadata enriched by the our sensor and additional metadata if provided by manual instrumentations (such as OpenTelemetry traces) . Utilizing these Tags enhances understanding and context of your traces, allowing for more comprehensive analysis and easier filtering by the relevant information.
Each tag can be effortlessly integrated into your filters and search queries. You can add them directly from the trace side-panel with a simple click or input them manually into the search bar.
Example: If you want to filter all traces from mysql containers - The query would be formatted as: container.image.name:mysql. For a comprehensive guide on the query syntax, see Syntax table below.
Request and Response Body Search
groundcover enables searching within trace payloads using the request_body and response_body fields. This powerful capability allows you to find traces based on the actual content of HTTP requests and responses, making it easier to debug issues and investigate specific interactions.
Supported Protocols
Body search works across all protocols that capture payload data, including HTTP, gRPC, and other supported protocols.
Search Syntax
The body fields support the same wildcard semantics as other filter fields:
request_body:value
Exact match in request payload
request_body:error
request_body:*value*
Substring match (contains)
request_body:*user_id*
request_body:value*
Prefix match (starts with)
request_body:success*
request_body:*value
Suffix match (ends with)
request_body:*completed
response_body:value
Exact match in response payload
response_body:success
-request_body:*error*
Exclude traces with pattern
-request_body:*error*
Examples
Find traces where the request body contains a specific user ID:
Find error responses containing a specific message:
Find all traces where the body contains the text "password":
Combine body search with other filters:
If payload obfuscation changes the body, searches match the transformed content. If privacy controls clear or remove the body, that payload is not searchable.
Search and filter
The Traces Explorer integrates dynamic filters and a versatile search functionality to enhance your trace data analysis. You can filter out traces using specific criteria, including trace-status, workload, namespace and more, as well as limit your search to a specific time range.
Learn more about how to use our search syntaxes
Traces Pipelines
groundcover natively supports setting up log pipelines using Vector transforms. This allow for full flexibility in the processing and manipulation of traces being collected - parsing additional patterns by regex, renaming attributes, and many more.
Learn more about how to configure traces pipelines
Controlling retention
groundcover allows full control over the retention of your traces. Read here to learn more.
Custom Configuration
Tracing can be customized in several ways:
Last updated