Traces Pipeline

Overview

Traces pipeline is powered by OpenTelemetry Transformation Language (OTTL), a powerful and flexible language designed for transforming telemetry data. OTTL provides a rich set of functions and operators that allow you to filter, enrich, and transform your traces with precision.

Here are some common use cases:

• Transform attributes - Rename or reshape span attributes so they match your naming or schema.

• Filter traces - Drop spans you don’t need (e.g. health checks) to reduce storage and noise.

• Enrich data - Parse or derive new attributes from existing fields for better filtering in the UI.

• Obfuscation - Remove sensitive information from traces.

Setting up pipelines

Traces pipelines can only be edited by Admins

Go to Settings > Data Pipelines > Traces Pipeline to see your existing pipelines.

To edit your rules, click on Edit Pipeline at the top right.

Writing Pipeline Rules

Configuration rule format

The below snippet is the format of a rule

ottlRules:
  - ruleName: <rule name>
    ruleDisabled: false
    statements:
      - <list of statements>
    conditions:
      - <list of conditions>
    statementsErrorMode: ignore
    conditionLogicOperator: and

• ruleName – Use a meaningful name for a rule. This name will appear in the main screen and will allow you to track its performance.

• ruleDisabled - use this parameter to temporary disable rules

• statements - use 1 or more functions to describe the transformation to do. For example, use replace_all_matches to locate a specific value and replace it with another.

• conditions - use this optional parameter to set conditions on when the statements will run. Use the OTTL Comparison Rules to define the list of conditions.

Common Functions

Some commonly used functions in groundcover traces pipelines:

• set – Set attribute values

• delete_key – Remove specific attributes

• merge_maps – Merge extracted data into attributes

• replace_pattern – Replace text patterns (useful for obfuscation)

• ParseJSON – Parse JSON strings into structured attributes

• IsMatch – Match patterns in span fields

• Concat – Concatenate multiple values

• Substring – Extract portion of a string

• obfuscate_pii – Automatically detect and redact sensitive data

For a complete list of available OTTL functions, see the OTTL Functions Reference.

drop - Span Filtering

The drop field is a special boolean flag for filtering spans:

statements:
  # Drop unconditionally
  - 'set(drop, true)'

  # Drop conditionally
  - 'set(drop, true) where IsMatch(span_name, "/health")'

Setting Conditions

Use conditions to apply transformations only when specific criteria match. This ensures your pipeline runs efficiently and only on relevant spans.

Common fields you can use (Full List Mentioned Above):

• workload – Name of the service or app

• namespace – Kubernetes namespace

• protocol_type – Protocol type (e.g. http, grpc)

• source – Data source (ebpf, otel)

• span_name – Name of the span operation

• kind – Span kind (Client, Server, etc.)

• status – Span status code

Available Fields and Keys

When writing pipeline rules, you have access to various fields that represent different aspects of the span. Understanding which fields are writable and which are read-only is crucial for effective rule creation.

Writable Fields

These fields can be modified in your pipeline rules:

Field	Type	Description
cache	Map	Temporary storage for intermediate values. Access sub-keys with cache["key"]
attributes	Map	Custom span attributes. Access sub-keys with attributes["key"]
drop	Boolean	Set to true to drop/filter the span
span_name	String	The operation name of the span
trace_id	String	Trace identifier (32 hex characters)
span_id	String	Span identifier (16 hex characters)
parent_id	String	Parent span identifier (16 hex characters)
start_time	Time	Start timestamp of the span
end_time	Time	End timestamp of the span
request_body	String	HTTP request payload content
response_body	String	HTTP response payload content
request_headers	Map	HTTP request headers. Access sub-keys with request_headers["header-name"]
response_headers	Map	HTTP response headers. Access sub-keys with response_headers["header-name"]
is_pii	Boolean	Mark span as containing PII. Automatically set by obfuscate_pii

Read-Only Fields

These fields provide context but cannot be modified. Attempting to set them will result in an error:

Field	Type	Description
workload	String	Name of the Kubernetes workload
source	String	Source of the trace data
cluster	String	Kubernetes cluster name
env	String	Environment name
namespace	String	Kubernetes namespace
protocol_type	String	Protocol type (e.g. http, grpc, redis)
kind	String	Span kind (Client, Server, Internal, etc.)
status	String	Span status code (Ok, Error, Unset)
duration	Duration	Span duration (computed from start_time and end_time)
tags	Map	Span tags (read-only). Access sub-keys with tags["key"]

Special Fields

cache - Temporary Storage

The cache field is useful for multi-step transformations:

statements:
  # Extract to cache first
  - 'merge_maps(cache, ParseJSON(response_body), "insert") where response_body != nil'
  # Process cached values
  - 'set(attributes["user_id"], cache["id"]) where cache["id"] != nil'

attributes - Custom Span Attributes

The attributes map is where you store extracted or enriched structured data:

statements:
  - 'set(attributes["team"], "payments")'
  - 'set(attributes["priority"], "high")'

request_body / response_body - Payload Access

Access HTTP payloads for obfuscation or inspection:

statements:
  - 'replace_pattern(request_body, "password=([^&]+)", "password=[REDACTED]")'
  - 'set(response_body, "<REDACTED>") where is_pii == true'

request_headers / response_headers - HTTP Headers

Access and modify HTTP headers:

statements:
  - 'set(response_headers["set-cookie"], "[REDACTED]")'
  - 'set(attributes["user_agent"], request_headers["user-agent"])'

Troubleshooting Rule Issues

High error rates:

• Review the rule syntax for typos

• Check that field names exist on your spans

• Verify regex patterns are valid

High processing latency:

• Simplify complex regex patterns

• Split multi step rules into separate rules

• Place drop rules earlier to reduce processing volume

Low condition met percentage:

• Verify your conditions match actual span attributes

• Check if upstream rules are dropping spans unexpectedly

• Review filter conditions for typos

Unexpected drop rates:

• Review the rule logic and conditions

• Check if drop statements have unintended where clauses

• Verify the rule order isn't causing issues