Filter Traces

Filter Traces

Overview

Filter out noisy, irrelevant, or low-value spans before they're stored. By dropping spans at the pipeline level, you can significantly reduce storage costs while focusing on the traces that matter most.

Why Filter Traces?

Not all spans provide equal value. Filtering traces allows you to:

• Reduce storage costs by removing noise before ingestion

• Focus on critical traffic without distractions from health checks and internal probes

• Improve query performance by reducing the dataset size

• Eliminate redundant data such as duplicate internal spans

Spans are dropped in the pipeline before they're sent to storage, achieving maximum savings in terms of networking and ingestion.

Best Practices

1. Be specific with conditions - Use precise conditions to avoid accidentally dropping important spans

2. Start conservative - Begin by dropping obvious noise, then expand gradually

3. Monitor the impact - Track how many spans are being dropped and adjust as needed

4. Document your rules - Use clear ruleName values to explain what each rule does

5. Avoid dropping errors - Unless absolutely necessary, preserve error spans for debugging

6. Consider performance - Dropping spans early saves processing and storage costs

How Span Filtering Works

To drop a span, you define a condition (e.g., span name, workload, protocol type) and use a set(drop, true) statement. If the condition matches, the span is removed from the pipeline before storage.

Key Concepts:

• drop - A reserved field. If set to true, the span is filtered out

• where - Used to specify when the dropping should apply

• IsMatch() - A function for matching regex or substrings within fields

Common Use Cases

Drop Health Check Spans

Health check endpoints often generate high-volume, low-value spans.

ottlRules:
  - ruleName: drop_health_checks
    conditions:
      - protocol_type == "http"
    statements:
      - set(drop, true) where IsMatch(span_name, "/health")
      - set(drop, true) where IsMatch(span_name, "/healthz")
      - set(drop, true) where IsMatch(span_name, "/readyz")
      - set(drop, true) where IsMatch(span_name, "/livez")

💡 What it does: Drops all HTTP spans for common health check, readiness, and liveness probe endpoints.

Drop Spans by Protocol Type

Filter out spans for specific protocols you don't need to monitor.

ottlRules:
  - ruleName: drop_dns_traces
    conditions:
      - protocol_type == "dns"
    statements:
      - set(drop, true)

💡 What it does: Drops all DNS resolution spans to focus on application-level traces.

Drop Spans Based on Attributes

Filter based on custom span attributes.

ottlRules:
  - ruleName: drop_synthetic_traffic
    conditions:
      - attributes["synthetic"] == "true"
    statements:
      - set(drop, true)

💡 What it does: Drops spans flagged as synthetic/test traffic.

Conditional Dropping with Complex Logic

Combine multiple conditions for fine-grained control.

ottlRules:
  - ruleName: "drop_successful_health_checks"
    conditions:
      - protocol_type == "http"
      - status == "Ok"
    conditionLogicOperator: "and"
    statements:
      - set(drop, true) where IsMatch(span_name, "/health") or IsMatch(span_name, "/ready")

💡 What it does: Drops only successful health check spans. Failed health checks are preserved for debugging.