> For the complete documentation index, see [llms.txt](https://docs.groundcover.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.groundcover.com/use-groundcover/data-pipelines/traces-pipeline/span-to-metrics.md).

# Span to Metrics

## Spans to Metrics

#### Overview

Transform your trace data into queryable metrics for long-term monitoring, alerting, and cost-effective analysis. Spans-to-metrics extracts numerical data from span attributes and converts them into time-series metrics that can be visualized, alerted on, and retained at a fraction of the cost.

{% hint style="warning" %}
**Sampling matters.** For **eBPF-captured traces**, the pipeline runs **after sampling.** metrics generated from eBPF spans reflect only sampled data, not total traffic. For **ingested traces**, the pipeline runs **before sampling -** metrics reflect the full, unsampled dataset. Keep this distinction in mind when interpreting counts and sums.
{% endhint %}

#### Why Use Spans to Metrics?

Traces are perfect for debugging specific requests, but they become expensive and unwieldy at scale. Metrics, on the other hand, are:

* **Cost-effective** - Store aggregated data instead of every span
* **Fast to query** - Optimized for time-series analysis
* **Perfect for alerting** - Track trends and thresholds over time

**The Transformation**

Think of it like converting request traces into spreadsheet rows:

**A Span (request):**

```
POST /api/orders — duration: 120ms, status: Ok, workload: order-service
```

**Becomes Metrics (structured data):**

| Timestamp | Metric Name                 | Value | Labels                                         |
| --------- | --------------------------- | ----- | ---------------------------------------------- |
| `[now]`   | `order_requests_total`      | `1`   | `workload:order-service, endpoint:/api/orders` |
| `[now]`   | `order_request_duration_ms` | `120` | `workload:order-service, endpoint:/api/orders` |

You're essentially turning trace data into countable, measurable data points.

#### When to Use Spans to Metrics

Spans to metrics doesn't replace traces, it complements them. Use it for these scenarios:

**1. Monitoring Latency Distribution**

Track response time trends to understand system performance.

**Use cases:**

* Average, minimum, and maximum response times per endpoint
* Service latency degradation detection
* Comparing latency across workloads or protocols

**Example:**

Create a metric `request_duration_ns` with sum, min, max, and count, then calculate average latency:

```
rate(request_duration_ns_sum) / rate(request_duration_ns_count)
```

**2. Tracking Error Ratios**

Derive error rates by comparing error span counts to total span counts.

**Use cases:**

* Error ratio per service or endpoint
* Detecting degradation trends over time
* Protocol-level error comparison (HTTP vs gRPC)

**Example:**

Create separate count metrics for all spans and error spans, then compute the ratio:

```
rate(span_errors_total) / rate(span_requests_total)
```

**3. Enrichment-Based Metrics**

Generate metrics from data you've extracted or enriched in earlier pipeline rules — such as parsed response fields, custom attributes, or header values.

**Use cases:**

* Metrics derived from JSON response body fields (e.g. `cache["order_total"]`)
* Counting spans by custom attributes set in transform rules
* Aggregating values extracted from request/response headers

**Example:**

After a transform rule parses `order_total` from the response body into `cache`, create a sum metric:

```yaml
- 'span_to_metric_sum("order_total_amount", s2m, Double(cache["order_total"]))'
```

**4. Counting Ingested (OTel) Request Volume**

For **ingested traces only** (not eBPF), you can count absolute request volume because the pipeline runs before sampling.

**Use cases:**

* Total API request counts per endpoint
* Service-to-service call frequency
* Request volume per protocol type

{% hint style="warning" %}
Absolute counts from **eBPF spans** reflect sampled data only. Use them for relative comparisons and trend analysis, not for exact volume measurement.
{% endhint %}

#### How Spans-to-Metrics Works

Spans-to-metrics uses the special `s2m` map to define what metrics to create and how to aggregate them.

**Available Operations**

groundcover supports four metric aggregation operations:

| Function               | Description                   | Use Case                                |
| ---------------------- | ----------------------------- | --------------------------------------- |
| `span_to_metric_count` | Count spans matching criteria | Request counts, event occurrences       |
| `span_to_metric_sum`   | Sum extracted values          | Total duration, total payload size      |
| `span_to_metric_max`   | Maximum value observed        | Peak response time, largest payload     |
| `span_to_metric_min`   | Minimum value observed        | Fastest response time, smallest payload |

{% hint style="info" %}
groundcover automatically adds a `_gc_op` suffix with the operation type to generated metrics (e.g., `_sum`, `_min`, `_max`, `_count`).
{% endhint %}

**Basic Structure**

```yaml
ottlRules:
  - ruleName: s2m-example
    conditions:
      - workload == "my-service"
    statements:
      # 1. Define metric labels in s2m map
      - set(s2m["label_name"], attributes["field"])
      
      # 2. Create metrics with aggregations
      - span_to_metric_count("metric_name", s2m)
      - span_to_metric_sum("metric_name", s2m, Double(attributes["value"]))
```

#### Best Practices

1. **Choose meaningful metric names** - Use descriptive names that indicate what's being measured
   * Good: `http_requests_total`, `order_processing_duration`
   * Bad: `metric1`, `counter`
2. **Use appropriate labels** - Add dimensions that help you slice and dice the data
   * Common labels: `workload`, `endpoint`, `protocol_type`, `status`, `namespace`
   * Avoid high-cardinality labels (unique trace IDs, user IDs, span IDs)
3. **Use conditions to scope rules** - Only generate metrics from relevant spans to minimize overhead
4. **Combine operations** - Use count, sum, min, and max together for comprehensive insights
   * Count requests + sum duration = average latency
   * Min/max provide performance bounds
5. **Use type conversion** - Always convert values to `Double()` for sum/min/max operations
   * `Double(attributes["duration"])` not `attributes["duration"]`
6. **Prefer ratios over absolute counts for eBPF** - Since eBPF spans are sampled, ratios (e.g. error rate) are more reliable than raw counts

#### Viewing Your Metrics

After creating spans-to-metrics rules:

1. Metrics appear in [**Metrics Explorer**](https://app.groundcover.com/explore/data-explorer) within minutes
2. Use PromQL to query your custom metrics
3. Create [dashboards](https://app.groundcover.com/dashboards) to visualize trends
4. Set up [monitors](https://app.groundcover.com/monitors) for alerting on thresholds

#### Common Use Cases

**Tracking Request Duration**

Monitor response times with min, max, and sum. Duration metrics are accurate even on sampled eBPF data.

```yaml
ottlRules:
  - ruleName: s2m-request-duration
    conditions:
      - workload == "api-gateway"
      - protocol_type == "http"
    conditionLogicOperator: "and"
    statements:
      - set(s2m["endpoint"], span_name)
      - set(s2m["workload"], workload)
      - span_to_metric_sum("request_duration_ns", s2m, Double(duration))
      - span_to_metric_max("request_duration_ns", s2m, Double(duration))
      - span_to_metric_min("request_duration_ns", s2m, Double(duration))
      - span_to_metric_count("request_duration_ns", s2m)
```

**Output metrics:**

```
request_duration_ns_sum{endpoint="GET /api/users", workload="api-gateway"} = 165000000
request_duration_ns_max{endpoint="GET /api/users", workload="api-gateway"} = 120000000
request_duration_ns_min{endpoint="GET /api/users", workload="api-gateway"} = 45000000
request_duration_ns_count{endpoint="GET /api/users", workload="api-gateway"} = 2
```

💡 **What it does:** Tracks latency distribution per endpoint. Calculate average latency with `rate(sum) / rate(count)`.

**Tracking Error Ratios**

Compare error spans to total spans for reliable error rate monitoring.

```yaml
ottlRules:
  - ruleName: s2m-all-requests
    conditions:
      - protocol_type == "http"
    statements:
      - set(s2m["workload"], workload)
      - set(s2m["namespace"], namespace)
      - span_to_metric_count("http_spans_total", s2m)

  - ruleName: s2m-error-requests
    conditions:
      - protocol_type == "http"
      - status == "Error"
    conditionLogicOperator: "and"
    statements:
      - set(s2m["workload"], workload)
      - set(s2m["namespace"], namespace)
      - span_to_metric_count("http_spans_errors", s2m)
```

💡 **What it does:** Creates two count metrics. Calculate error rate with `rate(http_spans_errors) / rate(http_spans_total)`. This ratio is reliable even on sampled eBPF data.

**Monitoring Payload Size**

Track request and response body sizes from span attributes.

```yaml
ottlRules:
  - ruleName: s2m-payload-size
    conditions:
      - workload == "api-gateway"
      - attributes["response_size"] != nil
    conditionLogicOperator: "and"
    statements:
      - set(s2m["endpoint"], span_name)
      - set(s2m["workload"], workload)
      - span_to_metric_sum("response_bytes", s2m, Double(attributes["response_size"]))
      - span_to_metric_max("response_bytes", s2m, Double(attributes["response_size"]))
      - span_to_metric_min("response_bytes", s2m, Double(attributes["response_size"]))
      - span_to_metric_count("response_bytes", s2m)
```

💡 **What it does:** Tracks payload size distribution. Min/max values are accurate regardless of sampling.

#### Key Functions

**s2m Map**

The `s2m` map stores the labels (dimensions) for your metrics.

```yaml
- set(s2m["label_name"], "value")
- set(s2m["endpoint"], span_name)
- set(s2m["workload"], workload)
```

**span\_to\_metric\_count**

Counts the number of spans matching the rule.

**Syntax:**

```yaml
- span_to_metric_count("metric_name", s2m)
```

**Use for:** Request counts, event occurrences, error counts

**span\_to\_metric\_sum**

Sums numerical values from spans.

**Syntax:**

```yaml
- span_to_metric_sum("metric_name", s2m, Double(attributes["value"]))
```

**Use for:** Total duration, total bytes, cumulative values

**span\_to\_metric\_max**

Tracks the maximum value observed.

**Syntax:**

```yaml
- span_to_metric_max("metric_name", s2m, Double(attributes["value"]))
```

**Use for:** Peak response times, largest payloads

**span\_to\_metric\_min**

Tracks the minimum value observed.

**Syntax:**

```yaml
- span_to_metric_min("metric_name", s2m, Double(attributes["value"]))
```

**Use for:** Fastest response times, smallest payloads


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.groundcover.com/use-groundcover/data-pipelines/traces-pipeline/span-to-metrics.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.