# Example Queries

AI Observability data is stored as traces with `gen_ai.*` attributes and [groundcover enrichment](/use-groundcover/ai-observability/attributes.md#groundcover-enrichment) (`gc.llm_cost.*`). Filter GenAI spans using `span_type:gen_ai` in any [GCQL](/use-groundcover/querying-your-groundcover-data/groundcover-query-language.md) query.

Run these queries in the [Data Explorer](/use-groundcover/querying-your-groundcover-data/explore-and-monitors-query-builder.md), the search bar in [AI Observability](https://app.groundcover.com/ai-observability), or as the basis for [dashboard widgets](/use-groundcover/dashboards-and-alerts/create-a-dashboard.md).

***

## Call Volume & Distribution

**Total LLM calls by model:**

```gcql
span_type:gen_ai | stats by(gen_ai.request.model) count()
```

**Calls by provider:**

```gcql
span_type:gen_ai | stats by(gen_ai.provider.name) count()
```

**Operation type breakdown:**

```gcql
span_type:gen_ai | stats by(gen_ai.operation.name) count()
```

***

## Token Usage

**Total tokens (input vs output):**

```gcql
span_type:gen_ai | stats sum(gen_ai.usage.input_tokens) as input_tokens, sum(gen_ai.usage.output_tokens) as output_tokens
```

**Average tokens per request by model:**

```gcql
span_type:gen_ai | stats by(gen_ai.request.model) avg(gen_ai.usage.input_tokens) as avg_input, avg(gen_ai.usage.output_tokens) as avg_output
```

**Top services by token consumption:**

```gcql
span_type:gen_ai | stats by(service.name) sum(gen_ai.usage.input_tokens) as total_tokens | sort by(total_tokens desc) | limit 10
```

***

## Cost Analysis

**Total cost by model:**

```gcql
span_type:gen_ai | stats by(gen_ai.request.model) sum(gc.llm_cost.input) as input_cost, sum(gc.llm_cost.output) as output_cost | math input_cost + output_cost as total_cost | sort by(total_cost desc)
```

**Cost by service:**

```gcql
span_type:gen_ai | stats by(service.name) sum(gc.llm_cost.input) as input_cost, sum(gc.llm_cost.output) as output_cost | math input_cost + output_cost as cost | sort by(cost desc)
```

**Most expensive individual calls:**

```gcql
span_type:gen_ai | math gc.llm_cost.input + gc.llm_cost.output as total_cost | sort by(total_cost desc) | limit 20
```

***

## Latency & Performance

**Average latency by provider:**

```gcql
span_type:gen_ai | stats by(gen_ai.provider.name) avg(duration_seconds) as avg_latency
```

**P99 latency by model:**

```gcql
span_type:gen_ai | stats by(gen_ai.request.model) quantile(0.99, duration_seconds) as p99_latency
```

**Latency distribution:**

```gcql
span_type:gen_ai | stats quantile(0.50, duration_seconds) as p50, quantile(0.95, duration_seconds) as p95, quantile(0.99, duration_seconds) as p99
```

***

## Errors & Issues

**Error count by model:**

```gcql
span_type:gen_ai status:error | stats by(gen_ai.request.model) count() as errors
```

**Errors with details:**

```gcql
span_type:gen_ai status:error | stats by(gen_ai.request.model, error.type) count()
```

***

## Agent & Tool Tracking

**Calls by agent:**

```gcql
span_type:gen_ai gen_ai.agent.name:* | stats by(gen_ai.agent.name) count()
```

**Tool execution frequency:**

```gcql
span_type:gen_ai gen_ai.operation.name:execute_tool | stats by(gen_ai.tool.name) count()
```

**Cost breakdown by agent and model:**

```gcql
span_type:gen_ai gen_ai.agent.name:* | stats by(gen_ai.agent.name, gen_ai.request.model) sum(gc.llm_cost.input) as input_cost, sum(gc.llm_cost.output) as output_cost | math input_cost + output_cost as total_cost | sort by(total_cost desc)
```

***

## Visualization Best Practices

| Use Case            | Recommended Display       | Group By               |
| ------------------- | ------------------------- | ---------------------- |
| Call volume trends  | Time Series               | `gen_ai.request.model` |
| Cost breakdown      | Pie Chart                 | `gen_ai.provider.name` |
| Top expensive calls | Top List                  | —                      |
| Token usage stats   | Stat                      | -                      |
| Model comparison    | Table                     | `gen_ai.request.model` |
| Latency percentiles | Time Series               | -                      |
| Error trends        | Time Series (Stacked Bar) | `error.type`           |

To build dashboard widgets with these queries, see [Creating Dashboards](/use-groundcover/dashboards-and-alerts/create-a-dashboard.md) — select **Traces** mode when adding a chart widget.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.groundcover.com/use-groundcover/ai-observability/example-queries.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.