Example Queries

AI Observability data is stored as traces with gen_ai.* attributes and groundcover enrichment (gc.llm_cost.*). Filter GenAI spans using span_type:gen_ai in any gcQL query.

Run these queries in the Data Explorer, the search bar in AI Observability, or as the basis for dashboard widgets.

---

Call Volume & Distribution

Total LLM calls by model:

span_type:gen_ai | stats by(gen_ai.request.model) count()

Calls by provider:

span_type:gen_ai | stats by(gen_ai.provider.name) count()

Operation type breakdown:

span_type:gen_ai | stats by(gen_ai.operation.name) count()

---

Token Usage

Total tokens (input vs output):

span_type:gen_ai | stats sum(gen_ai.usage.input_tokens) as input_tokens, sum(gen_ai.usage.output_tokens) as output_tokens

Average tokens per request by model:

span_type:gen_ai | stats by(gen_ai.request.model) avg(gen_ai.usage.input_tokens) as avg_input, avg(gen_ai.usage.output_tokens) as avg_output

Top services by token consumption:

span_type:gen_ai | stats by(service.name) sum(gen_ai.usage.input_tokens) as total_tokens | sort by(total_tokens desc) | limit 10

---

Cost Analysis

Total cost by model:

span_type:gen_ai | stats by(gen_ai.request.model) sum(gc.llm_cost.input) as input_cost, sum(gc.llm_cost.output) as output_cost | math input_cost + output_cost as total_cost | sort by(total_cost desc)

Cost by service:

span_type:gen_ai | stats by(service.name) sum(gc.llm_cost.input) as input_cost, sum(gc.llm_cost.output) as output_cost | math input_cost + output_cost as cost | sort by(cost desc)

Most expensive individual calls:

span_type:gen_ai | math gc.llm_cost.input + gc.llm_cost.output as total_cost | sort by(total_cost desc) | limit 20

---

Latency & Performance

Average latency by provider:

span_type:gen_ai | stats by(gen_ai.provider.name) avg(duration_seconds) as avg_latency

P99 latency by model:

span_type:gen_ai | stats by(gen_ai.request.model) quantile(0.99, duration_seconds) as p99_latency

Latency distribution:

span_type:gen_ai | stats quantile(0.50, duration_seconds) as p50, quantile(0.95, duration_seconds) as p95, quantile(0.99, duration_seconds) as p99

---

Errors & Issues

Error count by model:

span_type:gen_ai status:error | stats by(gen_ai.request.model) count() as errors

Errors with details:

span_type:gen_ai status:error | stats by(gen_ai.request.model, error.type) count()

---

Agent & Tool Tracking

Calls by agent:

span_type:gen_ai -gen_ai.agent.name:"" | stats by(gen_ai.agent.name) count()

Tool execution frequency:

span_type:gen_ai gen_ai.operation.name:execute_tool | stats by(gen_ai.tool.name) count()

Cost breakdown by agent and model:

span_type:gen_ai -gen_ai.agent.name:"" | stats by(gen_ai.agent.name, gen_ai.request.model) sum(gc.llm_cost.input) as input_cost, sum(gc.llm_cost.output) as output_cost | math input_cost + output_cost as total_cost | sort by(total_cost desc)

---

Visualization Best Practices

Use Case	Recommended Display	Group By
Call volume trends	Time Series	gen_ai.request.model
Cost breakdown	Pie Chart	gen_ai.provider.name
Top expensive calls	Top List	—
Token usage stats	Stat	-
Model comparison	Table	gen_ai.request.model
Latency percentiles	Time Series	-
Error trends	Time Series (Stacked Bar)	error.type

To build dashboard widgets with these queries, see Creating Dashboards — select Traces mode when adding a chart widget.