> For the complete documentation index, see [llms.txt](https://docs.groundcover.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.groundcover.com/reference/remote-access-and-apis/api-examples/query-ai-observability.md).

# Query AI Observability

Search AI spans, chart token usage and costs, and retrieve or manage model pricing.

AI observability queries operate on AI spans. The API automatically restricts searches to `protocol_type:gen_ai`, and free-text queries search AI input/output messages where collected.

### Endpoints

| Operation                                       | Endpoint                                  |
| ----------------------------------------------- | ----------------------------------------- |
| Search AI spans                                 | **POST** `/api/gen-ai/search`             |
| Query a time series                             | **POST** `/api/gen-ai/search-time-series` |
| List effective model pricing                    | **GET** `/api/gen-ai/pricing`             |
| Read custom pricing status — **Admin required** | **GET** `/api/gen-ai/pricing/custom`      |
| Replace custom price file — **Admin required**  | **PUT** `/api/gen-ai/pricing/custom`      |
| Remove custom pricing — **Admin required**      | **DELETE** `/api/gen-ai/pricing/custom`   |
| Download custom price file — **Admin required** | **GET** `/api/gen-ai/pricing/custom/file` |

### Authentication

Send an [API key](https://docs.groundcover.com/administer/api-keys.md) in `Authorization: Bearer <YOUR_API_KEY>` and select the backend with `X-Backend-Id: <YOUR_BACKEND_ID>`. Self-hosted deployments use their own API base URL. Custom-pricing operations require Admin permissions; uploads and deletion also require an authenticated user identity.

### Search Request

| Field            | Type    | Description                                                       |
| ---------------- | ------- | ----------------------------------------------------------------- |
| `start`, `end`   | string  | Time range in RFC3339 format.                                     |
| `query`          | string  | gcQL query.                                                       |
| `sources`        | array   | Optional structured scope conditions.                             |
| `filters`        | string  | Optional additional filters.                                      |
| `enableStream`   | boolean | Optional; `true` returns Server-Sent Events. Defaults to `false`. |
| `bucketDuration` | string  | Required for time series, for example `5m`.                       |
| `valueField`     | string  | Numeric result alias to chart.                                    |
| `fillValue`      | number  | Optional replacement for empty time buckets; defaults to `null`.  |

### Example: Token Usage

```bash
curl --request POST 'https://api.groundcover.com/api/gen-ai/search' \
  --header 'Authorization: Bearer <YOUR_API_KEY>' \
  --header 'X-Backend-Id: <YOUR_BACKEND_ID>' \
  --header 'Content-Type: application/json' \
  --data '{
    "start": "2026-10-28T10:00:00Z",
    "end": "2026-10-28T11:00:00Z",
    "query": "* | stats by (workload) sum(gen_ai.usage.input_tokens) as input_tokens, sum(gen_ai.usage.output_tokens) as output_tokens | limit 20"
  }'
```

The response contains `data` rows and `valueFieldHints` for numeric columns. For example:

```json
{
  "data": [{"workload": "assistant-api", "input_tokens": 2000, "output_tokens": 500}],
  "valueFieldHints": ["input_tokens", "output_tokens"]
}
```

The values above are illustrative. When `enableStream` is `true`, consume SSE events named `gen_ai_search` through the final `done: true` event. Events contain `data`, `valueFieldHints`, `estimate`, and `done`; handle streamed errors as described in [Search v3 and Streaming](https://docs.groundcover.com/reference/remote-access-and-apis/api-examples/search-v3-and-streaming.md).

### Example: Model Cost over Time

```bash
curl --request POST 'https://api.groundcover.com/api/gen-ai/search-time-series' \
  --header 'Authorization: Bearer <YOUR_API_KEY>' \
  --header 'X-Backend-Id: <YOUR_BACKEND_ID>' \
  --header 'Content-Type: application/json' \
  --data '{
    "start": "2026-10-28T10:00:00Z",
    "end": "2026-10-28T11:00:00Z",
    "query": "* | stats by (workload) sum(gc.llm_cost.input) as input_cost",
    "bucketDuration": "5m",
    "valueField": "input_cost",
    "fillValue": 0
  }'
```

The response is an array of series with `metric` labels and `velocity` points. Input cost is one cost component; output, cache-read, and cache-write costs have separate fields. See [Time-Series Queries](https://docs.groundcover.com/reference/remote-access-and-apis/api-examples/time-series-queries.md).

### List Model Pricing

```bash
curl --get 'https://api.groundcover.com/api/gen-ai/pricing' \
  --header 'Authorization: Bearer <YOUR_API_KEY>' \
  --header 'X-Backend-Id: <YOUR_BACKEND_ID>' \
  --data-urlencode 'query=provider:openai'
```

The optional `query` parameter accepts gcQL filters only, without pipes or aggregations. Omit it to list all prices. The response contains a `pricing` array with `provider`, `model`, `modelId`, `displayName`, `inputPrice`, `outputPrice`, `cacheReadPrice`, `cacheWritePrice`, `source`, and `fetchedAt`.

### Manage Custom Pricing

**Admin required** for every custom-pricing operation, including reading status and downloading the file. The AI queries and effective model-pricing lookup above do not require Admin permissions.

The custom-pricing API uses the file formats described in [Custom Model Pricing](https://docs.groundcover.com/investigate/ai-observability/ai-observability/custom-model-pricing.md). Upload CSV or supported JSON as multipart field `file`; prices use USD per token. The maximum file size is 10 MiB.

For example, save this synthetic configuration as `model-prices.csv`, replacing the model identity and rates with your own:

```csv
provider,model,input_cost_per_token,output_cost_per_token
example-provider,example-model,0.000001,0.000002
```

Use an Admin user credential that supplies a user identity for this operation. An API key without an associated user identity cannot upload or delete pricing and receives HTTP `401`.

```bash
curl --request PUT 'https://api.groundcover.com/api/gen-ai/pricing/custom' \
  --header 'Authorization: Bearer <YOUR_ADMIN_USER_TOKEN>' \
  --header 'X-Backend-Id: <YOUR_BACKEND_ID>' \
  --form 'file=@model-prices.csv'
```

GET, PUT and DELETE return `file`, `savedRevision`, `activeRevision`, `status` (`applying` or `active`), and an optional `error`. Poll **GET** `/api/gen-ai/pricing/custom` until the saved revision is active. Download the original file with **GET** `/api/gen-ai/pricing/custom/file`. **DELETE** `/api/gen-ai/pricing/custom` removes the custom file and restores shared prices for new spans.

### Errors

Invalid gcQL or time-series selection returns HTTP `400`. An unsupported backend returns `404`. Custom-pricing operations can also return `403` for missing Admin permissions, `413` for oversized uploads, or `503` when storage or activation services are unavailable.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.groundcover.com/reference/remote-access-and-apis/api-examples/query-ai-observability.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
