> For the complete documentation index, see [llms.txt](https://docs.groundcover.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.groundcover.com/reference/remote-access-and-apis/api-examples/query-data-gcql/query-traces.md). # Query Traces Search and aggregate trace spans over HTTP using gcQL queries - find failing requests, compute latency percentiles, and calculate error rates per endpoint. ### Endpoint **POST** `/api/traces/v2/search` ### Authentication This endpoint requires API Key authentication via the Authorization header. ### Headers | Header | Required | Description | | --------------- | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Authorization` | Yes | Bearer token with your API key | | `Content-Type` | Yes | Must be `application/json` | | `X-Backend-Id` | Only for multi-backend accounts | Your backend identifier, found under **Settings → Access → API Keys** ([details](/reference/remote-access-and-apis/api-examples/query-data-gcql.md#authentication)) | ### Request Body | Parameter | Type | Required | Description | | --------- | ------ | -------- | -------------------------------------------- | | `start` | string | Yes | Start of the time range (RFC3339 / ISO 8601) | | `end` | string | Yes | End of the time range (RFC3339 / ISO 8601) | | `query` | string | Yes | The gcQL query to execute | ### Error Conventions in Traces Traces carry two distinct error dimensions - pick the one that matches your question: | Filter | Meaning | | ------------------ | -------------------------------------------------------------------------- | | `status_code>=500` | HTTP-layer server errors (use `status_code>=400` to include client errors) | | `status:error` | Span-level error status, set by instrumentation | ### Examples #### Recent Server Errors Fetch HTTP spans that returned a 5xx status: ```bash curl 'https://api.groundcover.com/api/traces/v2/search' \ -H 'authorization: Bearer ' \ -H 'content-type: application/json' \ --data-raw '{ "start": "2026-08-22T14:00:00Z", "end": "2026-08-22T15:00:00Z", "query": "span_type:http status_code>=500 | sort by (_time desc) | fields _time, workload, resource, status_code, duration_seconds | limit 20" }' ``` **Example Response:** ```json [ { "_time": "2026-08-22T14:02:07.264995950Z", "workload": "api-gateway", "resource": "POST /api/orders", "status_code": "500", "duration_seconds": 31.230929898 } ] ``` #### Latency per Endpoint Compute request counts, average latency, and p99 latency per workload and resource in a single query: ```bash curl 'https://api.groundcover.com/api/traces/v2/search' \ -H 'authorization: Bearer ' \ -H 'content-type: application/json' \ --data-raw '{ "start": "2026-08-22T14:00:00Z", "end": "2026-08-22T15:00:00Z", "query": "span_type:http | stats by (workload, resource) count() as requests, avg(duration_seconds) as avg_latency, quantile(0.99, duration_seconds) as p99_latency | sort by (requests desc) | limit 10" }' ``` **Example Response:** ```json [ { "workload": "fleet-manager", "resource": "POST /api/client/v2/config", "requests": 1733026, "avg_latency": 0.001108079438705479, "p99_latency": 0.0029195323400000193 } ] ``` {% hint style="info" %} There is no `p99()` function - use `quantile(0.99, duration_seconds)`, or `quantiles(0.5, 0.95, 0.99, duration_seconds)` to compute several percentiles in one pass. {% endhint %} #### Error Rate per Workload Combine conditional counting with `| math` to compute an error percentage, and `| filter` to keep only meaningful traffic: ```bash curl 'https://api.groundcover.com/api/traces/v2/search' \ -H 'authorization: Bearer ' \ -H 'content-type: application/json' \ --data-raw '{ "start": "2026-08-22T14:00:00Z", "end": "2026-08-22T15:00:00Z", "query": "* | stats by (workload) count() as requests, count() if (status_code>=500) as errors | math errors / requests * 100 as error_pct keep_existing | filter requests > 100 | sort by (error_pct desc) | limit 10" }' ``` **Example Response:** ```json [ { "workload": "api-gateway", "requests": 26667037, "errors": 905, "error_pct": 0.0033937028699513932 } ] ``` ### Best Practices 1. **Filter by `span_type`** (`http`, `grpc`, `sql`, etc.) when your question targets a specific protocol. 2. **Traces do not support bare free-text keywords** - use `field:*term*` or `*:*term*` for substring matching, and prefer exact field filters where possible. 3. **Use the right error dimension**: `status_code` for HTTP status errors, `status:error` for instrumentation-reported span errors. 4. **Always bound results** with `| limit N`. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.groundcover.com/reference/remote-access-and-apis/api-examples/query-data-gcql/query-traces.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `build a script that syncs our docs to a CMS` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.