Query Traces

Search and aggregate trace spans over HTTP using gcQL queries - find failing requests, compute latency percentiles, and calculate error rates per endpoint.

Endpoint

POST /api/traces/v2/search

Authentication

This endpoint requires API Key authentication via the Authorization header.

Headers

Header	Required	Description
Authorization	Yes	Bearer token with your API key
Content-Type	Yes	Must be application/json
X-Backend-Id	Only for multi-backend accounts	Your backend identifier, found under Settings → Access → API Keys (details)

Request Body

Parameter	Type	Required	Description
start	string	Yes	Start of the time range (RFC3339 / ISO 8601)
end	string	Yes	End of the time range (RFC3339 / ISO 8601)
query	string	Yes	The gcQL query to execute

Error Conventions in Traces

Traces carry two distinct error dimensions - pick the one that matches your question:

Filter	Meaning
status_code>=500	HTTP-layer server errors (use status_code>=400 to include client errors)
status:error	Span-level error status, set by instrumentation

Examples

Recent Server Errors

Fetch HTTP spans that returned a 5xx status:

curl 'https://api.groundcover.com/api/traces/v2/search' \
  -H 'authorization: Bearer <YOUR_API_KEY>' \
  -H 'content-type: application/json' \
  --data-raw '{
    "start": "2026-08-22T14:00:00Z",
    "end": "2026-08-22T15:00:00Z",
    "query": "span_type:http status_code>=500 | sort by (_time desc) | fields _time, workload, resource, status_code, duration_seconds | limit 20"
  }'

Example Response:

[
  {
    "_time": "2026-08-22T14:02:07.264995950Z",
    "workload": "api-gateway",
    "resource": "POST /api/orders",
    "status_code": "500",
    "duration_seconds": 31.230929898
  }
]

Latency per Endpoint

Compute request counts, average latency, and p99 latency per workload and resource in a single query:

curl 'https://api.groundcover.com/api/traces/v2/search' \
  -H 'authorization: Bearer <YOUR_API_KEY>' \
  -H 'content-type: application/json' \
  --data-raw '{
    "start": "2026-08-22T14:00:00Z",
    "end": "2026-08-22T15:00:00Z",
    "query": "span_type:http | stats by (workload, resource) count() as requests, avg(duration_seconds) as avg_latency, quantile(0.99, duration_seconds) as p99_latency | sort by (requests desc) | limit 10"
  }'

Example Response:

[
  {
    "workload": "fleet-manager",
    "resource": "POST /api/client/v2/config",
    "requests": 1733026,
    "avg_latency": 0.001108079438705479,
    "p99_latency": 0.0029195323400000193
  }
]

There is no p99() function - use quantile(0.99, duration_seconds), or quantiles(0.5, 0.95, 0.99, duration_seconds) to compute several percentiles in one pass.

Error Rate per Workload

Combine conditional counting with | math to compute an error percentage, and | filter to keep only meaningful traffic:

curl 'https://api.groundcover.com/api/traces/v2/search' \
  -H 'authorization: Bearer <YOUR_API_KEY>' \
  -H 'content-type: application/json' \
  --data-raw '{
    "start": "2026-08-22T14:00:00Z",
    "end": "2026-08-22T15:00:00Z",
    "query": "* | stats by (workload) count() as requests, count() if (status_code>=500) as errors | math errors / requests * 100 as error_pct keep_existing | filter requests > 100 | sort by (error_pct desc) | limit 10"
  }'

Example Response:

[
  {
    "workload": "api-gateway",
    "requests": 26667037,
    "errors": 905,
    "error_pct": 0.0033937028699513932
  }
]

Best Practices

1. Filter by span_type (http, grpc, sql, etc.) when your question targets a specific protocol.

2. Traces do not support bare free-text keywords - use field:*term* or *:*term* for substring matching, and prefer exact field filters where possible.

3. Use the right error dimension: status_code for HTTP status errors, status:error for instrumentation-reported span errors.

4. Always bound results with | limit N.