> For the complete documentation index, see [llms.txt](https://docs.groundcover.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.groundcover.com/reference/remote-access-and-apis/api-examples/query-apm-measurements.md). # Query APM Measurements Query groundcover's APM measurements (requests, errors, latency) with gcQL. Two endpoints are available: search for tabular results and search-time-series for charted results over time. The APM measurements APIs let you query groundcover's pre-aggregated application performance data using [gcQL](/investigate/querying-your-groundcover-data/groundcover-query-language.md). They back the **APM** data source in [Explore](/investigate/querying-your-groundcover-data/explore-and-monitors-query-builder.md#apm), so anything you can build in the UI you can also run from a script. {% hint style="info" %} These endpoints replace the legacy `/api/resources/v3/apis/*` API catalog endpoints (`list`, `requests`, `errors`, `latencies`, `filters`) for automated reporting. Use `/api/apm-measurements/search` where you used `/api/resources/v3/apis/list`, and `/api/apm-measurements/search-time-series` where you charted `requests`, `errors` or `latencies` over time. {% endhint %} ### Endpoints #### Search **POST** `https://api.groundcover.com/api/apm-measurements/search` Run a gcQL query and get back rows. Use this for catalog-style listings and top-N reports. #### Search Time Series **POST** `https://api.groundcover.com/api/apm-measurements/search-time-series` Run the same gcQL query bucketed over time and get back one series per group. Use this for charts and trends. ### Authentication Both endpoints require an [API Key](/administer/api-keys.md) in the Authorization header: ```bash Authorization: Bearer ``` If your account has multiple backends, also send the Backend ID: ```bash X-Backend-Id: ``` To locate your Backend ID, navigate to **Settings → Access**, open the **API Keys** tab, and find it in the section header. ### The query contract Every APM query **must** include two top-level filters, otherwise the request fails with `400`: 1. **`resource_type`** - the protocol, using an equality (`resource_type:http`) or an `in` filter (`resource_type:in("http","grpc")`). 2. **A traffic direction** - either `is_inbound:true` or `is_outbound:true`. Pick exactly one. "Top-level" means the filters must sit in the pre-pipe filter section, not inside an `OR` subgroup and not after a `| filter` stage: ```bash # Works - both filters top-level, before the first pipe resource_type:http is_inbound:true | stats sum(total_counter) as requests # Works - multiple protocols via a top-level in() filter resource_type:in(http, grpc) is_inbound:true | stats by (resource_type) sum(total_counter) as requests # Fails (400) - the direction sits after a pipe instead of top-level resource_type:http | filter is_inbound:true | stats sum(total_counter) as requests # Fails (400) - resource_type is inside an OR subgroup; use in() instead (resource_type:http OR resource_type:grpc) is_inbound:true | stats sum(total_counter) as requests ``` **Traffic direction** | Direction | Meaning | | ------------------ | --------------------------------------------------------------- | | `is_inbound:true` | The workload as a **server**, receiving calls | | `is_outbound:true` | The workload as a **client**, issuing calls to its dependencies | **Resource types** `resource_type` is the protocol the eBPF sensor decoded. The values are: | Category | Values | | ------------------ | ----------------------------------------------------------------------- | | Web / RPC | `http`, `grpc`, `graphql` | | Databases & caches | `postgresql`, `mysql`, `redis`, `mongodb` | | Messaging | `kafka`, `amqp`, `sqs` | | Cloud & network | `s3`, `dns`, `sip` | | LLM providers | `openai`, `anthropic`, `bedrock`, `vertex_ai`, `google_genai`, `gen_ai` | {% hint style="info" %} SQL traffic is reported under its **dialect**, not a generic `sql` value: use `resource_type:postgresql` or `resource_type:mysql`. {% endhint %} Which values are present depends on your sensor version and what your workloads actually talk to. A `resource_type` that matches nothing returns an empty result set rather than an error. **Measurement columns** These are the numeric columns you aggregate over in a `| stats` stage: | Column | Description | | --------------------------- | -------------------------------------------------------------------------------- | | `total_counter` | Number of requests | | `success_counter` | Requests that completed successfully | | `error_counter` | Requests that completed with an error status | | `issue_counter` | Requests flagged as issues. Every issue is an error, not every error is an issue | | `total_latency_seconds` | Summed request latency, for computing averages | | `latency_seconds_quantiles` | Latency distribution. Read it with `quantile(, latency_seconds_quantiles)` | {% hint style="warning" %} The same traffic can be reported by more than one `source`, so aggregating across sources double-counts it. Filter to a single source (for example `| filter source:eBPF`) or add `source` to your `stats by (...)` clause. APM measurements come from two sources: Source values are **case-sensitive**, and a value that matches nothing returns an empty result set rather than an error - so note the mixed case on `eBPF` against the lower case on `opentelemetry`. Run ` | field_values_with_hits source` to see which sources are present in your own environment. {% endhint %} | Source | Reported by | | --------------- | --------------------------------------------- | | `eBPF` | The groundcover eBPF sensor | | `opentelemetry` | Spans ingested from an OpenTelemetry pipeline | ### Search Endpoint #### Request **Headers** ```bash Authorization: Bearer Accept: application/json Content-Type: application/json X-Backend-Id: ``` **Request Body** | Parameter | Type | Required | Description | | ---------- | ------ | -------- | ----------------------------------------------------------------------------------------- | | `start` | string | **Yes** | Range start in RFC3339 format | | `end` | string | **Yes** | Range end in RFC3339 format. Must be at or after `start` | | `query` | string | **Yes** | gcQL query. Required unless you send `pipeline` instead | | `pipeline` | object | No | Structured pipeline, as an alternative to `query`. Same top-level filter contract applies | | `filters` | string | No | Extra gcQL filters ANDed onto the query | | `sources` | array | No | Additional filter conditions. Prefer expressing these in `query` | #### Response ```json { "data": [ {"server": "checkout", "requests": 462617906, "errors": 4169099}, {"server": "payments", "requests": 412072465, "errors": 7403} ], "valueFieldHints": ["requests", "errors"] } ``` | Field | Type | Description | | ----------------- | ----- | ----------------------------------------------------------------------------------------------- | | `data` | array | One object per result row. Keys are the group-by labels and the aliases from your `stats` stage | | `valueFieldHints` | array | Names of the numeric columns in `data`, useful for deciding what to plot | #### Examples **Top 20 servers by inbound HTTP requests and errors** This is the direct replacement for `/api/resources/v3/apis/list`: ```bash curl 'https://api.groundcover.com/api/apm-measurements/search' \ -H 'accept: application/json' \ -H 'authorization: Bearer ' \ -H 'content-type: application/json' \ -H 'X-Backend-Id: ' \ --data-raw '{ "start": "2026-08-09T00:00:00.000Z", "end": "2026-08-10T00:00:00.000Z", "query": "resource_type:http is_inbound:true | filter source:eBPF | stats by (server) sum(total_counter) as requests, sum(error_counter) as errors | sort by (requests desc) | limit 20" }' ``` **Slowest HTTP routes by p95 latency** ```bash curl 'https://api.groundcover.com/api/apm-measurements/search' \ -H 'accept: application/json' \ -H 'authorization: Bearer ' \ -H 'content-type: application/json' \ -H 'X-Backend-Id: ' \ --data-raw '{ "start": "2026-08-09T00:00:00.000Z", "end": "2026-08-10T00:00:00.000Z", "query": "resource_type:http is_inbound:true | filter source:eBPF | stats by (server, clustered_path, method) quantile(0.95, latency_seconds_quantiles) as p95 | sort by (p95 desc) | limit 20" }' ``` **Error rate of outbound PostgreSQL calls, per client** ```bash curl 'https://api.groundcover.com/api/apm-measurements/search' \ -H 'accept: application/json' \ -H 'authorization: Bearer ' \ -H 'content-type: application/json' \ -H 'X-Backend-Id: ' \ --data-raw '{ "start": "2026-08-09T00:00:00.000Z", "end": "2026-08-10T00:00:00.000Z", "query": "resource_type:postgresql is_outbound:true | filter source:eBPF | stats by (client) sum(error_counter) as errors, sum(total_counter) as requests | sort by (errors desc) | limit 20" }' ``` **Scoping to a namespace with `filters`** ```bash curl 'https://api.groundcover.com/api/apm-measurements/search' \ -H 'accept: application/json' \ -H 'authorization: Bearer ' \ -H 'content-type: application/json' \ -H 'X-Backend-Id: ' \ --data-raw '{ "start": "2026-08-09T00:00:00.000Z", "end": "2026-08-10T00:00:00.000Z", "query": "resource_type:http is_inbound:true | filter source:eBPF | stats by (server) sum(total_counter) as requests | sort by (requests desc) | limit 20", "filters": "namespace:production" }' ``` **Discovering filter values** This is the replacement for `/api/resources/v3/apis/filters`. Use the `field_values_with_hits` pipe to list the values a label actually takes in the selection, along with how many requests each one accounts for: ```bash curl 'https://api.groundcover.com/api/apm-measurements/search' \ -H 'accept: application/json' \ -H 'authorization: Bearer ' \ -H 'content-type: application/json' \ -H 'X-Backend-Id: ' \ --data-raw '{ "start": "2026-08-09T00:00:00.000Z", "end": "2026-08-10T00:00:00.000Z", "query": "resource_type:http is_inbound:true | field_values_with_hits server | limit 50" }' ``` ```json { "data": [ {"value": "checkout", "hits": 472251224}, {"value": "payments", "hits": 421683425} ], "valueFieldHints": ["hits"] } ``` Swap `server` for any label you want to populate a picker with, for example `namespace`, `clustered_path`, `span_name` or `response_status_code`. The mandatory top-level `resource_type` and direction filters still apply. ### Search Time Series Endpoint Same body as the search endpoint, plus the bucketing fields. groundcover applies the bucket to your query, so do not add a bucket stage yourself. {% hint style="warning" %} **`limit` applies per bucket here, not to the whole result.** When the bucket is applied, a trailing `| limit N` is rewritten to cap rows *within each time bucket*. With a `stats by (...)` clause that means you get the top N groups **in each bucket independently**, so a group that ranks in the top N during some buckets and not others produces a series with gaps, and the set of series can change from bucket to bucket. If you need a stable set of series, pick the groups first with a [search](#search-endpoint) call, then query the time series with those groups pinned in the `query` filters rather than relying on `limit` to choose them. {% endhint %} #### Request **Request Body** | Parameter | Type | Required | Description | | ---------------- | ------ | ----------- | -------------------------------------------------------------------------------------------------------------- | | `start` | string | **Yes** | Range start in RFC3339 format | | `end` | string | **Yes** | Range end in RFC3339 format | | `query` | string | Conditional | gcQL query. Provide either `query` or `pipeline`. Sending neither returns `400` | | `pipeline` | object | Conditional | Structured pipeline, as an alternative to `query`. Same top-level filter contract applies | | `bucketDuration` | string | **Yes** | Bucket width, for example `"1m"`, `"5m"`, `"1h"`. Also accepts `"5 minutes"` style values | | `valueField` | string | No | Which numeric column to plot, or a comma-separated list. Resolved automatically from the response when omitted | | `fillValue` | number | No | Value used for empty buckets. Buckets are filled with `null` when omitted | | `filters` | string | No | Extra gcQL filters ANDed onto the query | | `sources` | array | No | Additional filter conditions. Prefer expressing these in `query` | {% hint style="info" %} On live queries (where `end` is at or near the current time) the trailing partial bucket is dropped, so the last point is always a complete bucket. {% endhint %} #### Response A bare top-level array of series, each with its labels and its data points: ```json [ { "velocity": [ [1786310400, "19204.5"], [1786314000, "20117"], [1786317600, null] ], "metric": {"server": "checkout"} }, { "velocity": [ [1786310400, "8021"], [1786314000, "8115.25"], [1786317600, "7998"] ], "metric": {"server": "payments"} } ] ``` | Field | Type | Description | | ------------ | ------ | ------------------------------------------- | | `velocity` | array | Data points for the series, one per bucket | | `velocity[]` | array | A `[timestamp, value]` pair | | `metric` | object | The group-by labels identifying this series | Each data point contains: * **Timestamp**: bucket start as a Unix timestamp in seconds * **Value**: aggregated value for the bucket as a string, or `null` for an empty bucket when `fillValue` is not set {% hint style="warning" %} Each series object has the same `velocity` / `metric` shape as the [metrics range query](/reference/remote-access-and-apis/api-examples/query-metrics.md), but the envelope differs: that endpoint wraps its series in `{"velocities": [...], "promql": "..."}`, while this one returns the array directly. Parse the response body itself, not `response.velocities`. {% endhint %} #### Examples **Hourly inbound HTTP request count per server** This replaces charted reporting off `/api/resources/v3/apis/requests`. To chart a stable set of series, first pick the servers with a [search](#search-endpoint) call (for example, the top 20 by requests), then pin them in the time-series query instead of using a trailing `| limit`: ```bash curl 'https://api.groundcover.com/api/apm-measurements/search-time-series' \ -H 'accept: application/json' \ -H 'authorization: Bearer ' \ -H 'content-type: application/json' \ -H 'X-Backend-Id: ' \ --data-raw '{ "start": "2026-08-09T00:00:00.000Z", "end": "2026-08-10T00:00:00.000Z", "bucketDuration": "1h", "query": "resource_type:http is_inbound:true server:in(checkout, payments) | filter source:eBPF | stats by (server) sum(total_counter) as requests", "valueField": "requests", "fillValue": 0 }' ``` **p99 latency over time per route** This replaces charted reporting off `/api/resources/v3/apis/latencies`. Here the trailing `| limit 10` is used deliberately for its per-bucket semantics: each 5-minute bucket independently keeps its 10 slowest routes, so the chart surfaces whichever routes were slowest at each point in time rather than a fixed set of series: ```bash curl 'https://api.groundcover.com/api/apm-measurements/search-time-series' \ -H 'accept: application/json' \ -H 'authorization: Bearer ' \ -H 'content-type: application/json' \ -H 'X-Backend-Id: ' \ --data-raw '{ "start": "2026-08-09T00:00:00.000Z", "end": "2026-08-10T00:00:00.000Z", "bucketDuration": "5m", "query": "resource_type:http is_inbound:true | filter source:eBPF | stats by (clustered_path) quantile(0.99, latency_seconds_quantiles) as p99 | sort by (p99 desc) | limit 10", "valueField": "p99" }' ``` **Errors over time, replacing `/api/resources/v3/apis/errors`** Like the latency example, the `| limit 10` keeps the 10 highest-error servers within each 15-minute bucket independently. Pin the servers with `server:in(...)` instead if you need a fixed set of series: ```bash curl 'https://api.groundcover.com/api/apm-measurements/search-time-series' \ -H 'accept: application/json' \ -H 'authorization: Bearer ' \ -H 'content-type: application/json' \ -H 'X-Backend-Id: ' \ --data-raw '{ "start": "2026-08-09T00:00:00.000Z", "end": "2026-08-10T00:00:00.000Z", "bucketDuration": "15m", "query": "resource_type:http is_inbound:true | filter source:eBPF | stats by (server) sum(error_counter) as errors | sort by (errors desc) | limit 10", "valueField": "errors", "fillValue": 0 }' ``` ### Errors | Status | Condition | | ------ | -------------------------------------------------------------------------------------------------------- | | `400` | The query is missing a top-level `resource_type` filter | | `400` | The query is missing a top-level `is_inbound:true` or `is_outbound:true` filter | | `400` | gcQL parse error, or an invalid `bucketDuration` | | `400` | The query exceeded the row cap. Narrow `resource_type` or the time range, or reduce group-by cardinality | Errors are returned as a JSON object with a human-readable `message`, a `trace_id`, and for some cases a stable machine-readable `code`. A query missing its `resource_type` filter: ```json {"query": "is_inbound:true | stats sum(total_counter) as requests"} ``` ```json { "message": "gcql parse error: apm search requires a top-level `resource_type` filter (EQ or IN) in the GCQL query", "trace_id": "93c33976207805aff1f4175a05483a98" } ``` A query that grouped by too many distinct values: ```json { "message": "query exceeded the row cap; narrow your filters (resource_type / time range) or reduce group-by cardinality", "code": "EXCEEDED_MAX_ROWS_TO_GROUP_BY", "trace_id": "93c33976207805aff1f4175a05483a98" } ``` ### Best Practices 1. **Always aggregate.** These endpoints read pre-aggregated measurements, but a bare filter query over a wide window still returns a lot of rows. Add a `| stats` stage and a `| limit`. 2. **Disambiguate the source.** Filter to one `source` or group by it, so eBPF and OpenTelemetry rows are not summed together. 3. **Keep group-by cardinality in check.** Grouping by a high-cardinality label over a wide window can hit the row cap. Check a candidate label first with `| stats count_uniq(