Query APM Measurements

Query groundcover's APM measurements (requests, errors, latency) with gcQL. Two endpoints are available: search for tabular results and search-time-series for charted results over time.

The APM measurements APIs let you query groundcover's pre-aggregated application performance data using gcQL. They back the APM data source in Explore, so anything you can build in the UI you can also run from a script.

These endpoints replace the legacy /api/resources/v3/apis/* API catalog endpoints (list, requests, errors, latencies, filters) for automated reporting. Use /api/apm-measurements/search where you used /api/resources/v3/apis/list, and /api/apm-measurements/search-time-series where you charted requests, errors or latencies over time.

Endpoints

Search

POST https://api.groundcover.com/api/apm-measurements/search

Run a gcQL query and get back rows. Use this for catalog-style listings and top-N reports.

Search Time Series

POST https://api.groundcover.com/api/apm-measurements/search-time-series

Run the same gcQL query bucketed over time and get back one series per group. Use this for charts and trends.

Authentication

Both endpoints require an API Key in the Authorization header:

Authorization: Bearer <YOUR_API_KEY>

If your account has multiple backends, also send the Backend ID:

X-Backend-Id: <YOUR_BACKEND_ID>

To locate your Backend ID, navigate to Settings → Access, open the API Keys tab, and find it in the section header.

The query contract

Every APM query must include two top-level filters, otherwise the request fails with 400:

1. resource_type - the protocol, using an equality (resource_type:http) or an in filter (resource_type:in("http","grpc")).

2. A traffic direction - either is_inbound:true or is_outbound:true. Pick exactly one.

"Top-level" means the filters must sit in the pre-pipe filter section, not inside an OR subgroup and not after a | filter stage:

# Works - both filters top-level, before the first pipe
resource_type:http is_inbound:true | stats sum(total_counter) as requests

# Works - multiple protocols via a top-level in() filter
resource_type:in(http, grpc) is_inbound:true | stats by (resource_type) sum(total_counter) as requests

# Fails (400) - the direction sits after a pipe instead of top-level
resource_type:http | filter is_inbound:true | stats sum(total_counter) as requests

# Fails (400) - resource_type is inside an OR subgroup; use in() instead
(resource_type:http OR resource_type:grpc) is_inbound:true | stats sum(total_counter) as requests

Traffic direction

Direction	Meaning
is_inbound:true	The workload as a server, receiving calls
is_outbound:true	The workload as a client, issuing calls to its dependencies

Resource types

resource_type is the protocol the eBPF sensor decoded. The values are:

Category	Values
Web / RPC	http, grpc, graphql
Databases & caches	postgresql, mysql, redis, mongodb
Messaging	kafka, amqp, sqs
Cloud & network	s3, dns, sip
LLM providers	openai, anthropic, bedrock, vertex_ai, google_genai, gen_ai

SQL traffic is reported under its dialect, not a generic sql value: use resource_type:postgresql or resource_type:mysql.

Which values are present depends on your sensor version and what your workloads actually talk to. A resource_type that matches nothing returns an empty result set rather than an error.

Measurement columns

These are the numeric columns you aggregate over in a | stats stage:

Column	Description
total_counter	Number of requests
success_counter	Requests that completed successfully
error_counter	Requests that completed with an error status
issue_counter	Requests flagged as issues. Every issue is an error, not every error is an issue
total_latency_seconds	Summed request latency, for computing averages
latency_seconds_quantiles	Latency distribution. Read it with quantile(<phi>, latency_seconds_quantiles)

The same traffic can be reported by more than one source, so aggregating across sources double-counts it. Filter to a single source (for example | filter source:eBPF) or add source to your stats by (...) clause.

APM measurements come from two sources:

Source values are case-sensitive, and a value that matches nothing returns an empty result set rather than an error - so note the mixed case on eBPF against the lower case on opentelemetry. Run <filters> | field_values_with_hits source to see which sources are present in your own environment.

Source	Reported by
eBPF	The groundcover eBPF sensor
opentelemetry	Spans ingested from an OpenTelemetry pipeline

Search Endpoint

Request

Headers

Authorization: Bearer <YOUR_API_KEY>
Accept: application/json
Content-Type: application/json
X-Backend-Id: <YOUR_BACKEND_ID>

Request Body

Parameter	Type	Required	Description
start	string	Yes	Range start in RFC3339 format
end	string	Yes	Range end in RFC3339 format. Must be at or after start
query	string	Yes	gcQL query. Required unless you send pipeline instead
pipeline	object	No	Structured pipeline, as an alternative to query. Same top-level filter contract applies
filters	string	No	Extra gcQL filters ANDed onto the query
sources	array	No	Additional filter conditions. Prefer expressing these in query

Response

{
  "data": [
    {"server": "checkout", "requests": 462617906, "errors": 4169099},
    {"server": "payments", "requests": 412072465, "errors": 7403}
  ],
  "valueFieldHints": ["requests", "errors"]
}

Field	Type	Description
data	array	One object per result row. Keys are the group-by labels and the aliases from your stats stage
valueFieldHints	array	Names of the numeric columns in data, useful for deciding what to plot

Examples

Top 20 servers by inbound HTTP requests and errors

This is the direct replacement for /api/resources/v3/apis/list:

curl 'https://api.groundcover.com/api/apm-measurements/search' \
  -H 'accept: application/json' \
  -H 'authorization: Bearer <YOUR_API_KEY>' \
  -H 'content-type: application/json' \
  -H 'X-Backend-Id: <YOUR_BACKEND_ID>' \
  --data-raw '{
    "start": "2026-08-09T00:00:00.000Z",
    "end": "2026-08-10T00:00:00.000Z",
    "query": "resource_type:http is_inbound:true | filter source:eBPF | stats by (server) sum(total_counter) as requests, sum(error_counter) as errors | sort by (requests desc) | limit 20"
  }'

Slowest HTTP routes by p95 latency

curl 'https://api.groundcover.com/api/apm-measurements/search' \
  -H 'accept: application/json' \
  -H 'authorization: Bearer <YOUR_API_KEY>' \
  -H 'content-type: application/json' \
  -H 'X-Backend-Id: <YOUR_BACKEND_ID>' \
  --data-raw '{
    "start": "2026-08-09T00:00:00.000Z",
    "end": "2026-08-10T00:00:00.000Z",
    "query": "resource_type:http is_inbound:true | filter source:eBPF | stats by (server, clustered_path, method) quantile(0.95, latency_seconds_quantiles) as p95 | sort by (p95 desc) | limit 20"
  }'

Error rate of outbound PostgreSQL calls, per client

curl 'https://api.groundcover.com/api/apm-measurements/search' \
  -H 'accept: application/json' \
  -H 'authorization: Bearer <YOUR_API_KEY>' \
  -H 'content-type: application/json' \
  -H 'X-Backend-Id: <YOUR_BACKEND_ID>' \
  --data-raw '{
    "start": "2026-08-09T00:00:00.000Z",
    "end": "2026-08-10T00:00:00.000Z",
    "query": "resource_type:postgresql is_outbound:true | filter source:eBPF | stats by (client) sum(error_counter) as errors, sum(total_counter) as requests | sort by (errors desc) | limit 20"
  }'

Scoping to a namespace with filters

curl 'https://api.groundcover.com/api/apm-measurements/search' \
  -H 'accept: application/json' \
  -H 'authorization: Bearer <YOUR_API_KEY>' \
  -H 'content-type: application/json' \
  -H 'X-Backend-Id: <YOUR_BACKEND_ID>' \
  --data-raw '{
    "start": "2026-08-09T00:00:00.000Z",
    "end": "2026-08-10T00:00:00.000Z",
    "query": "resource_type:http is_inbound:true | filter source:eBPF | stats by (server) sum(total_counter) as requests | sort by (requests desc) | limit 20",
    "filters": "namespace:production"
  }'

Discovering filter values

This is the replacement for /api/resources/v3/apis/filters. Use the field_values_with_hits pipe to list the values a label actually takes in the selection, along with how many requests each one accounts for:

curl 'https://api.groundcover.com/api/apm-measurements/search' \
  -H 'accept: application/json' \
  -H 'authorization: Bearer <YOUR_API_KEY>' \
  -H 'content-type: application/json' \
  -H 'X-Backend-Id: <YOUR_BACKEND_ID>' \
  --data-raw '{
    "start": "2026-08-09T00:00:00.000Z",
    "end": "2026-08-10T00:00:00.000Z",
    "query": "resource_type:http is_inbound:true | field_values_with_hits server | limit 50"
  }'

{
  "data": [
    {"value": "checkout", "hits": 472251224},
    {"value": "payments", "hits": 421683425}
  ],
  "valueFieldHints": ["hits"]
}

Swap server for any label you want to populate a picker with, for example namespace, clustered_path, span_name or response_status_code. The mandatory top-level resource_type and direction filters still apply.

Search Time Series Endpoint

Same body as the search endpoint, plus the bucketing fields. groundcover applies the bucket to your query, so do not add a bucket stage yourself.

limit applies per bucket here, not to the whole result. When the bucket is applied, a trailing | limit N is rewritten to cap rows within each time bucket. With a stats by (...) clause that means you get the top N groups in each bucket independently, so a group that ranks in the top N during some buckets and not others produces a series with gaps, and the set of series can change from bucket to bucket.

If you need a stable set of series, pick the groups first with a search call, then query the time series with those groups pinned in the query filters rather than relying on limit to choose them.

Request

Request Body

Parameter	Type	Required	Description
start	string	Yes	Range start in RFC3339 format
end	string	Yes	Range end in RFC3339 format
query	string	Conditional	gcQL query. Provide either query or pipeline. Sending neither returns 400
pipeline	object	Conditional	Structured pipeline, as an alternative to query. Same top-level filter contract applies
bucketDuration	string	Yes	Bucket width, for example "1m", "5m", "1h". Also accepts "5 minutes" style values
valueField	string	No	Which numeric column to plot, or a comma-separated list. Resolved automatically from the response when omitted
fillValue	number	No	Value used for empty buckets. Buckets are filled with null when omitted
filters	string	No	Extra gcQL filters ANDed onto the query
sources	array	No	Additional filter conditions. Prefer expressing these in query

On live queries (where end is at or near the current time) the trailing partial bucket is dropped, so the last point is always a complete bucket.

Response

A bare top-level array of series, each with its labels and its data points:

[
  {
    "velocity": [
      [1786310400, "19204.5"],
      [1786314000, "20117"],
      [1786317600, null]
    ],
    "metric": {"server": "checkout"}
  },
  {
    "velocity": [
      [1786310400, "8021"],
      [1786314000, "8115.25"],
      [1786317600, "7998"]
    ],
    "metric": {"server": "payments"}
  }
]

Field	Type	Description
velocity	array	Data points for the series, one per bucket
velocity[]	array	A [timestamp, value] pair
metric	object	The group-by labels identifying this series

Each data point contains:

• Timestamp: bucket start as a Unix timestamp in seconds

• Value: aggregated value for the bucket as a string, or null for an empty bucket when fillValue is not set

Each series object has the same velocity / metric shape as the metrics range query, but the envelope differs: that endpoint wraps its series in {"velocities": [...], "promql": "..."}, while this one returns the array directly. Parse the response body itself, not response.velocities.

Examples

Hourly inbound HTTP request count per server

This replaces charted reporting off /api/resources/v3/apis/requests. To chart a stable set of series, first pick the servers with a search call (for example, the top 20 by requests), then pin them in the time-series query instead of using a trailing | limit:

curl 'https://api.groundcover.com/api/apm-measurements/search-time-series' \
  -H 'accept: application/json' \
  -H 'authorization: Bearer <YOUR_API_KEY>' \
  -H 'content-type: application/json' \
  -H 'X-Backend-Id: <YOUR_BACKEND_ID>' \
  --data-raw '{
    "start": "2026-08-09T00:00:00.000Z",
    "end": "2026-08-10T00:00:00.000Z",
    "bucketDuration": "1h",
    "query": "resource_type:http is_inbound:true server:in(checkout, payments) | filter source:eBPF | stats by (server) sum(total_counter) as requests",
    "valueField": "requests",
    "fillValue": 0
  }'

p99 latency over time per route

This replaces charted reporting off /api/resources/v3/apis/latencies. Here the trailing | limit 10 is used deliberately for its per-bucket semantics: each 5-minute bucket independently keeps its 10 slowest routes, so the chart surfaces whichever routes were slowest at each point in time rather than a fixed set of series:

curl 'https://api.groundcover.com/api/apm-measurements/search-time-series' \
  -H 'accept: application/json' \
  -H 'authorization: Bearer <YOUR_API_KEY>' \
  -H 'content-type: application/json' \
  -H 'X-Backend-Id: <YOUR_BACKEND_ID>' \
  --data-raw '{
    "start": "2026-08-09T00:00:00.000Z",
    "end": "2026-08-10T00:00:00.000Z",
    "bucketDuration": "5m",
    "query": "resource_type:http is_inbound:true | filter source:eBPF | stats by (clustered_path) quantile(0.99, latency_seconds_quantiles) as p99 | sort by (p99 desc) | limit 10",
    "valueField": "p99"
  }'

Errors over time, replacing /api/resources/v3/apis/errors

Like the latency example, the | limit 10 keeps the 10 highest-error servers within each 15-minute bucket independently. Pin the servers with server:in(...) instead if you need a fixed set of series:

curl 'https://api.groundcover.com/api/apm-measurements/search-time-series' \
  -H 'accept: application/json' \
  -H 'authorization: Bearer <YOUR_API_KEY>' \
  -H 'content-type: application/json' \
  -H 'X-Backend-Id: <YOUR_BACKEND_ID>' \
  --data-raw '{
    "start": "2026-08-09T00:00:00.000Z",
    "end": "2026-08-10T00:00:00.000Z",
    "bucketDuration": "15m",
    "query": "resource_type:http is_inbound:true | filter source:eBPF | stats by (server) sum(error_counter) as errors | sort by (errors desc) | limit 10",
    "valueField": "errors",
    "fillValue": 0
  }'

Errors

Status	Condition
400	The query is missing a top-level resource_type filter
400	The query is missing a top-level is_inbound:true or is_outbound:true filter
400	gcQL parse error, or an invalid bucketDuration
400	The query exceeded the row cap. Narrow resource_type or the time range, or reduce group-by cardinality

Errors are returned as a JSON object with a human-readable message, a trace_id, and for some cases a stable machine-readable code.

A query missing its resource_type filter:

{"query": "is_inbound:true | stats sum(total_counter) as requests"}

{
  "message": "gcql parse error: apm search requires a top-level `resource_type` filter (EQ or IN) in the GCQL query",
  "trace_id": "93c33976207805aff1f4175a05483a98"
}

A query that grouped by too many distinct values:

{
  "message": "query exceeded the row cap; narrow your filters (resource_type / time range) or reduce group-by cardinality",
  "code": "EXCEEDED_MAX_ROWS_TO_GROUP_BY",
  "trace_id": "93c33976207805aff1f4175a05483a98"
}

Best Practices

1. Always aggregate. These endpoints read pre-aggregated measurements, but a bare filter query over a wide window still returns a lot of rows. Add a | stats stage and a | limit.

2. Disambiguate the source. Filter to one source or group by it, so eBPF and OpenTelemetry rows are not summed together.

3. Keep group-by cardinality in check. Grouping by a high-cardinality label over a wide window can hit the row cap. Check a candidate label first with | stats count_uniq(<label>).

4. Match bucketDuration to the window. A 24-hour range at 1m buckets produces 1440 points per series. Prefer 1h for daily reports and reserve fine buckets for short windows.

5. Set fillValue for counters. Counters read better as 0 in empty buckets than as gaps.

6. Don't lean on limit to pick series. On the time-series endpoint it caps rows per bucket, so it will not give you a stable global top N. Select the groups with a search call first, then pin them in the time-series query.