Ctrlk
Log in|Playground
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Search Performance

How groundcover speeds up log and trace searches, and what you can do to make a slow query fast.

Overview

Every search has to scan some amount of data. The larger the time range and the broader the filters, the more data a query touches — and the longer it takes to return.

groundcover uses several mechanisms to shrink how much data a search scans. They are not alternatives to each other — they stack, and each one fits a different situation. Some are always on, some turn on automatically based on how you query, and some you configure yourself.

Standardized attributes

Some fields are optimized out of the box, with no setup. The only requirement is that your queries actually use them.

Scope keys

env, cluster, namespace, and workload are first-class indexed fields. Including them in a query dramatically narrows what the search scans, and it's the easiest speedup available.

cluster:prod-us namespace:checkout level:error

Rule of thumb: start every query with the scope keys you can (env, cluster, namespace, workload), then add more specific filters.

Good fields to filter by

Beyond the scope keys, each data type has its own first-class fields. Reaching for these rather than a custom attribute or free text gives the search something selective to narrow on straight away.

Data type
Useful fields

Logs

level for severity — level:error, or level:in(error, fatal) for several at once. Also pod_name and trace_id.

Traces

status_code for HTTP errors — status_code>=500, or status_code>=400 for 4xx and 5xx together. Also status:error for span-level errors, plus span_type, resource, duration_seconds, trace_id.

Kubernetes events

type for severity — type:Warning. Also reason, such as reason:OOMKilling, and entity_workload.

These error fields are not interchangeable. Use level for log severity, status_code for HTTP errors on traces, and status:error for span status — each applies only to its own data type.

Trace IDs

Trace ID lookups are optimized by default, in both logs and traces, so pivoting from a log line to its trace is fast with no configuration.

groundcover also recognizes the common spellings of trace ID fields at ingestion, so the optimization works across the usual variants. See Log and Trace Correlation for the full list of recognized field names.

If your services emit trace IDs under a custom field name that isn't recognized, the best fix is to map it onto a standard name using a log pipeline rather than querying the custom field directly. That way your trace ID lookups ride the default optimization.

Favorite attributes

Out of all the string attributes you send, only a handful tend to be searched regularly. groundcover automatically identifies that frequently-used subset and keeps it in a dedicated, smaller place. Searches on those keys then run against a much smaller set of attributes instead of every attribute in your data, which makes them noticeably faster.

How it turns on. Automatically, based on usage. When the same attribute shows up in your queries repeatedly over a recent window of time, groundcover promotes it on its own. There is nothing to configure and no request to make.

The speedup applies going forward. Promotion accelerates searches over data ingested after an attribute becomes a favorite. Historical data is not re-indexed, so the benefit builds up over time rather than appearing at once.

Inverted index

An inverted index is a targeted index on one specific field that makes needle-in-a-haystack searches fast — finding the small number of records that carry one particular value.

How it works. Many high-cardinality identifiers only appear within short windows of time. An inverted index maps values of a field to the time windows they appear in. Looking a value up means first consulting the index to find where it occurs, then scanning only those windows instead of the whole time range.

Searching for a correlation ID over the last 7 days, where the ID only appears on two of those days:

When it's a good fit. The value you search for is rare across the dataset and concentrated in time rather than scattered everywhere — an order ID, a request or correlation ID, a job or upload ID, a user ID. These are the fields you reach for when stitching together the story of a single request across services.

When it's the wrong tool. Values that recur constantly. If a value matches a large or steady share of your records over time, the index can't narrow anything down and there's little to gain. Indexing something like level would not help, because every record has one.

How to enable it. Inverted indexes are configured by groundcover rather than self-served. If you have a field that fits the pattern above, contact groundcover support with the field name and the data type it applies to. Trace IDs already benefit from this by default.

A field that's worth an inverted index is usually worth having as a favorite attribute too. Since favorite attributes are promoted automatically from real query usage, this generally takes care of itself once you start searching the field regularly.

Index tiers

Index tiers let you carve out a defined slice of your data — a critical cluster, production errors, a key tenant — so that searches within that slice are consistently fast. You define the slice with a gcQL filter, and matching records are grouped together as they arrive. Querying a tier by name then touches only that group rather than the full dataset.

Reach for a tier when you repeatedly search the same well-defined subset of your data. Keep the number of tiers small: a handful of genuinely high-value slices stays easy to reason about, while carving up everything dilutes the benefit.

A record belongs to exactly one tier, so a tier holds only what tiers defined before it didn't already claim. If you expect a tier to contain every record matching its filter, check that no broader tier sits above it — see How overlap is resolved.

See Storage Management for access requirements, how to define a tier, and how to filter on one with index_tier.

Making a slow query faster

When a search is slower than you'd like, work through these in order:

  1. Add scope keys. Can you add env, cluster, namespace, or workload? This is the most common and most effective fix.

  2. Narrow the time range. Scanning 14 days costs roughly fourteen times what scanning one day costs. Start narrow and widen only if you need to.

  3. Prefer key:value over searching everything. A filter on a specific attribute is far cheaper than *:value, which has to look at every attribute. On logs, the same applies to bare free-text terms. Bare free text isn't supported on traces — reach for *:value or a field-qualified filter there instead. See Search & Filter for which syntax each section supports.

  4. Go easy on wildcards. * at the start or end of a term prevents an exact-match lookup and forces a broader scan. An exact term is significantly faster than the same term with a wildcard attached.

  5. Filter, don't exclude. Where you can, say what you want rather than what you don't. cluster:prod-us narrows the search to one cluster; -cluster:staging still has to consider everything else.

  6. Ask about an inverted index. If you're routinely looking up rare identifiers, this is the mechanism built for it.

  7. Define an index tier. If you keep coming back to the same well-defined subset of your data, make it a tier and query it with index_tier.

Example

A phrase searched across two weeks of data, scoped only by excluding one cluster:

The exclusion still leaves every other cluster in scope, and nothing narrows the search before the phrase is matched. Naming the cluster and namespace instead:

The phrase is unchanged — what changed is how much data it runs against. This is deliberately a narrower search: it asks whether this happened in prod-us and the orders namespace, rather than anywhere outside staging. That narrowing is the point. If you genuinely need the broader scope, keep the exclusion and shorten the time range instead.

Related

Last updated