Monitor YAML structure

While we strongly suggest building monitors using our Wizard or Catalog, groundcover also supports building and editing monitors directly in YAML. This page documents the current schema.

For the query language used inside monitor queries, see the gcQL Reference. For ClickHouse SQL escape hatch monitors, see SQL Based Monitors.

Top-level fields

Field	Description	Allowed values
title (required)	Human-readable name of the monitor. Shown in the Monitor List.	string
display	Display settings controlling how issues from this monitor are rendered. See Display.	object
severity	Severity reported on firing issues.	S1, S2, S3, S4
measurementType	Type of measurement the monitor represents.	state, event
model (required)	Queries, reducers, and thresholds that define what the monitor evaluates. See Model.	object
labels	Static or templated labels attached to the issue. Values can reference query results via {{ $values.<threshold_name>.Labels.<key> }}.	map<string,string>
annotations	Annotations attached to the alert, often used to wire monitors into workflows.	map<string,string>
category	Free-form category used for grouping in the Monitor List.	string
executionErrorState	State the monitor enters when query execution fails. Set in the wizard as Treat Evaluation Errors As: OK (Normal) treats the failure as healthy; Error (Error) sets a distinct Error status; Alerting (Firing) creates an Issue. Defaults to OK.	OK, Error, Alerting
noDataState	State the monitor enters when the query returns no rows. Set in the wizard as Treat No Data As: OK (Normal) treats it as healthy; NoData (No Data) sets the No Data status — visible on the monitor but not an Issue; Alerting (Firing) creates an Issue. Defaults to NoData. To notify on no data, see Alerting on No Data.	OK, NoData, Alerting
evaluationInterval	Evaluation cadence and pending window. See EvaluationInterval.	object
notificationSettings	How alerts are delivered. See NotificationSettings.	object
autoResolve	When true, issues from this monitor automatically resolve once the condition no longer holds. Optional.	boolean
isPaused	When true, the monitor is defined but not evaluated.	boolean

display

Field	Description
header	Template for the issue header. Supports alert label substitution, e.g. "gRPC API Error {{ labels.status_code }}".
description	Template for the issue description. Supports the same substitutions as header.
resourceHeaderLabels	List of labels identifying the resource the issue relates to. Rendered as the secondary header across Issues tables. Example: ["span_name", "role"].
contextHeaderLabels	List of labels identifying the location of the issue. Rendered as a subset of the issue's labels. Example: ["cluster", "namespace", "workload"].
templateLanguage	Template engine for header and description. Set to jinja2 to opt into Jinja2 syntax (enablesblocks and filters). Omit for the default Go-template syntax.

model

Field	Description
queries (required)	One or more queries that produce the data the monitor evaluates. See model.queries.
reducers	Aggregations applied on top of queries before thresholds run. See model.reducers.
thresholds	Conditions evaluated against a query or reducer output. See model.thresholds.

model.queries

Each query describes one data source and one expression. The combination of dataType and the query body determines which engine runs the query.

Field	Description
name (required)	Identifier used by reducers and thresholds to reference this query's output.
dataType	Data source for a gcQL query. One of logs, traces, events. Omit dataType for MetricsQL queries — see the field compatibility note.
expression	The query itself. The language depends on dataType: gcQL for logs, traces, events. See the gcQL Reference. MetricsQL when dataType is omitted. MetricsQL is VictoriaMetrics' query language and is backwards-compatible with PromQL, with extra functions (topk_last, rollup_rate, etc.). It does not use the pipe (|) operator; combine operations with nested functions or arithmetic.
datasourceType	Required for MetricsQL queries. Set to prometheus (the name refers to the Prometheus-compatible API served by the metrics backend).
queryType	Required for MetricsQL queries. Use instant.
filters	Optional standalone gcQL filter expression applied to the query. For most monitors, put filters directly inside expression instead.
relativeTimerange	Time window relative to evaluation time. Object with from and optional to durations (e.g. from: 5m).
instantRollup	Bucket size for gcQL queries (logs/traces/events), e.g. 1 minutes, 5 minutes. Controls the time granularity the monitor evaluates over.
rollup	Required for MetricsQL queries (whenever datasourceType: prometheus is set). Server-side rollup. Object with function (avg, max, min, sum, count, stddev, stdvar, last) and time (duration). Not interchangeable with instantRollup.
evaluationDelay	Optional. Number of seconds (0–3600) to shift the evaluation window into the past, so the monitor evaluates a window ending evaluationDelay seconds before "now" (window length unchanged). Useful for sources that backfill recent data (AWS CloudWatch, GCP). Omit or set to 0 for no delay. See Evaluation delay.

Query-engine field compatibility

The query fields are tied to the query engine, and several of them must appear together. Mixing them across engines is the most common cause of a rejected monitor.

Engine	Required fields	Must not be set
gcQL (dataType = logs, traces, events)	dataType, expression, instantRollup	datasourceType, queryType, rollup
MetricsQL (no dataType)	expression, datasourceType: prometheus, queryType: instant, rollup	dataType, instantRollup

Key rules the backend enforces:

• datasourceType: prometheus requires rollup (object with function + time). Omitting it is rejected with "rollup is required for prometheus datasource type".

• For MetricsQL, omit dataType — datasourceType: prometheus together with the MetricsQL expression and rollup is all you need.

• rollup (MetricsQL) and instantRollup (gcQL) are not interchangeable — pick the one for your engine.

model.reducers

Reducers aggregate a query's output into a single value (or per-group value) before thresholds run. This is how you turn a timeseries into a single number to compare against a threshold.

Field	Description
name (required)	Identifier used by thresholds.
inputName	Name of the query (or another reducer) to read from. Required unless type: math.
type (required)	One of last, min, max, mean, sum, count, math.
expression	Required when type: math. An arithmetic expression over reducer outputs, e.g. $errors / $total * 100.
relativeTimerange	Optional time window specific to this reducer.

model.thresholds

Thresholds are the final condition that determines whether the monitor fires.

Field	Description
name (required)	Identifier for this threshold.
inputName (required)	Name of the query or reducer this threshold evaluates.
operator (required)	One of gt, lt, gte, lte, eq, neq, within_range, outside_range, within_range_included, outside_range_included.
values (required)	Array of numbers. One value for comparison operators; two values for within_range / outside_range and their inclusive variants.
relativeTimerange	Optional time window specific to this threshold.
customResolveThreshold	Optional hysteresis recovery condition. When set, a firing issue resolves only once this separate predicate is met, which reduces flapping for values that hover around the firing threshold. Object with operator and values. The operator must be the directional opposite of the threshold's operator (gt↔lt, within_range↔outside_range) and is only supported for gt, lt, within_range, and outside_range. The resolve values must not overlap the firing range, e.g. fire when gt 100 and resolve when lt 80.

evaluationInterval

Field	Description
interval	How often the monitor is evaluated, e.g. 1m, 5m.
pendingFor	Duration the threshold must hold before the issue transitions from Pending to Alerting. Use 0s to fire immediately.

notificationSettings

Field	Description
method	How notifications are delivered. notificationRoutes uses the matching routes defined in Notification Routes; connectedApps sends directly to the apps listed in connectedApps, bypassing routes; noNotifications suppresses all notifications for this monitor.
connectedApps	List of Destination IDs. Used with method: connectedApps. (The YAML field name connectedApps is retained for backward compatibility; in the UI these are called Destinations.)
connectedAppParams	Per-destination delivery options keyed by Destination ID. For Slack App connectors, use channels to set the target channels — see Slack. Example: { "<slack-app-id>": { "channels": [{ "id": "C123456", "name": "#alerts" }] } }. Used with method: connectedApps.
renotificationInterval	Duration between repeat notifications while the issue remains firing, e.g. 4h.
disableRenotification	When true, suppresses repeat notifications.
statusFilters	List of issue statuses that trigger notifications. Allowed values: Alerting, Resolved. Used with method: connectedApps. Note: the monitor wizard labels these as Firing and Resolved — Firing in the UI corresponds to Alerting in YAML.

Examples

Traces monitor (gcQL)

Fires when gRPC traces return a non-zero status code.

title: gRPC API Errors Monitor
display:
  header: gRPC API Error {{ labels.status_code }}
  description: |-
    This monitor detects gRPC API errors by identifying responses with a status code indicating failure.
    Cluster: {{ labels.cluster }}
    Namespace: {{ labels.namespace }}
    Workload: {{ labels.workload }}
    Span: {{ labels.span_name }}
  resourceHeaderLabels:
    - span_name
    - role
  contextHeaderLabels:
    - env
    - cluster
    - namespace
    - workload
  templateLanguage: jinja2
severity: S3
measurementType: event
model:
  queries:
    - name: threshold_input_query
      dataType: traces
      expression: >
        span_type:grpc status_code:!=0 status:error source:ebpf
        | stats by (env, cluster, namespace, workload, status_code, span_name, role) count() errors_total
      instantRollup: 1 minutes
  thresholds:
    - name: threshold_1
      inputName: threshold_input_query
      operator: gt
      values:
        - 0
executionErrorState: OK
noDataState: OK
evaluationInterval:
  interval: 1m
  pendingFor: 0s

Logs monitor (gcQL)

Fires when sensor logs contain panic or fatal errors.

title: Sensor Panic / Fatal Errors
display:
  header: Sensor Panic or Fatal Errors
  description: |-
    Detects panic or fatal errors in sensor logs.
    Cluster: {{ labels.cluster }}
    Namespace: {{ labels.namespace }}
    Pod: {{ labels.pod }}
  contextHeaderLabels:
    - pod
    - workload
    - cluster
    - env
    - namespace
severity: S2
measurementType: event
model:
  queries:
    - name: threshold_input_query
      dataType: logs
      expression: >
        container:sensor level:in(panic, fatal)
        | stats by (env, cluster, namespace, workload, pod) count() count_all_result
      instantRollup: 5 minutes
  thresholds:
    - name: threshold_1
      inputName: threshold_input_query
      operator: gt
      values:
        - 0
executionErrorState: OK
noDataState: OK
evaluationInterval:
  interval: 5m
  pendingFor: 0s
notificationSettings:
  renotificationInterval: 4h

Monitor that delivers directly to Slack channels

Bypasses notification routes and sends issues from this monitor straight to two Slack channels via the Slack Destination. Replace the slack-app-id placeholder with the ID of your Slack App Destination, and the channel id values with the Slack channel IDs (e.g., C0123456789) you want to deliver to.

title: Checkout Service 5xx Spike
display:
  header: Checkout 5xx Spike in {{ labels.cluster }}
  description: |-
    HTTP 5xx responses from the checkout service are above threshold.
    Cluster: {{ labels.cluster }}
    Namespace: {{ labels.namespace }}
    Workload: {{ labels.workload }}
severity: S2
measurementType: event
model:
  queries:
    - name: threshold_input_query
      dataType: traces
      expression: >
        workload:checkout status_code:>=500
        | stats by (env, cluster, namespace, workload) count() errors_total
      instantRollup: 1 minutes
  thresholds:
    - name: threshold_1
      inputName: threshold_input_query
      operator: gt
      values:
        - 10
executionErrorState: OK
noDataState: OK
evaluationInterval:
  interval: 1m
  pendingFor: 0s
notificationSettings:
  method: connectedApps
  connectedApps:
    - <slack-app-id>
  connectedAppParams:
    <slack-app-id>:
      channels:
        - id: C0123456789
          name: "#checkout-alerts"
        - id: C0987654321
          name: "#oncall-prod"
  statusFilters:
    - Alerting
    - Resolved
  renotificationInterval: 1h

Channel id is the canonical Slack channel ID (it stays stable if the channel is renamed); name is optional and only used for display. You can find the channel ID in Slack via Channel name → View channel details → About (the ID is shown at the bottom).

Metrics monitor (MetricsQL)

Fires when a Kubernetes pod is in CrashLoopBackOff for more than 5 minutes. Uses a reducer to collapse the timeseries before the threshold runs.

Metrics queries use MetricsQL (PromQL-compatible). kube-state-metrics names are prefixed with groundcover_, and the node identity label is node_name (not node).

title: K8s Pod Crash Looping Monitor
display:
  header: K8s Pod Crash Looping
  description: Kubernetes pod has been in CrashLoopBackOff for more than 5 minutes.
  resourceHeaderLabels:
    - workload
  contextHeaderLabels:
    - env
    - cluster
    - namespace
severity: S2
measurementType: state
model:
  queries:
    - name: crash_looping_query
      expression: >
        avg_over_time(
          avg by (env, cluster, namespace, workload) (
            groundcover_kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}
          )[5m]
        )
      datasourceType: prometheus
      queryType: instant
      rollup:
        function: avg
        time: 5m
  reducers:
    - name: crash_looping_mean
      inputName: crash_looping_query
      type: mean
  thresholds:
    - name: threshold_1
      inputName: crash_looping_mean
      operator: gt
      values:
        - 0
executionErrorState: OK
noDataState: OK
evaluationInterval:
  interval: 1m
  pendingFor: 5m

ClickHouse SQL monitor

For advanced cases that need joins, CTEs, or comparisons across time windows, you can drop to ClickHouse SQL. See SQL Based Monitors for details and examples.