Monitor YAML structure

While we strongly suggest building monitors using our Wizard or Catalog, groundcover also supports building and editing monitors directly in YAML. This page documents the current schema.

For the query language used inside monitor queries, see the gcQL Reference. For ClickHouse SQL escape hatch monitors, see SQL Based Monitors.

Top-level fields

Field	Description	Allowed values
title (required)	Human-readable name of the monitor. Shown in the Monitor List.	string
display	Display settings controlling how issues from this monitor are rendered. See Display.	object
severity	Severity reported on firing issues.	S1, S2, S3, S4
measurementType	Controls how issues are visualized. state renders as a line chart; event renders as a bar chart counting events.	state, event
model (required)	Queries, reducers, and thresholds that define what the monitor evaluates. See Model.	object
labels	Static or templated labels attached to the issue. Values can reference query results via {{ $values.<threshold_name>.Labels.<key> }}.	map<string,string>
annotations	Annotations attached to the alert, often used to wire monitors into workflows.	map<string,string>
category	Free-form category used for grouping in the Monitor List.	string
executionErrorState	State when query execution fails. OK suppresses; Error raises an error state; Alerting fires an issue.	OK, Error, Alerting
noDataState	State when the query returns no rows. OK stays normal; NoData enters a No Data state; Alerting fires.	OK, NoData, Alerting
evaluationInterval	Evaluation cadence and pending window. See EvaluationInterval.	object
notificationSettings	How alerts are delivered. See NotificationSettings.	object
isPaused	When true, the monitor is defined but not evaluated.	boolean

display

Field	Description
header	Template for the issue header. Supports alert label substitution, e.g. "gRPC API Error {{ labels.status_code }}".
description	Template for the issue description. Supports the same substitutions as header.
resourceHeaderLabels	List of labels identifying the resource the issue relates to. Rendered as the secondary header across Issues tables. Example: ["span_name", "role"].
contextHeaderLabels	List of labels identifying the location of the issue. Rendered as a subset of the issue's labels. Example: ["cluster", "namespace", "workload"].
templateLanguage	Template engine for header and description. Set to jinja2 to opt into Jinja2 syntax (enablesblocks and filters). Omit for the default Go-template syntax.

model

Field	Description
queries (required)	One or more queries that produce the data the monitor evaluates. See model.queries.
reducers	Aggregations applied on top of queries before thresholds run. See model.reducers.
thresholds	Conditions evaluated against a query or reducer output. See model.thresholds.

model.queries

Each query describes one data source and one expression. The combination of dataType and the query body determines which engine runs the query.

Field	Description
name (required)	Identifier used by reducers and thresholds to reference this query's output.
dataType	Data source to query. One of logs, traces, events, rum, entities, issues, metrics, infra.
expression	The query itself. The language depends on dataType: gcQL for logs, traces, events, rum, entities, issues. See the gcQL Reference. MetricsQL for metrics and infra. MetricsQL is VictoriaMetrics' query language and is backwards-compatible with PromQL, with extra functions (topk_last, rollup_rate, etc.). It does not use the pipe (|) operator; combine operations with nested functions or arithmetic.
datasourceType	Required for MetricsQL queries. Set to prometheus (the name refers to the Prometheus-compatible API served by the metrics backend).
queryType	For MetricsQL queries, use instant.
filters	Optional standalone gcQL filter expression, used alongside sqlPipeline. For modern monitors, filters belong directly inside expression.
relativeTimerange	Time window relative to evaluation time. Object with from and optional to durations (e.g. from: 5m).
instantRollup	Bucket size for gcQL queries (logs/traces/events/rum/entities/issues), e.g. 1 minutes, 5 minutes. Controls the time granularity the monitor evaluates over.
rollup	Server-side rollup for MetricsQL queries (metrics/infra). Object with function (avg, max, min, sum, count, stddev, stdvar, last) and time (duration).
sqlPipeline	Legacy. Structured SQL pipeline object. Still accepted for backwards compatibility but new monitors should use expression with gcQL.

rollup vs instantRollup is a common point of confusion. They are two different fields for two different query engines: use rollup (object with function and time) for MetricsQL queries, and instantRollup (a duration string) for gcQL queries. They are not interchangeable.

model.reducers

Reducers aggregate a query's output into a single value (or per-group value) before thresholds run. This is how you turn a timeseries into a single number to compare against a threshold.

Field	Description
name (required)	Identifier used by thresholds.
inputName	Name of the query (or another reducer) to read from. Required unless type: math.
type (required)	One of last, min, max, mean, sum, count, math.
expression	Required when type: math. An arithmetic expression over reducer outputs, e.g. $errors / $total * 100.
relativeTimerange	Optional time window specific to this reducer.

model.thresholds

Thresholds are the final condition that determines whether the monitor fires.

Field	Description
name (required)	Identifier for this threshold.
inputName (required)	Name of the query or reducer this threshold evaluates.
operator (required)	One of gt, lt, gte, lte, eq, neq, within_range, outside_range, within_range_included, outside_range_included.
values (required)	Array of numbers. One value for comparison operators; two values for within_range / outside_range and their inclusive variants.
relativeTimerange	Optional time window specific to this threshold.

evaluationInterval

Field	Description
interval	How often the monitor is evaluated, e.g. 1m, 5m.
pendingFor	Duration the threshold must hold before the issue transitions from Pending to Alerting. Use 0s to fire immediately.

notificationSettings

Field	Description
method	How notifications are delivered. notificationRoutes uses the matching routes defined in Notification Routes; connectedApps sends directly to the apps listed in connectedApps, bypassing routes; noNotifications suppresses all notifications for this monitor.
connectedApps	List of connected app IDs. Used with method: connectedApps.
renotificationInterval	Duration between repeat notifications while the issue remains firing, e.g. 4h.
disableRenotification	When true, suppresses repeat notifications.
statusFilters	List of issue statuses that trigger notifications. Used with method: connectedApps.

Examples

Traces monitor (gcQL)

Fires when gRPC traces return a non-zero status code.

title: gRPC API Errors Monitor
display:
  header: gRPC API Error {{ labels.status_code }}
  description: |-
    This monitor detects gRPC API errors by identifying responses with a status code indicating failure.
    Cluster: {{ labels.cluster }}
    Namespace: {{ labels.namespace }}
    Workload: {{ labels.workload }}
    Span: {{ labels.span_name }}
  resourceHeaderLabels:
    - span_name
    - role
  contextHeaderLabels:
    - env
    - cluster
    - namespace
    - workload
  templateLanguage: jinja2
severity: S3
measurementType: event
model:
  queries:
    - name: threshold_input_query
      dataType: traces
      expression: >
        span_type:grpc status_code:!=0 status:error source:ebpf
        | stats by (env, cluster, namespace, workload, status_code, span_name, role) count() errors_total
      instantRollup: 1 minutes
  thresholds:
    - name: threshold_1
      inputName: threshold_input_query
      operator: gt
      values:
        - 0
executionErrorState: OK
noDataState: OK
evaluationInterval:
  interval: 1m
  pendingFor: 0s

Logs monitor (gcQL)

Fires when sensor logs contain panic or fatal errors.

title: Sensor Panic / Fatal Errors
display:
  header: Sensor Panic or Fatal Errors
  description: |-
    Detects panic or fatal errors in sensor logs.
    Cluster: {{ labels.cluster }}
    Namespace: {{ labels.namespace }}
    Pod: {{ labels.pod }}
  contextHeaderLabels:
    - pod
    - workload
    - cluster
    - env
    - namespace
severity: S2
measurementType: event
model:
  queries:
    - name: threshold_input_query
      dataType: logs
      expression: >
        container:sensor level:in(panic, fatal)
        | stats by (env, cluster, namespace, workload, pod) count() count_all_result
      instantRollup: 5 minutes
  thresholds:
    - name: threshold_1
      inputName: threshold_input_query
      operator: gt
      values:
        - 0
executionErrorState: OK
noDataState: OK
evaluationInterval:
  interval: 5m
  pendingFor: 0s
notificationSettings:
  renotificationInterval: 4h

Metrics monitor (MetricsQL)

Fires when a Kubernetes pod is in CrashLoopBackOff for more than 5 minutes. Uses a reducer to collapse the timeseries before the threshold runs.

Metrics queries use MetricsQL (PromQL-compatible). kube-state-metrics names are prefixed with groundcover_, and the node identity label is node_name (not node).

title: K8s Pod Crash Looping Monitor
display:
  header: K8s Pod Crash Looping
  description: Kubernetes pod has been in CrashLoopBackOff for more than 5 minutes.
  resourceHeaderLabels:
    - workload
  contextHeaderLabels:
    - env
    - cluster
    - namespace
severity: S2
measurementType: state
model:
  queries:
    - name: crash_looping_query
      dataType: metrics
      expression: >
        avg_over_time(
          avg by (env, cluster, namespace, workload) (
            groundcover_kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}
          )[5m]
        )
      datasourceType: prometheus
      queryType: instant
      rollup:
        function: avg
        time: 5m
  reducers:
    - name: crash_looping_mean
      inputName: crash_looping_query
      type: mean
  thresholds:
    - name: threshold_1
      inputName: crash_looping_mean
      operator: gt
      values:
        - 0
executionErrorState: OK
noDataState: OK
evaluationInterval:
  interval: 1m
  pendingFor: 5m

ClickHouse SQL monitor

For advanced cases that need joins, CTEs, or comparisons across time windows, you can drop to ClickHouse SQL. See SQL Based Monitors for details and examples.