1 of 1

Monitor YAML structure

While we strongly suggest building monitors using our Wizard or Catalog, groundcover supports building and editing your Monitors using YAML. If you choose to do so, the following will provide you the necessary definitions.

Monitor fields explained

In this section, you'll find a breakdown of the key fields used to define and configure Monitors within the groundcover platform. Each field plays a critical role in how a Monitor behaves, what data it tracks, and how it responds to specific conditions. Understanding these fields will help you set up effective Monitors to track performance, detect issues, and provide timely alerts.

Below is a detailed explanation of each field, along with examples to illustrate their usage, ensuring your team can manage and respond to incidents efficiently.

Field

Explanation

Example

Monitor YAML Examples

Traces Based Monitors

MySQL Query Errors Monitor

gRPC API Errors Monitor

Log Based Monitors

High Error Log Rate Monitor

Monitor YAML structure

Monitor fields explained

Below is a detailed explanation of each field, along with examples to illustrate their usage, ensuring your team can manage and respond to incidents efficiently.

Field

Explanation

Example

Monitor YAML Examples

Traces Based Monitors

MySQL Query Errors Monitor

gRPC API Errors Monitor

Log Based Monitors

High Error Log Rate Monitor

ContextHeaderLabels

A list of contextual labels that help you identify the location of the issue. This appears as a subset of the Issue’s labels, and is displayed on all Issues tables across the platform.

["cluster", "namespace", "pod_name"]

Labels

A set of pre-defined labels that were set to Issues related to the selected Monitor. Labels can be static, or dynamic using a Monitor's query results.

team: sre_team

ExecutionErrorState

Defines the actions that take place when a Monitor encounters query execution errors.

Valid options are Alerting, OK and Error.

When Alerting is set, query execution errors will result in a firing issue.

NoDataState

This defines what happens when queries in the Monitor return empty datasets.

Valid options are: NoData , Alerting, OK

When NoData is set, monitor instance's state will be No Data.

Interval

Defines how frequently the Monitor evaluates the conditions. Common intervals could be 1m, 5m, etc.

Trigger

Defines the condition under which the Monitor fires. This is the definition of threshold for the Monitor, with op - operator and value .

op: gt, value: 5

Model

Describes the queries, thresholds and data processing of the Monitor. It can have the following fields:

Queries: List of one or more queries to run, this can be either SQL over ClickHouse, PromQL over VictoriaMetrics, SqlPipeline. Each query will have a name for reference in the monitor.
Thresholds: This is the threshold of your Monitor, a threshold has a name, inputName for data input, operator one of gt , lt ,

measurementType

Describe how will we present issues of this Monitor. Some Monitors count events, and some a state. And we will display them differently in our dashboards.

state - Will present issues in line chart.
event - Will present issues in bar chart, counting events.

s1 for Critical

s2 for High

s3 for Medium

s4 for Low

title: MySQL Query Errors Monitor
display:
  header: MySQL Error {{ alert.labels.statusCode }}
  description: This monitor detects MySQL Query errors.
  resourceHeaderLabels:
    - span_name
    - role
  contextHeaderLabels:
    - cluster
    - namespace
    - workload
severity: S3
measurementType: event
model:
  queries:
    - name: threshold_input_query
      dataType: traces
      sqlPipeline:
        selectors:
          - key: _time
            origin: root
            type: string
            processors:
              - op: toStartOfInterval
                args:
                  - 1 minutes
            alias: bucket_timestamp
          - key: statusCode
            origin: root
            type: string
            alias: statusCode
          - key: span_name
            origin: root
            type: string
            alias: span_name
          - key: cluster
            origin: root
            type: string
            alias: cluster
          - key: namespace
            origin: root
            type: string
            alias: namespace
          - key: role
            origin: root
            type: string
            alias: role
          - key: workload
            origin: root
            type: string
            alias: workload
          - key: "*"
            origin: root
            type: string
            processors:
              - op: count
            alias: logs_total
        groupBy:
          - key: _time
            origin: root
            type: string
            processors:
              - op: toStartOfInterval
                args:
                  - 1 minutes
          - key: statusCode
            origin: root
            type: string
            alias: statusCode
          - key: span_name
            origin: root
            type: string
            alias: span_name
          - key: cluster
            origin: root
            type: string
            alias: cluster
          - key: namespace
            origin: root
            type: string
            alias: namespace
          - key: role
            origin: root
            type: string
            alias: role
          - key: workload
            origin: root
            type: string
            alias: workload
        orderBy:
          - selector:
              key: bucket_timestamp
              origin: root
              type: string
            direction: ASC
        limit:
        filters:
          operator: and
          conditions:
            - filters:
                - op: match
                  value: mysql
              key: eventType
              origin: root
              type: string
            - filters:
                - op: match
                  value: error
              key: status
              origin: root
              type: string
            - filters:
                - op: match
                  value: eBPF
              key: source
              origin: root
              type: string
      instantRollup: 1 minutes
  thresholds:
    - name: threshold_1
      inputName: threshold_input_query
      operator: gt
      values:
        - 0
executionErrorState: OK
noDataState: OK
evaluationInterval:
  interval: 1m
  pendingFor: 0s
labels:
  team: infra

title: gRPC API Errors Monitor
display:
  header: gRPC API Error {{ alert.labels.statusCode }}
  description: This monitor detects gRPC API errors by identifying responses with a non-zero status code.
  resourceHeaderLabels:
    - span_name
    - role
  contextHeaderLabels:
    - cluster
    - namespace
    - workload
severity: S3
measurementType: event
model:
  queries:
    - name: threshold_input_query
      dataType: traces
      sqlPipeline:
        selectors:
          - key: _time
            origin: root
            type: string
            processors:
              - op: toStartOfInterval
                args:
                  - 1 minutes
            alias: bucket_timestamp
          - key: statusCode
            origin: root
            type: string
            alias: statusCode
          - key: span_name
            origin: root
            type: string
            alias: span_name
          - key: cluster
            origin: root
            type: string
            alias: cluster
          - key: namespace
            origin: root
            type: string
            alias: namespace
          - key: role
            origin: root
            type: string
            alias: role
          - key: workload
            origin: root
            type: string
            alias: workload
          - key: "*"
            origin: root
            type: string
            processors:
              - op: count
            alias: logs_total
        groupBy:
          - key: _time
            origin: root
            type: string
            processors:
              - op: toStartOfInterval
                args:
                  - 1 minutes
          - key: statusCode
            origin: root
            type: string
            alias: statusCode
          - key: span_name
            origin: root
            type: string
            alias: span_name
          - key: cluster
            origin: root
            type: string
            alias: cluster
          - key: namespace
            origin: root
            type: string
            alias: namespace
          - key: role
            origin: root
            type: string
            alias: role
          - key: workload
            origin: root
            type: string
            alias: workload
        orderBy:
          - selector:
              key: bucket_timestamp
              origin: root
              type: string
            direction: ASC
        limit:
        filters:
          operator: and
          conditions:
            - filters:
                - op: match
                  value: grpc
              key: eventType
              origin: root
              type: string
            - filters:
                - op: ne
                  value: "0"
              key: statusCode
              origin: root
              type: string
            - filters:
                - op: match
                  value: error
              key: status
              origin: root
              type: string
            - filters:
                - op: match
                  value: eBPF
              key: source
              origin: root
              type: string
      instantRollup: 1 minutes
  thresholds:
    - name: threshold_1
      inputName: threshold_input_query
      operator: gt
      values:
        - 0
executionErrorState: OK
noDataState: OK
evaluationInterval:
  interval: 1m
  pendingFor: 0s

title: High Error Log Rate Monitor
severity: S4
display:
  header: High Log Error Rate
  description: This monitor will trigger an alert when we have a rate of error logs.
  resourceHeaderLabels:
    - workload
  contextHeaderLabels:
    - cluster
    - namespace
evaluationInterval:
  interval: 1m
  pendingFor: 0s
model:
  queries:
    - name: threshold_input_query
      dataType: logs
      sqlPipeline:
        selectors:
          - key: _time
            origin: root
            type: string
            processors:
              - op: toStartOfInterval
                args:
                  - 1 minutes
            alias: bucket_timestamp
          - key: workload
            origin: root
            type: string
            alias: workload
          - key: namespace
            origin: root
            type: string
            alias: namespace
          - key: cluster
            origin: root
            type: string
            alias: cluster
          - key: "*"
            origin: root
            type: string
            processors:
              - op: count
            alias: logs_total
        groupBy:
          - key: _time
            origin: root
            type: string
            processors:
              - op: toStartOfInterval
                args:
                  - 1 minutes
          - key: workload
            origin: root
            type: string
            alias: workload
          - key: namespace
            origin: root
            type: string
            alias: namespace
          - key: cluster
            origin: root
            type: string
            alias: cluster
        orderBy:
          - selector:
              key: bucket_timestamp
              origin: root
              type: string
            direction: ASC
        limit:
        filters:
          conditions:
            - filters:
                - op: match
                  value: error
              key: level
              origin: root
              type: string
          operator: and
      instantRollup: 1 minutes
  thresholds:
    - name: threshold_1
      inputName: threshold_input_query
      operator: gt
      values:
        - 150
noDataState: OK
measurementType: event

title: MySQL Query Errors Monitor
display:
  header: MySQL Error {{ alert.labels.statusCode }}
  description: This monitor detects MySQL Query errors.
  resourceHeaderLabels:
    - span_name
    - role
  contextHeaderLabels:
    - cluster
    - namespace
    - workload
severity: S3
measurementType: event
model:
  queries:
    - name: threshold_input_query
      dataType: traces
      sqlPipeline:
        selectors:
          - key: _time
            origin: root
            type: string
            processors:
              - op: toStartOfInterval
                args:
                  - 1 minutes
            alias: bucket_timestamp
          - key: statusCode
            origin: root
            type: string
            alias: statusCode
          - key: span_name
            origin: root
            type: string
            alias: span_name
          - key: cluster
            origin: root
            type: string
            alias: cluster
          - key: namespace
            origin: root
            type: string
            alias: namespace
          - key: role
            origin: root
            type: string
            alias: role
          - key: workload
            origin: root
            type: string
            alias: workload
          - key: "*"
            origin: root
            type: string
            processors:
              - op: count
            alias: logs_total
        groupBy:
          - key: _time
            origin: root
            type: string
            processors:
              - op: toStartOfInterval
                args:
                  - 1 minutes
          - key: statusCode
            origin: root
            type: string
            alias: statusCode
          - key: span_name
            origin: root
            type: string
            alias: span_name
          - key: cluster
            origin: root
            type: string
            alias: cluster
          - key: namespace
            origin: root
            type: string
            alias: namespace
          - key: role
            origin: root
            type: string
            alias: role
          - key: workload
            origin: root
            type: string
            alias: workload
        orderBy:
          - selector:
              key: bucket_timestamp
              origin: root
              type: string
            direction: ASC
        limit:
        filters:
          operator: and
          conditions:
            - filters:
                - op: match
                  value: mysql
              key: eventType
              origin: root
              type: string
            - filters:
                - op: match
                  value: error
              key: status
              origin: root
              type: string
            - filters:
                - op: match
                  value: eBPF
              key: source
              origin: root
              type: string
      instantRollup: 1 minutes
  thresholds:
    - name: threshold_1
      inputName: threshold_input_query
      operator: gt
      values:
        - 0
executionErrorState: OK
noDataState: OK
evaluationInterval:
  interval: 1m
  pendingFor: 0s
labels:
  team: infra

title: gRPC API Errors Monitor
display:
  header: gRPC API Error {{ alert.labels.statusCode }}
  description: This monitor detects gRPC API errors by identifying responses with a non-zero status code.
  resourceHeaderLabels:
    - span_name
    - role
  contextHeaderLabels:
    - cluster
    - namespace
    - workload
severity: S3
measurementType: event
model:
  queries:
    - name: threshold_input_query
      dataType: traces
      sqlPipeline:
        selectors:
          - key: _time
            origin: root
            type: string
            processors:
              - op: toStartOfInterval
                args:
                  - 1 minutes
            alias: bucket_timestamp
          - key: statusCode
            origin: root
            type: string
            alias: statusCode
          - key: span_name
            origin: root
            type: string
            alias: span_name
          - key: cluster
            origin: root
            type: string
            alias: cluster
          - key: namespace
            origin: root
            type: string
            alias: namespace
          - key: role
            origin: root
            type: string
            alias: role
          - key: workload
            origin: root
            type: string
            alias: workload
          - key: "*"
            origin: root
            type: string
            processors:
              - op: count
            alias: logs_total
        groupBy:
          - key: _time
            origin: root
            type: string
            processors:
              - op: toStartOfInterval
                args:
                  - 1 minutes
          - key: statusCode
            origin: root
            type: string
            alias: statusCode
          - key: span_name
            origin: root
            type: string
            alias: span_name
          - key: cluster
            origin: root
            type: string
            alias: cluster
          - key: namespace
            origin: root
            type: string
            alias: namespace
          - key: role
            origin: root
            type: string
            alias: role
          - key: workload
            origin: root
            type: string
            alias: workload
        orderBy:
          - selector:
              key: bucket_timestamp
              origin: root
              type: string
            direction: ASC
        limit:
        filters:
          operator: and
          conditions:
            - filters:
                - op: match
                  value: grpc
              key: eventType
              origin: root
              type: string
            - filters:
                - op: ne
                  value: "0"
              key: statusCode
              origin: root
              type: string
            - filters:
                - op: match
                  value: error
              key: status
              origin: root
              type: string
            - filters:
                - op: match
                  value: eBPF
              key: source
              origin: root
              type: string
      instantRollup: 1 minutes
  thresholds:
    - name: threshold_1
      inputName: threshold_input_query
      operator: gt
      values:
        - 0
executionErrorState: OK
noDataState: OK
evaluationInterval:
  interval: 1m
  pendingFor: 0s

title: High Error Log Rate Monitor
severity: S4
display:
  header: High Log Error Rate
  description: This monitor will trigger an alert when we have a rate of error logs.
  resourceHeaderLabels:
    - workload
  contextHeaderLabels:
    - cluster
    - namespace
evaluationInterval:
  interval: 1m
  pendingFor: 0s
model:
  queries:
    - name: threshold_input_query
      dataType: logs
      sqlPipeline:
        selectors:
          - key: _time
            origin: root
            type: string
            processors:
              - op: toStartOfInterval
                args:
                  - 1 minutes
            alias: bucket_timestamp
          - key: workload
            origin: root
            type: string
            alias: workload
          - key: namespace
            origin: root
            type: string
            alias: namespace
          - key: cluster
            origin: root
            type: string
            alias: cluster
          - key: "*"
            origin: root
            type: string
            processors:
              - op: count
            alias: logs_total
        groupBy:
          - key: _time
            origin: root
            type: string
            processors:
              - op: toStartOfInterval
                args:
                  - 1 minutes
          - key: workload
            origin: root
            type: string
            alias: workload
          - key: namespace
            origin: root
            type: string
            alias: namespace
          - key: cluster
            origin: root
            type: string
            alias: cluster
        orderBy:
          - selector:
              key: bucket_timestamp
              origin: root
              type: string
            direction: ASC
        limit:
        filters:
          conditions:
            - filters:
                - op: match
                  value: error
              key: level
              origin: root
              type: string
          operator: and
      instantRollup: 1 minutes
  thresholds:
    - name: threshold_1
      inputName: threshold_input_query
      operator: gt
      values:
        - 150
noDataState: OK
measurementType: event

When Alerting is set, monitor instance's state will be Pending and then will change to Alerting once the pending period of the monitor ends.

Monitor YAML structure

hashtagMonitor fields explained

hashtagMonitor YAML Examples

hashtagTraces Based Monitors

hashtagMySQL Query Errors Monitor

hashtaggRPC API Errors Monitor

hashtagLog Based Monitors

hashtagHigh Error Log Rate Monitor

Monitor YAML structure

hashtagMonitor fields explained

hashtagMonitor YAML Examples

hashtagTraces Based Monitors

hashtagMySQL Query Errors Monitor

hashtaggRPC API Errors Monitor

hashtagLog Based Monitors

hashtagHigh Error Log Rate Monitor

Monitor fields explained

Monitor YAML Examples

Traces Based Monitors

MySQL Query Errors Monitor

gRPC API Errors Monitor

Log Based Monitors

High Error Log Rate Monitor

Monitor fields explained

Monitor YAML Examples

Traces Based Monitors

MySQL Query Errors Monitor

gRPC API Errors Monitor

Log Based Monitors

High Error Log Rate Monitor