Ctrlk
Log in|Playground
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Create a new Monitor

Learn how to create and configure monitors using the Wizard, Monitor Catalog, or Import options. The following guide will help you set up queries, thresholds, and alert routing for effective monitoring.

You can either create monitors using our web application following this guide, or use our API, see: Monitors or use our Terraform provider, see: groundcover Terraform Provider.

In the Monitors section (left navigation bar), navigate to the Issues page or the Monitor List page to create a new Monitor. Click on the “Create Monitor” button at the top right and select one of the following options from the dropdown:

Monitor Wizard

Monitor Catalog

Import

Using the Monitor Wizard

Overview

The Monitor Wizard is a guided, user-friendly approach to creating and configuring monitors tailored to your observability needs. By breaking down the process into simple steps, it ensures consistency and accuracy.

Section 1: Query

Select the data source, build the query and define thresholds for the monitor.

If you're unfamiliar with query building in groundcover, refer to the Query Builder section for full details on the different components.

  • Data Source (Required):

    • Select the type of data (Metrics, Logs, Traces, Events, APM, or Ingestion).

    • Ingestion monitors how much data groundcover ingests (volume in bytes or entry count, for logs or traces). Useful for alerting on sudden ingestion drops or spikes. See Ingestion in the Query Builder for the available signals and options.

  • Query Functions:

    • Choose how to process the data (e.g., average, count).

    • Add aggregation (group by) clauses if applicable, you MUST use aggregations if you want to add labels to your issues.

    • Important: The labels used for aggregation (group by) maybe also be used for notification routes and the issue summary & description.

    • Examples: cluster, node, container_name

  • Time Window (Required):

    • Specify the period over which data is aggregated (the look-behind window).

    • Example: “Over the last 5 minutes.”

  • Window Aggregation (Required):

    • Specify the aggregation function to be used on the selected time window.

    • Example: "avg over the last 5m"

  • Threshold Conditions (Required):

    • Define when a monitor should trigger an Issue. You can use:

      • Greater Than - Trigger when the value exceeds X.

      • Lower Than - Trigger when the value falls below X.

      • Within Range - Trigger when the value is between X and Y.

      • Outside Range - Trigger when the value is not between X and Y.

    • Important: The units in which the threshold is being measured in must be the same as the units the query uses.

      • For metrics queries the threshold should match the unit the metric is measured in.

      • For APM queries the threshold should match the selected metric's unit (e.g. seconds for latency, % for Error Rate, a plain number for request/error counts).

      • For logs, traces, and events, it's just a number.

    • Example: “Trigger if disk space usage is greater than 10%.”

  • Preview Settings (Optional):

    • Preview data using Stacked Bar or Line Chart for better clarity while building the monitor.

    • Choose the Y axis units.

    • Choose the rollup to present.

    • Important: These configurations only affect the preview graph, not the monitor's evaluation.

  • Advanced (Optional):

    • Evaluation Interval:

      • Specify how often the monitor evaluates the query

      • Example: “Evaluate every 1 minute.”

    • Pending Period:

      • Specify how many times the evaluation needs to pass the threshold in order to trigger an Issue. This refers to a consecutive evaluations passing the threshold.

      • Monitors that have entered the pending period (the first evaluation passed the threshold) will be in 'Pending' state, only after all consecutive evaluations passed the threshold, the monitor will be 'Firing' and an Issue will be created. If even 1 of the evaluations did not pass the threshold, the Monitor will be set right back to 'Normal'.

      • Example: “When Evaluation Interval of 5m, setting this to 2 (10m) ensures the condition must be evaluated 3 times before a monitor will fire.

        • Evaluation #1 at 0m

        • Evaluation #2 at 5m

        • Evaluation #3 at 10m -> If all 3 passed the threshold, an the monitor will 'Fire'

      • Note: This ensures that transient conditions do not trigger alerts, reducing false positives or smoothing sudden unwanted spikes.

      • Important: The default configuration is 0, which means the monitor will trigger an Issue immediately when an evaluation was run and the threshold was passed.

    • Evaluation delay:

      • Evaluates the query against a window that ends this many seconds in the past instead of "now". Useful for sources that backfill recent data (like AWS CloudWatch and GCP), which can otherwise cause false alerts.

      • Input is in seconds (0–3600). Leave it empty or 0 for no delay (the default). The monitor shifts its whole evaluation window back by the delay without changing your query, and the Preview graph reflects the same shift.

      • For new metric monitors, groundcover prefills a recommended delay from the metric name (aws_ → 15m, gcp → 5m), which you can override or clear.

    • Treat No Data As:

      • Controls what the monitor does when its query returns no data (an empty result set) on an evaluation. Choose one of:

        • Normal — Treated as healthy. No Issue is created and no notification is sent; the monitor stays green.

        • Firing — Creates an Issue (and sends notifications, if a notification route matches), exactly like a threshold breach. Choose this to be alerted when data stops arriving — see Alerting on No Data.

        • No Data — Sets the monitor's status to No Data, shown on the Monitor List and in the monitor's timeline and facets. No Issue is created and no notification is sent: No Data is a monitor status only, not an Issue.

      • Example: "I want to be notified if the metric has a gap for the entire look-behind window of the query, so I will set it to 'Firing'."

      • No data vs. a zero value: "No data" means the query returned no rows, not a row whose value is 0. For example, workload:agent | stats count() as cnt returns a single row with cnt = 0 when nothing matches — that is data (a zero value), so it is not treated as No Data. To make the query return no rows when there are no matches, drop the zero row with a filter:

        workload:agent | stats count() as cnt | filter cnt > 0
    Monitor advanced evaluation settings, including Treat No Data As

Section 2: Monitor Details

Set up the basic information for the monitor.

  • Monitor Name (Required):

    • Add a title for the monitor. The title will appear in notifications and in the Monitor List page.

    • Give the Monitor a clear, short name, that describes its function at a high level.

    • Examples:

      • “Workload High API Error Rate”

      • “Node High Memory”

The title will appear in the monitors page table and be accessible in notification routes.

  • Severity (required):

    • Use severity to categorize alerts by importance.

    • Select a severity level (S1-S4).

    • Important: For Destinations (OpsGenie, PagerDuty) that require specific severities like P1-P4 or Critical-Info, we translate automatically to the relevant respective severity.

  • Custom Labels (formally called 'metadata labels'):

    • Add custom labels (key:value) that will be added to all Issues generated by this monitor

    • Example: To create a notification route for my team's issues, add "Team:Infra" and use it in the notification route's scope

Section 3: Issue Details

Customize how the Monitor’s Issues will appear and what content will be sent in it's notifications. This section also includes a live preview of the way it will appear in the notification.

Ensure that the labels you wish to use dynamically (e.g., cluster, workload) or statically (e.g. team:infra) are defined in the query and monitor details.

  • Issue Summary (required):

    • Define a short title for issues that this Monitor will raise. It's useful to include variables that can be informative at first glance.

    • Example: Adding {{ labels.statusCode }} to the header will inject the status code to the name of the issue - this becomes especially useful when one Monitor raises multiple issues and you want to quickly understand their content without having to open each one.

      • “HTTP API Error {{ labels.status_code }}” -> HTTP API Error 500

      • “Workload {{ labels.workload }} Pod Restart” -> Workload frontend Pod Restart

      • “{{ labels.team }} APIs High Latency” -> Infra APIs High Latency

    • Note: Autocomplete is supported to view what is usable in the Issue and will help ensure you put in the variables correctly.

The new format for templating variables is {{ variable }} or {{ labels.<label> }} , but the previous format {{ alert.labels.statusCode }} used for keep is still supported.

  • Description:

    • Used as the body of the message for the Issue.

    • The default templating uses simple {{ variable }} and {{ labels.<label> }} substitution. Opt in to full Jinja2 by setting display.templateLanguage: jinja2 in the monitor YAML.

      • Example: Adding all the labels to be shown in the Slack message's body should be inserted into here using {{labels.<label>}} , you can add the severity {{severity}}, the monitor's name {{monitor_name}} and many more.

    • URLs can be rendered using <url link|url title>

      • Example: <www.groundcover.com/{{labels.env}}|text> will add the env label from the issue to the URL link and put the link inside a text called 'text'

Full Jinja2 features like {% if %} blocks and filters require display.templateLanguage: jinja2 in the monitor YAML. Without that field, only plain {{ variable }} substitution is supported.

  • Advanced (Optional)

    • Display Labels (formally called 'context labels'):

      • These Labels will be displayed and filterable in Monitors>Issues page.

      • This list gets automatically populated based on the labels used in the aggregation function in the Query.

      • Note: You can remove labels from this list if you do not wish to see them in the Issues page.

Section 4: Notifications

Set up notifications behavior for issues from this monitor.

Workflows (used for Keep) and Notification Routes may work in parallel and do not affect each other.

Choose one of three notification methods:

  • Based on matching notification routes (default)

    • The issues generated by this monitor will be evaluated by the Notification Route's scopes and rules and notifications will be sent accordingly.

    • Note: The Preview can be used in order to align expectations on which notification routes may match this monitor's future issues.

  • Directly to destinations

    • Send notifications directly to specific Destinations for this monitor only, bypassing notification routes.

    • Click Set rule, choose an issue status filter (Firing, Resolved, or both), then in the Send to dropdown select the Destination.

    • For a Slack App Destination, the dropdown opens a drilldown panel where you pick the target Slack channel. The bot can post to any public channel in the workspace without being added; private channels only appear in the picker if the bot has been invited to them.

    • Click + to add more destinations — additional Slack channels (same Slack App), other Slack Apps, or any other Destination.

    • Note: Issues from this monitor will be skipped by all matching notification routes when this method is selected.

  • Don't send notifications

    • Suppresses all notifications for this monitor. The monitor still evaluates and creates issues, but no Destinations or notification routes are notified.

The wizard labels issue states as Firing and Resolved. The underlying YAML schema uses the enum values Alerting and Resolved (see Monitor YAML structure) — Firing in the UI is the same state as Alerting in YAML.

To set up the Slack App connector itself, see Slack.

  • Routing (Workflows) (Optional)

    • Select Workflow:

      • Route alerts to existing workflows only, this means that other workflows will not process them. Use this to send alerts for a critical application such as Slack or PagerDuty.

    • No Routing:

      • This means that any workflow (without filters), will process the issue.

Configure Routing for Keep Workflows only

  • Advanced (Optional)

    • Override Renotify Interval

      • Used to override the interval configured on the Notification Route for when a certain monitior's issue should send another notification at a different inerval.

      • Example: If it's set to 1m while the evaluation interval is 1m a notification will be sent with every firing evaluation. If it's set to 2d, even if the monitor evaluates every 1m, a notification will be sent once every 2 days.

      • Note: If the Issue stops firing and starts firing again, a new notification will be sent, this is not considered 'renotification'

      • Important: Minimum interval is the evaluation interval

Using the Import option

This is an advanced feature, please use it with caution.

In the "Import Bulk Monitors" you can add multiple monitors using an array of Monitors that follows the Monitor YAML structure.

Example of importing multiple monitors

Click on "Create Monitors" to create them.

Query Best Practices

Performance Recommendations

To ensure reliable monitor evaluation and avoid timeouts:

  • Avoid excessively long time ranges with free-text search queries. Maximum recommended range is 7 days for most query types.

  • Use attribute filters instead of free-text search when possible - free-text searches across large time ranges are expensive.

  • Set appropriate dashboard refresh intervals - avoid refreshing complex dashboards every 1-2 minutes with week-long queries.

  • Consider logs-to-metrics for aggregation queries that would otherwise scan large log volumes.

  • Use parsing rules to add attributes for frequently-filtered fields instead of relying on free-text search.

MetricsQL Query Limitations

When using MetricsQL in monitors:

Regex Operators: The regex not-match operator !~ may have limitations with certain patterns:

  • Pattern .+ (one or more characters) may not work as expected in some cases

  • Use .* (zero or more characters) as an alternative: label!~".*Error" instead of label!~".+Error"

Switching between data sources: Changing the data source type (e.g., from Metrics to Logs) will clear the current query.

gcQL Query Limitations in Monitors

Some gcQL operations behave differently in monitors compared to the Data Explorer:

Operation
Data Explorer
Monitors

grok parsing

Full support

Supported (fixed in recent versions)

join operations

Full support

Limited - complex joins may timeout

Free-text search on message fields

Works

May have limitations in Explore filter bar

If your query works in Data Explorer but fails in monitors, try simplifying the query or breaking it into multiple monitors.

Troubleshooting

Monitor Not Firing

  1. Check the preview graph - Verify the query returns data and exceeds the threshold

  2. Review pending period - If set, the condition must be met for multiple consecutive evaluations

  3. Check "No Data" handling - If data is intermittent, "Treat No Data As: Firing" may cause unexpected behavior

  4. Verify aggregation labels - Ensure "group by" labels match what you expect

Labels Not Appearing in Notifications

  1. Labels must be in "group by" - Only labels used in aggregation are available as {{ labels.<name> }}

  2. Check variable syntax - Use {{ labels.workload }} not {{ alert.labels.workload }} (though both are supported)

  3. Preview the notification - Use the live preview to verify variable expansion

Query Works in Explorer but Not in Monitor

  1. Time alignment - Explorer and monitors may use different time window handling

  2. Aggregation function order - For metrics like sum(sum_over_time(...)), ensure the step and aggregation window match to avoid inflated values

  3. gcQL limitations - Some advanced operations may not be fully supported in monitor queries

Dashboard Shows Different Values Than Monitor

This can occur due to:

  • Step vs aggregation window mismatch - Using sum_over_time(...[5m]) with step=1m creates a sliding window that can inflate values. Align step with aggregation window.

  • Different aggregation functions - Verify both use the same aggregation

  • Time range differences - Monitors use a specific look-behind window

Last updated