Notification Routes

Notification Routes let you automatically send notifications to Connected Apps when monitor issues change state.

circle-info

This capability is only available to BYOC deployments. Check out our pricing pagearrow-up-right for more information about subscription plans and the available deployment modes.

circle-info

Editor permissions are required for creating and editing Notification Routes. Reader permissions may list the existing Notification Routes.

How It Works

  1. Scope: A gcQL query that filters which monitors' issues will trigger this route

  2. Rules: Define what happens when an issue is in a specific state (Firing or Resolved)

  3. Connected Apps: Choose where to send the notification (e.g., Slack Webhook, Pagerduty, etc)

Prerequisites

Before creating notification routes, set up your Connected Apps in Settings → Connected-Apps.

Create a Notification Route

  1. Go to Monitors → Notification Routes

  2. Click Create Notification Route

  3. Complete the wizard:

Step 1: Route Name

Give your route a descriptive name (e.g., prod-critical-alerts, infra-team-notifications).

Step 2: Scope Monitors

Define which monitors this route applies to using gcQL on:

  1. The grouping labels defined in the query

  2. The custom labels

  3. The Monitor's metadata such as the name or severity

Examples:

  • env:prod — All monitors with grouping key 'env' and possible value of 'prod'

  • env:prod AND severity:S1 — Only critical production alerts

  • team:platform — Monitors with a custom label for the platform team

  • *:* — To match all monitors

Step 3: Rules

Rules define what happens when a scoped monitor's issue changes state.

Each rule has:

  • Status: When to trigger — Firing (issue is active) or Resolved (issue cleared)

  • Connected Apps: Where to send the notification

Example setup:

  • When Firing or Resolved → Send to #prod-alerts Slack channel

  • When Firing (only) → Send to Pagerduty service directory

Click Add Rule to create multiple rules with different status/destination combinations.

Re-notification Interval

Configure how long to wait before sending another notification while an alert is still firing.

Options: 1m, 5m, 10m, 30m, 1h, 2h, 4h, 8h, 12h, 1d, 2d

This prevents notification fatigue from long-running alerts.

Managing Notification Routes

The Notification Routes page shows all your routes with:

  • Name: Route identifier

  • Scope: The gcQL query defining which monitors are affected

  • Connected Apps: Summary of destinations by type

  • Creator: Who created the route

Edit, Duplicate, or Delete

Hover over any row to access the action menu:

  • Edit: Modify the route configuration

  • Duplicate: Create a copy as a starting point

  • Delete: Remove the route

Example Use Cases

Route Critical Production Alerts to PagerDuty and Slack

Separate Routes by Team

Development Alerts — Firing Only

Testing Notifications

To test your notification configuration before enabling it on production monitors:

  1. Test Connected Apps directly: In Settings → Connected Apps, click the "Test" button on any configured app to send a test notification with sample data

  2. Test from a monitor: Open any monitor, click the "Test" button to send a test notification using that monitor's actual payload

circle-info

The test button in Connected Apps requires Admin permissions. The test button within monitors works for Editor permissions and respects your RBAC data scope.

Re-notification Behavior

Understanding how re-notification intervals work:

  • The re-notification timer starts when the first notification is sent

  • If an alert resolves and fires again within the re-notification window, and the fingerprint is the same, the timer continues (no new notification)

  • The renotification_count field in payloads starts at 0 for the first notification and increments with each re-notification

  • When an alert resolves, the counter resets for the next firing cycle

Permissions

Action
Required Permission

Create/Edit Notification Routes

Editor

View Notification Routes

Reader

Create/Edit Connected Apps

Admin

Test Connected Apps

Admin

Test from Monitor

Editor

How Routes Apply to Monitors

Notification Routes apply automatically based on label matching. You do not need to edit existing monitors to use routes:

  1. Routes match monitors based on the scope query (gcQL)

  2. When a monitor fires, all routes whose scope matches the monitor's labels will trigger

  3. Multiple routes can match the same monitor

Troubleshooting

Notifications not being sent

  1. Verify the scope query matches your monitor's labels

  2. Check that the Connected App is properly configured (use Test button)

  3. Review dispatch-center traces in groundcover for delivery errors

  4. Ensure the monitor has transitioned state (not just hovering at threshold)

If clicking notification links shows "Last 5 minutes" instead of the actual alert time, this typically affects legacy Workflow-based notifications. Migrate to Notification Routes for proper time range handling.

Debugging notification delivery

Filter traces by workload:dispatch-center to see notification delivery attempts and any errors.

Last updated