Workflow Examples

This page provides practical examples of workflows for different use cases and integrations.

Triggers Examples

Filter by Monitor Name

This example shows how to create a workflow that only triggers for a specific monitor (by its name):

workflow: 
  id: specific-monitor-workflow
  description: Workflow triggered only by Workload Pods Crashed Monitor
  triggers:
    - type: alert
      filters:
        - key: alertname
          value: Workload Pods Crashed Monitor

Filter by Environment

Execute only on the Prod environment. The "env" attribute needs to be part of the monitor context attributes (either by using it in the group by section or by explicitly adding it as a context label):

workflow: 
  id: prod-only-workflow
  description: Workflow triggered only by production environment alerts
  triggers:
    - type: alert
      filters:
        - key: env
          value: prod

Filter by Multiple Conditions

This example shows how to combine multiple filters. In this case it will match events from the prod environment and also monitors that explicitly routed the workflow with the name "actual-name-of-workflow":

workflow: 
  id: multi-filter-workflow
  description: Workflow triggered by critical alerts in production
  triggers:
    - type: alert
      filters:
        - key: env
          value: prod
        - key: annotations.actual-name-of-workflow
          value: enabled

Filter by Regex

In this case we will use a regular expression to filter on events coming from the groundcover OR monitoring namespaces. Note that any regular expression can be used:

workflow: 
  id: regex-filter-workflow
  description: Workflow triggered by alerts from groundcover or monitoring namespaces
  triggers:
    - type: alert
      filters:
        - key: namespace
          value: r"(groundcover|monitoring)"

Consts Examples

The consts section is the best location to create pre-defined attributes and apply different transformations on the monitor's metadata for formatting the notification messaging.

Map Severities

Severities in your notified destination may not match the groundcover predefined severities. By using a dictionary, you can map any groundcover severity value to another, and extract it by using the actual monitor severity. Use the "keep.dictget" function to extract from a dictionary and apply a default in case the value is missing.

workflow:
  id: severity-mapping-example
  description: Example of mapping severities using consts
  triggers:
    - type: alert
  consts:
    severities: '{"S1": "P1","S2": "P2","S3": "P3","S4": "P4","critical": "P1","error": "P2","warning": "P3","info": "P4"}'
    severity: keep.dictget({{ consts.severities }}, {{ alert.annotations._gc_severity }}, "P3")

Best Practice for Accessing Monitor Labels

When accessing a context label via alert.labels, if this label is not transferred within the monitor - the workflow might crash. Best practice to pre-define labels is to declare them in the consts section with a default value, using "keep.dictget" so the value is gracefully pulled from the labels object.

workflow:
  id: labels-best-practice-example
  description: Example of safely accessing monitor labels
  triggers:
    - type: alert
  consts:
    region: keep.dictget({{ alert.labels }}, "cloud.region", "")

Note: Label names that are dotted, like "cloud.region" in this example, cannot be referenced in the monitor itself and can only be retrieved using this technique of pulling the value with "keep.dictget" from the alert.labels object.

Additional Useful Functions

keep.dict_pop({{alert.labels}}, "_gc_monitor_id", "_gc_monitor_name", "_gc_severity", "backend_id", "grafana_folder", "_gc_issue_header") - "Clean" a key-value dictionary from some irrelevant values (keys). In this case, the groundcover labels dictionary has some internal keys that you might not want to include in your notification content.
keep.join(["a", "b", "c"], ",") - Joins a list of elements into a string using a given delimiter. In this case the output is "a,b,c".

Action Examples

Conditional Statements

Use "if" condition to apply logic on different actions.

Create a separate block for a firing monitor (a resolved monitor can use different logic to change formatting of the notification):

workflow:
  id: conditional-actions-example
  description: Example of conditional actions based on alert status
  triggers:
    - type: alert
  actions:
    - if: '{{ alert.status }} == "firing"'
      name: slack-action-firing
      provider:
        config: '{{ providers.groundcover-alerts-dev }}'
        type: slack
        with:
          attachments:
          - color: '{{ consts.red_color }}'
            footer: '{{ consts.footer_url }}'
            footer_icon: '{{ consts.footer_icon }}'
            text: '{{ consts.slack_message }}'
            title: 'Firing: {{ alert.alertname }}'
            title_link: '{{ consts.title_link }}'
            ts: keep.utcnowtimestamp()
            type: plain_text
          message: ' '

"If" statements can include and/or logic for multiple conditions:

workflow:
  id: multi-condition-actions-example
  description: Example of multiple conditions in actions
  triggers:
    - type: alert
  actions:
    - if: '{{ alert.status }} == "firing" and {{ alert.labels.namespace }} == "namespace1"'
      name: slack-action-firing
      provider:
        config: '{{ providers.groundcover-alerts-dev }}'
        type: slack
        with:
          attachments:
          - color: '{{ consts.red_color }}'
            footer: '{{ consts.footer_url }}'
            footer_icon: '{{ consts.footer_icon }}'
            text: '{{ consts.slack_message }}'
            title: 'Firing: {{ alert.alertname }}'
            title_link: '{{ consts.title_link }}'
            ts: keep.utcnowtimestamp()
            type: plain_text
          message: ' '

Notification by Specific Hours

Use the function keep.is_business_hours combined with an "if" statement to trigger an action within specific hours only.

In this example the action block will execute on Sundays (6) between 20-23 (8pm to 11pm) or on Mondays (0) between 0-1am:

workflow:
  id: time-based-notification-example
  description: Example of time-based conditional actions
  triggers:
    - type: alert
  actions:
    - if: '({{ alert.status }} == "firing" and (keep.is_business_hours(timezone="America/New_York", business_days=[6], start_hour=20, end_hour=23) or keep.is_business_hours(timezone="America/New_York", business_days=[0], start_hour=0, end_hour=1)))'
      name: time-based-notification
      provider:
        type: slack
        config: '{{ providers.slack_webhook }}'
        with:
          message: "Time-sensitive alert: {{ alert.alertname }}"

Last updated 17 days ago