Integration Examples with Workflows

Workflows are getting an upgrade: meet Notification Routes

This page provides examples of how to integrate workflows with different notification systems and external services.

Slack Notification

This workflow sends a simple Slack message when triggered:

workflow: 
  id: slack-notification
  description: Send Slack notification for alerts
  triggers:
    - type: alert
      filters:
        - key: annotations.slack-notification
          value: enabled                
  actions:
    - name: slack-notification
      provider:
        type: slack
        config: '{{ providers.slack_webhook }}'
        with:
          message: "Alert: {{ alert.alertname }} - Status: {{ alert.status }}"

Slack with Rich Formatting

This workflow sends a formatted Slack message using Block Kit:

workflow: 
  id: slack-rich-notification
  description: Send formatted Slack notification
  triggers:
    - type: alert
      filters:
        - key: annotations.slack-rich-notification
          value: enabled                
  actions:
    - name: slack-rich-notification
      provider:
        type: slack
        config: '{{ providers.slack_webhook }}'
        with:
          blocks:
          - type: header
            text:
              type: plain_text
              text: ':rotating_light: {{ alert.alertname }} :rotating_light:'
              emoji: true
          - type: divider
          - type: section
            fields:
            - type: mrkdwn
              text: |-
                *Cluster:*
                {{ alert.labels.cluster}}
            - type: mrkdwn
              text: |-
                *Namespace:*
                {{ alert.labels.namespace}}
            - type: mrkdwn
              text: |-
                *Status:*
                {{ alert.status}}

PagerDuty Integration

This workflow creates a PagerDuty incident:

 workflow:
  id: pagerduty-incident-workflow
  description: Create PagerDuty incident for alerts
  name: pagerduty-incident-workflow
  triggers:
    - type: alert
      filters:
        - key: annotations.pagerduty-incident-workflow
          value: enabled
  consts:
    severities: '{"S1": "critical","S2": "error","S3": "warning","S4": "info","critical": "critical","error": "error","warning": "warning","info": "info"}'
    severity: keep.dictget( '{{ consts.severities }}', '{{ alert.annotations._gc_severity }}', 'info')
    description: keep.dictget( {{ alert.annotations }}, "_gc_description", "")
    title: keep.dictget( {{ alert.annotations }}, "_gc_issue_header", '{{ alert.alertname }}')
    redacted_labels: keep.dict_pop({{ alert.labels }}, "_gc_monitor_id", "_gc_monitor_name", "_gc_severity", "backend_id", "grafana_folder")
    env: keep.dictget( {{ alert.labels }}, "env", "- no env -")
    namespace: keep.dictget( {{ alert.labels }}, "namespace", "- no namespace -")
    workload: keep.dictget( {{ alert.labels }}, "workload", "- no workload -")
    pod: keep.dictget( {{ alert.labels }}, "podName", "- no pod -")
    issue: https://app.groundcover.com/monitors/issues?backendId={{ alert.labels.backend_id }}&selectedObjectId={{ alert.fingerprint }}
    monitor: https://app.groundcover.com/monitors?backendId={{ alert.labels.backend_id }}&selectedObjectId={{ alert.labels._gc_monitor_id }}
    silence: https://app.groundcover.com/monitors/create-silence?keep.replace(keep.join({{ consts.redacted_labels }}, "&", "matcher_"), " ", "+")
  actions:
  - name: pagerduty-alert
    provider:
      config: '{{ providers.pagerduty-integration-name }}'
      type: pagerduty
      with:
        title: '{{ consts.title }}'
        severity: '{{ consts.severity }}'
        dedup_key: '{{alert.fingerprint}}'
        custom_details:
          01_environment: '{{ consts.env }}'
          02_namespace: '{{ consts.namespace }}'
          03_service_name: '{{ consts.workload }}'
          04_pod: '{{ consts.pod }}'
          05_labels: '{{ consts.redacted_labels }}'
          06_monitor: '{{ consts.monitor }}'
          07_issue: '{{ consts.issue }}'
          08_silence: '{{ consts.silence }}'

Opsgenie Integration

This workflow creates an Opsgenie alert:

• Alias is used to group identical events together in Opsgenie (alias key in the payload)

• Severities must be mapped to Opsgenie valid severities (priority key in the payload)

• Tags are a list of string values (tags key in the payload)

workflow:
  id: Opsgenie Example
  description: "Opsgenie workflow"
  triggers:
    - type: alert
      filters:
        - key: annotations.Opsgenie Example
          value: enabled
  consts:
    description: keep.dictget( {{ alert.annotations }}, "_gc_description", "")
    redacted_labels: keep.dict_pop({{ alert.labels }}, "_gc_monitor_id", "_gc_monitor_name", "_gc_severity", "backend_id", "grafana_folder", "CampaignName")
    severities: '{"S1": "P1","S2": "P2","S3": "P3","S4": "P4","critical": "P1","error": "P2","warning": "P3","info": "P4"}'
    severity: keep.dictget({{ consts.severities }}, {{ alert.annotations._gc_severity }}, "P3")
    title: keep.dictget( {{ alert.annotations }}, "_gc_issue_header", '{{ alert.alertname }}')
    region: keep.dictget( {{ alert.labels }}, "cloud.region", "")
    TenantID: keep.dictget( {{ alert.labels }}, "tenantID", "")
  name: Opsgenie Example
  actions:
  - if: '{{ alert.status }} == "firing"'
    name: opesgenie-alert
    provider:
      config: "{{ providers.Opsgenie }}"
      type: opsgenie
      with:
        alias: '{{ alert.fingerprint }}'
        description: '{{ consts.description }}'
        details: '{{ consts.redacted_labels }}'
        message: '{{ consts.title }}'
        priority: '{{ consts.severity }}'
        source: groundcover
        tags:
        - '{{ alert.alertname }}'
        - '{{ consts.TenantID }}'
        - '{{ consts.region }}'
  

Jira Ticket Creation

This workflow creates a Jira ticket using webhook integration:

workflow:
  id: jira-ticket-creation
  description: Create Jira ticket for alerts
  triggers:
    - type: alert
      filters:
        - key: annotations.jira-ticket-creation
          value: enabled    
  consts:
    description: keep.dictget({{ alert.annotations }}, "_gc_description", '')
    title: keep.dictget({{ alert.annotations }}, "_gc_issue_header", "{{ alert.alertname }}")
  actions:
    - name: jira-ticket
      provider:
        type: webhook
        config: '{{ providers.jira_webhook }}'
        with:
          body:
            fields:
              description: '{{ consts.description }}'
              issuetype:
                id: 10001
              project:
                id: 10000
              summary: '{{ consts.title }}'

Multiple Actions

This workflow performs multiple actions for the same alert:

workflow:
  id: multi-action-workflow
  description: Perform multiple actions for critical alerts
  triggers:
    - type: alert
      filters:
        - key: annotations.multi-action-workflow
          value: enabled      
        - key: severity
          value: critical
  actions:
    - name: slack-notification
      provider:
        type: slack
        config: '{{ providers.slack_webhook }}'
        with:
          message: "Critical alert: {{ alert.alertname }}"
    - name: pagerduty-incident
      provider:
        type: pagerduty
        config: '{{ providers.pagerduty_prod }}'
        with:
          title: "Critical: {{ alert.alertname }}"
    - name: jira-ticket
      provider:
        type: webhook
        config: '{{ providers.jira_webhook }}'
        with:
          body:
            fields:
              summary: "Critical Alert: {{ alert.alertname }}"
              description: "Critical alert triggered in {{ alert.labels.namespace }}"
              issuetype:
                id: 10001
              project:
                id: 10000