Workflow Examples

Slack Webhook Message on Specific Monitor

This workflow is triggered by issue with filter of alertname: Workload Pods Crashed Monitor . Which means only issues created by the monitor named "Workload Pods Crashed Monitor" will trigger the workflow, in this example we use a slack message action using labels from the issue.

workflow: 
  id: slack-alert-on-crashed-pods
  description: Send a slack message on Workload Pods Crashed Monitor 
  triggers:
    - type: alert
      filters:
        - key: alertname
          value: Workload Pods Crashed Monitor
  actions:
    - name: trigger-slack
      provider:
        type: slack
        config: '{{ providers.slack_webhook }}'
        with:
          message: "Pod Crashed - Pod: {{ alert.labels.pod_name }} Container: {{ alert.labels.container }} Exit Code: {{ alert.labels.exit_code }} Reason: {{ alert.labels.reason }}"

Slack Webhook Message on Issue

This workflow is triggered by an issue and uses the slack_webhook integration to send a Slack message formatted with Block Kit. For more details, see Slack Block Kit.

workflow: 
  id: slack-webhook
  description: Send a slack message on alerts
  triggers:
    - type: alert
  actions:
    - name: trigger-slack
      provider:
        type: slack
        config: ' {{ providers.slack_webhook }} '
        with:
          blocks:
          - type: header
            text:
              type: plain_text
              text: ':rotating_light: {{ alert.labels.alertname }} :rotating_light:'
              emoji: true
          - type: divider
          - type: section
            fields:
            - type: mrkdwn
              text: |-
                *Cluster:*
                {{ alert.labels.cluster}}
            - type: mrkdwn
              text: |-
                *Namespace:*
                {{ alert.labels.namespace}}
            - type: mrkdwn
              text: |-
                *Workload:*
                {{ alert.labels.workload}}

Slack Webhook Message with Logs Enrichment

This example enriches a “Workload Pods Restarts Monitor” issue with logs retrieved from ClickHouse. The workflow uses a step named clickhouse-step, which runs a SQL query based on the alert labels. The query results are included in the Slack message.

workflow: 
  id: enrich-with-clickhouse-workflow
  description: Enrich restart alert with clickhouse logs
  triggers:
    - type: alert
      filters:
        - key: alertname
          value: Workload Pods Crashed Monitor
  steps:
    - name: clickhouse-step
      provider:
        config: "{{ providers.groundcoverClickhouse }}"
        type: clickhouse
        with:
          query: "SELECT content FROM logs where timestamp > now() - interval '10 minutes' and workload = '{{ alert.labels.workload }}' and namespace = '{{ alert.labels.namespace }}' and cluster = '{{ alert.labels.cluster }}' and pod_name = '{{ alert.labels.pod_name }}' ORDER BY timestamp ASC LIMIT 20"
  actions:
    - name: slack-action
      provider:
        type: slack
        config: "{{ providers.slack_webhook }}"
        with:
          blocks:
          - type: header
            text:
              type: plain_text
              text: ":rotating_light: {{ alert.labels.alertname }} :rotating_light:"
              emoji: true
          - type: divider
          - type: section
            text:
              type: mrkdwn
              text: |-
                *Cluster:* {{ alert.labels.cluster }}
                *Namespace:* {{ alert.labels.namespace }}
                *Workload:* {{ alert.labels.workload }}
                *Pod:* {{ alert.labels.pod_name }}
                *Container:* {{ alert.labels.container }}
                *Exit Code:* {{ alert.labels.exit_code }}, *Reason:* {{ alert.labels.reason }}
          - type: divider
          - type: section
            text:
              type: mrkdwn
              text: |-
                *Last 20 Logs:*
                ```
                {{#steps.clickhouse-step.results}}
                {{body}}
                {{/steps.clickhouse-step.results}}
                ```