Create an alert

Alerts in groundcover leverage a fully integrated Grafana interface. To learn how you can create alerts using Grafana Terraform, follow this guide.

Setup an alert based on metrics

Setting up an alert in groundcover involves defining conditions based on data collected in the platform, such as metrics, traces, logs, or Kubernetes events. This guide will walk you through the process of creating an alert based on metrics. More guides will follow to include all different types of data.

Step 1: Access the Alerts section

1. Log in to groundcover and navigate to the Alerts section by clicking on it on the left navigation menu.

2. Once in the Alerts section, click on Alerting in the inner menu on the left.

   • If you can't see the inner menu, click on the 3 bars next to "Home" in the upper left corner.

3. Click on Alert Rules

4. Then click on the blue "+ New alert rule" button in the upper right.

Step 2: Give the alert a name and define query and conditions

1. Type a name for your alert. It's recommended to use a name that will make it easy for you to understand its function later.

2. Select the data source:

   1. ClickHouse: For alerts based on your traces, logs, and Kubernetes events.

   2. Prometheus: For alerts based on metrics (includes APM metrics, infrastructure metrics, and custom metrics from your environment)

3. Click on "Select metric"

   • Note: Make sure you are in "Builder" view (see screenshot) to see this option.

4. Click on "Metrics explorer"

5. Start typing the name of the metric you want this alert to be based on. Note that the Metrics explorer will start displaying matches as you type, so you can find your metric even if you don't remember it exact name. You can also check out our list of Metrics & Labels.

6. Once you see your metric in the list, click on "Select" in that row.

Note: You can click on "Run queries" to see the results of this query.

Step 3: Define expressions - Reduce & Threshold

1. In the Reduce section, open on the "Function" dropdown menu and choose the type of value you want to use.

   • Min - the lowest value

   • Max - the highest value

   • Mean - the average of the values

   • Sum - the sum of all values

   • Count - the number of values in the result

   • Last - the last value

2. In the Threshold section, type a value and choose whether you want the alert to fire when the query result is above or below that value. You can also select a range of values.

Step 4: Set evaluation behavior

1. Click on "+ New folder" and type a name for the folder in which this rule will be stored. You can choose any name, but it's recommended to use a name that will make it easy for you to find the relevant evaluation groups, should you want to use them again in future alerts.

2. Click on "+ New evaluation group" and type a name for this evaluation group. The same recommendation applies here too.

   In the Evaluation interval textbox, type how often the rule should be evaluated to see if it matches the conditions set in Step 3. Then, click "Create". Note: For the Evaluation interval, use the format (number)(unit), where units are:

   • s = seconds

   • m = minutes

   • h = hours

   • d = days

   • w = weeks

3. In the Pending period box, type how often you want the alert to match the conditions before it fires.

Evaluation interval = how often do you want to check if the alert should fire

Pending period = how long do you want this to be true before it fires

As an example, you can define the alert to fire only if the Mean percentage of memory used by a node is above 90% in the past 2 minutes (Pending period = 2m) and you want to check if that's true every 30 seconds (Evaluation interval = 30s).

Step 5: Choose contact point

If you already have a contact point set up, simply select it from the dropdown menu at the bottom of the "Configure lables and notifications" section. If not, click on the blue "View or create contact points" link, which will open a new tab.

Click on the blue "Add contact point" button

This will get you to the Contact points screen. Then:

1. Type a name for the contact point

2. From the dropdown menu, choose which system you want to use to push the alert to.

3. The information required to push the alert will change based on the system you select. Follow on-screen instructions (for example, if email is selected, you'll need to enter the email address(es) for that contact.

4. Click "Save contact point"

You can now close this tab to go back to the alert rule screen.

Next to the link you clicked to create this new contact point, you'll find a dropdown menu, where you can select the contact point you just created.

Step 6: Add annotations

Under "Add annotations", you have two free text boxes that give you the option to add any information that can be useful to you and/or the recipient(s) of this alert, such as a summary that reminds you of the alert's functionality or purpose, or next step instructions when this alert fires.

Step 7: Save and exit

Once all of it is ready, you can click the blue "Save rule and exit" button on the upper right of the screen, which will bring you back to the Alert rules screen. You will now be able to see your alert, as well as its status - normal (green), pending (yellow), or firing (red), as well as the Evaluation interval (blue).

Configuring Alert from existing dashboard:

1. Log in to your groundcover account and navigate to the dashboard that you want to create an alert from.

2. Locate the Grafana panel that you want to create an alert from and click on the panel's header and select edit .

3. Click on the alert tab as seen in the image below. Select the Manage alerts option from the dropdown menu.

4. Click on the New Alert Rule button.

Note: only time series panels support alert creation.

Create Alert button referenced in point 4 below

1. An alert is derived from three parts that will be configured in the screen that you are navigated to:

   • Expression - the query that defines the alert input itself,

   • Reduction - the value that should be leveraged from the aforementioned expression

   • Threshold - value to measure against said reduciton output to see if an alert should be triggered

2. Verify expression value and enter reduction and threshold values in line with your alerting expectation

Expression, reduction, and threshold entry screen

1. Select folder - if needed you can navigate to dashboard tab in left nav and create new folder

2. Select evaluation ground or type text in order to create a new group as shown below

1. Click "Save and Exit" on top right hand side of screen to create alert

2. Ensure your notification is configured to have alerts sent to end users. See "Configuring Slack Contact Point" section below if needed.

Note: Make sure to test the alert to ensure that it is working as expected. You can do this by triggering the conditions that you defined and verifying that the alert is sent to the specified notification channels.