Ctrlk
Log in|Playground
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Ingest CloudWatch Metrics

groundcover supports ingesting CloudWatch metrics directly into our platform, allowing you to visualize them using dashboards and create monitors.

Before setting thresholds on aws_* series, see Alerting on CloudWatch metrics - the CloudWatch period and the polling lag both affect which monitor settings work.

How does it work

CloudWatch integration is done by deploying a service called integrations-agent which is responsible for pulling metrics from CloudWatch using periodic polling of these APIs:

The integration setup is done directly through the App by following these steps:

  1. Navigate to the Data Sources page or follow this link. Note that only users with Admin permissions can navigate to this page.

  2. Select Amazon Web Services and follow the wizard steps. Note that in step 1 you'll need to provide an ARN, granting groundcover with permissions to poll metrics. To do that, please follow the guidelines in this section.

Things to know

Ingestion interval

The integration pulls data from CloudWatch according to this interval. The lower the interval, the higher the polling rate and as a result, the overall costs will be higher. The interval can be altered in the Advanced Settings in the configuration wizard.

Data storage

Data fetched is stored in the Victoria Metrics database, meaning metrics are queried via the CloudWatch API only one time per data point.

Metric Statistics

Each metric has a label called stat which denotes the AWS statistic used during querying. Some metrics have multiple stats which are useful for different cases.

Supported AWS services

Click to Open

Resource Discovery Methods

groundcover seamlessly integrates both methods below to avoid duplicate metric fetching.

The integration uses two methods to discover the AWS resources to fetch metrics for:

  1. Tagging-based discovery - this method uses the AWS tagging mechanism to discover resources across all metric namespaces. This method supports all AWS namespaces but only works for resources which are tagged with at least one AWS tag.

  2. List-based discovery - this method uses standard AWS APIs to list the resources in each namespace. It works for all resources regardless of tags, but the coverage is limited to specific namespaces as listed below:

    1. AWS/RDS

    2. AWS/S3

    3. AWS/SQS

    4. AWS/Lambda

    5. AWS/ElastiCache

    6. AWS/DynamoDB

    7. AWS/ELB

    8. AWS/NetworkELB

    9. AWS/ApplicationELB

If you're not seeing metrics for a specific resource, it likely has no tags and is not in the list of services above. Contact us on Slack to help with resolving the issue.

Scoping tagging-based discovery to specific resources

By default, tagging-based discovery picks up every tagged resource in a namespace. To narrow this down, add one or more search tags in the wizard's tag step: each is a tag key plus a value, where the value is matched as a regular expression rather than an exact string. Only resources with a tag matching both the key and the value pattern are scraped. For example, a tag key team with value pattern platform.* matches resources tagged team=platform, team=platform-core, and so on, but not team=data.

Create an IAM role and policy

This guide assumes your BYOC backend is set up on AWS. If it's not, please follow the guidelines in this guide.

Follow the below guidelines to set up a role manually, or configure it using CloudFormation using this script.

The following part requires two parameters:

  • YOUR_GROUNDCOVER_ACCOUNT_ID - the AWS account id hosting the groundcover backend, created during onboarding

  • GROUNDCOVER_SITE_ID - the groundcover site ID as extracted from your BYOC endpoint:

    • Fetch your BYOC endpoint from these docs It will look like <SITE_ID>.platform.grcv.io

    • The GROUNDCOVER_SITE_ID is the first part marked above as <SITE_ID>

    • For example, if your BYOC endpoint address is m234r1.platform.grcv.io, then the GROUNDCOVER_SITE_ID will be m234r1.

  1. Go to Amazon IAM

  2. Click on Roles in the side bar

  3. Click on Create Role

    1. Select Custom trust policy

    2. Paste the following policy:

    3. Click on Next twice (we'll attach permissions later)

    4. Provide a name for the role

    5. Click on Create Role

  4. Go to your newly created role

    1. In the Permissions section, click on Add permissions and then Create inline policy

    2. Click on JSON and paste the following:

    3. Click on Next

    4. Give the policy a name

    5. Click on Create Policy

Alerting on CloudWatch metrics

CloudWatch stamps each datapoint at the start of the period it covers - often 5 minutes - and the datapoint only reaches groundcover on the next poll, one ingestion interval later. A monitor evaluates the last few minutes of data at the moment it runs, not the chart you look at afterwards, so monitors built on aws_* metrics need settings that account for both.

Recommended settings

Setting
Recommendation

Evaluation delay

At least the metric's CloudWatch period plus the ingestion interval set for the integration, so the datapoint has arrived by the time the monitor looks for it. 900 (15 minutes) is what the wizard prefills for metrics named aws_* and covers a 5-minute period at the default interval - raise it if you lengthened the ingestion interval. Monitors created from YAML, the API or Terraform must set it explicitly.

Evaluation interval

Match the CloudWatch period of the metric - 5 minutes for most, 1 minute where detailed monitoring is enabled.

Rollup window

Match the same period.

Treat No Data As

Avoid Normal if you need to know when the evaluation window is empty.

Match the evaluation interval to the CloudWatch period

CloudWatch datapoints are sparse compared to a 1-minute evaluation interval, which makes an interval shorter than the rollup window the default trap here: the same datapoint is counted on consecutive evaluations, and a sum rollup returns a multiple of the value CloudWatch reported - a metric that reported 30 evaluated as 150, with a threshold tuned against a number the metric never had. Matching the interval to the rollup window removes the duplication. See Dashboard shows different values than monitor for the general case.

A 1-minute chart of a 5-minute metric also stays flat between datapoints. That plateau is one value carried forward, not extra samples.

Expect issues to lag the CloudWatch timestamp

With an evaluation delay of 15 minutes and no pending period, an issue opens roughly 15 minutes after the timestamp CloudWatch assigned to the datapoint that crossed the threshold, and proportionally later if a longer delay covers a longer ingestion interval. That gap is the cost of waiting for data that arrives late, and it is expected. A pending period adds its own duration on top.

Decide what an empty window means

A poll that lands later than usual leaves the evaluation window empty. With Treat No Data As set to Normal that is indistinguishable from a healthy evaluation and the monitor stays green, so a missed datapoint passes silently. See Treat No Data As for the alternatives and Alerting on No Data for notifying on them.

No Data applies to the query as a whole. A query grouped by resource keeps returning rows while any one resource still reports, so a single load balancer that stops publishing is not a No Data evaluation.

Example monitor

CloudWatch metric names are ingested as aws_<namespace>_<metric> in lower snake case, and each series carries a stat label for the AWS statistic it was queried with. Filter on stat so a metric with multiple statistics does not return several series per resource.

See Monitor YAML structure for the full schema.

Last updated