Ingest Google Cloud Monitoring Metrics

groundcover supports ingesting Google Cloud Monitoring Metrics directly into our platform, allowing you to visualize them using dashboards and create alerts.

Before setting thresholds on gcp_* series, see Alerting on Google Cloud Monitoring metrics - the sampling period and the polling lag both affect which monitor settings work.

Things to know

Integration Interval

The integration pulls data from GCP according to this interval. The lower the interval, the higher the polling rate and as a result, the overall costs will be higher. The interval can be altered in the configuration wizard.

Data storage

Data fetched is stored in the Victoria Metrics database, meaning metrics are queried via the Google Cloud Monitoring API only one time per data point.

Supported services

groundcover supports ingestion of every GCP service listed in the GCP docs. Users should provide a prefix of a service or metric in order to pull it, e.g. cloudsql.googleapis.com will pull all metrics for CloudSQL, while cloudsql.googleapis.com/database/active_directory/domain_reachable will pull this specific metric.

List of supported services

actions
aiplatform
alloydb
apigateway
apigee
appengine
apphub
artifactregistry
autoscaler
backupdr
baremetalsolution
bigquery
bigquerybiengine
bigquerydatatransfer
bigquerystorage
bigtable
billingbudgets
blockchainnodeengine
certificatemanager
chronicle
cloudaicompanion
cloudbuild
clouddeploy
cloudfunctions
cloudkms
cloudsql
cloudtasks
cloudtrace
composer
compute
connectors
contactcenterinsights
container
dataflow
datamigration
dataplex
dataproc
datastore
datastream
dbinsights
dialogflow
discoveryengine
displayvideo
dlp
dns
earthengine
edgecache
edgecontainer
eventarc
file
firebaseappcheck
firebaseapphosting
firebaseauth
firebasedatabase
firebasedataconnect
firebaseextensions
firebasehosting
firebasestorage
firestore
firewallinsights
fleetengine
gkebackup
healthcare
iam
identitytoolkit
ids
integrations
interconnect
livestream
loadbalancing
logging
managedflink
managedidentities
managedkafka
maps
memcache
memorystore
metastore
ml
monitoring
netapp
networkconnectivity
networking
networksecurity
networkservices
oracledatabase
osconfig
parallelstore
privateca
pubsub
pubsublite
recaptchaenterprise
recommendationengine
redis
retail
router
run
serviceruntime
spanner
storage
storagetransfer
telcoautomation
tpu
trafficdirector
transferappliance
translationhub
videostitcher
visionai
vpcaccess
vpn
workflows

Setting up the integration

The integration setup is done directly through the App by following these steps:

1. Navigate to the Data Sources page or follow this link. Note that only users with Admin permissions can navigate to this page.

2. Select Google Cloud Platform and follow the 3 steps in the wizard. Note that in step 1 you'll need to provide a Service Account, granting groundcover with permissions to poll metrics. To create a service account, please follow the guidelines in the next section.

Create a Service Account and Assign Permissions

This guide assumes your BYOC backend is set up on GCP. If it's not, please follow the guidelines in this guide.

The following steps requires the details of dedicated service account set up the integration - marked below as <integrations-sa>.

Make sure it has been provided to you by the groundcover team

1. Go to IAM & Admin in GCP Console

2. Navigate to IAM & Admin → Service Accounts.

3. Create a New Service Account

   • Click on Create Service Account.

   • Provide a name and description for the service account.

   • Click Create & Continue.

   • In the Grant this service account access to the project step, search for and select Monitoring Viewer (roles/monitoring.viewer).

   • Click Continue → Done.

4. Grant service account impersonation access

   • Click on the newly created service account → Permissions → View by Principals → Grant Access

   • In the Add principals → New principal section enter the <integrations-sa> provided to you.

   • In the Assign roles → Role section choose Service Account Token Creator

   • Click Save

Troubleshooting

1. In case there are missing metrics compared with the configuration, open the drawer of the relevant configuration and check the Activty Traces tab to see relevant error message.

2. In case of 429: Quota exceeded for quota metric, follow this guide to increase the quota or adjust your configuration by reducing the number of project ID's or services to query.

Alerting on Google Cloud Monitoring metrics

Google Cloud Monitoring stamps each datapoint at the start of the sampling period it covers, and the datapoint only reaches groundcover on the next poll, one integration interval later. A monitor evaluates the last few minutes of data at the moment it runs, not the chart you look at afterwards, so monitors built on gcp_* metrics need settings that account for both.

Recommended settings

Setting	Recommendation
Evaluation delay	At least the metric's sampling period plus the integration interval set for the integration, so the datapoint has arrived by the time the monitor looks for it. 300 (5 minutes) is what the wizard prefills for metrics named gcp_* and covers a 1-minute sampling period at the default interval - raise it if you lengthened the integration interval. Monitors created from YAML, the API or Terraform must set it explicitly.
Evaluation interval	Match the sampling period of the metric, commonly 1m.
Rollup window	Match the sampling period as well.
Treat No Data As	Avoid Normal if you need to know when the evaluation window is empty.

Sampling periods differ per metric - most are 60 seconds, but some services report far less often. Look up the metric in the Google Cloud metrics list and use its sampling period for both the evaluation interval and the rollup window.

Match the evaluation interval to the sampling period

An evaluation interval shorter than the rollup window counts the same datapoint on consecutive evaluations, so a sum rollup returns a multiple of the value the metric reported - a metric sampled every 5 minutes that reported 30, evaluated every minute, comes out as 150. Matching the interval to the rollup window removes the duplication. See Dashboard shows different values than monitor for the general case.

A 1-minute chart of a metric sampled every 5 minutes also stays flat between datapoints. That plateau is one value carried forward, not extra samples.

Expect issues to lag the datapoint timestamp

With an evaluation delay of 5 minutes and no pending period, an issue opens roughly 5 minutes after the timestamp Google Cloud Monitoring assigned to the datapoint that crossed the threshold, and proportionally later if a longer delay covers a longer integration interval. That gap is the cost of waiting for data that arrives late, and it is expected. A pending period adds its own duration on top - the example below waits a further 5 minutes before the issue opens.

Decide what an empty window means

A poll that lands later than usual leaves the evaluation window empty. With Treat No Data As set to Normal that is indistinguishable from a healthy evaluation and the monitor stays green, so a missed datapoint passes silently. See Treat No Data As for the alternatives and Alerting on No Data for notifying on them.

No Data applies to the query as a whole. A query grouped by resource keeps returning rows while any one resource still reports, so a single instance that stops publishing is not a No Data evaluation.

Example monitor

gcp_* metric names are built from the monitored resource and the metric type in lower snake case, for example gcp_gce_instance_compute_googleapis_com_instance_cpu_utilization, with resource labels such as project_id, zone, instance_name and instance_id available for grouping.

title: GCE Instance CPU High
display:
  header: High CPU on {{ labels.instance_name }}
  description: Instance CPU utilization has been above 90% for 5 minutes.
  resourceHeaderLabels:
    - instance_name
  contextHeaderLabels:
    - project_id
    - zone
severity: S2
measurementType: state
model:
  queries:
    - name: cpu_query
      expression: max(gcp_gce_instance_compute_googleapis_com_instance_cpu_utilization) by (project_id, zone, instance_name)
      datasourceType: prometheus
      queryType: instant
      rollup:
        function: avg
        time: 1m
      evaluationDelay: 300
  thresholds:
    - name: threshold_1
      inputName: cpu_query
      operator: gt
      values:
        - 0.9
executionErrorState: OK
noDataState: NoData
evaluationInterval:
  interval: 1m
  pendingFor: 5m

See Monitor YAML structure for the full schema.