Scraping Metrics in Kubernetes

Automatically scrape metrics from pods and other services

Scraping Pods using Prometheus Annotations

groundcover supports standard Prometheus annotations instructing scrapers to periodically fetch metrics from pods.

Once enabled groundcover will automatically discover every pod with those annotations and periodically scrape the metrics path specified.

Annotating Your Pods

Add the following annotations to your Kubernetes pods exposing Prometheus metrics

Ensure the relevant port is specified in the container's containerPort.

annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "<port>"
  prometheus.io/path: "<metrics-path>" # Optional, defaults to /metrics

Scraping from the groundcover Sensor

To avoid duplicate metric collection, you must disable the custom-metrics default jobs.

If your existing custom-metrics setup includes additional scrape jobs defined via extraScrapeConfigs please refer to the steps below.

In this configuration, each Sensor scrapes metrics locally from its own node. This reduces overall load and minimizes cross-availability-zone traffic.

agent:
  sensor:
    receivers:
      metricsscraper: 
        enabled: true
custom-metrics:
  config:
    scrape_configs: []

Additional Sensor Scrape Jobs

For targets that don't carry standard Prometheus annotations — or to scrape pods that the default job filters out — you can append additional scrape jobs to the sensor's metrics scraper using extraScrapeConfigs.

Each entry is an independent scrape job rendered as a sibling of the built-in kubernetes-pods job, with its own relabel pipeline. The default job's drop rules do not apply to your custom jobs.

Like the built-in sensor scrape, role: pod discovery is automatically scoped to the local sensor's node, so each sensor scrapes only the pods on its own node — no cross-availability-zone traffic and no duplicate metrics.

agent:
  sensor:
    receivers:
      metricsscraper:
        enabled: true
        extraScrapeConfigs:
          - job_name: my-extra-job
            kubernetes_sd_configs:
              - role: pod
            relabel_configs:
              - source_labels: [__meta_kubernetes_pod_label_app]
                action: keep
                regex: my-app
              - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
                action: replace
                regex: ([^:]+)(?::\d+)?;(\d+)
                replacement: $1:$2
                target_label: __address__

Only role: pod discovery is node-scoped automatically. Other roles (service, endpoints, node) will run on every sensor in the cluster and produce duplicate metrics. Prefer role: pod unless you have a specific reason to use another role.

When to use this instead of custom-metrics.extraScrapeConfigs

	agent.sensor.receivers.metricsscraper.extraScrapeConfigs	custom-metrics.extraScrapeConfigs
Runs on	Sensor DaemonSet (per-node)	A separate vmagent Deployment (cluster-wide)
Discovery	Local node only (for role: pod)	Cluster-wide
Extra deployment	None — reuses the sensor	Adds a vmagent pod
Use case	Per-node pod targets	External targets, full-cluster discovery, non-pod roles

Scraping from the sensors while keeping extraScrapeConfigs on the custom-metrics agent

Below is only relevant for setups where custom-metrics is used with additional scrape jobs.

agent: 
  sensor:
    receivers:
      metricsscraper: 
        enabled: true
custom-metrics:
  enabled: true
  config:
    scrape_configs: []
  extraScrapeConfigs:
   - job_name: extra-job
     scrape_interval: 10s
     scrape_timeout: 10s
     honor_timestamps: true
     basic_auth:
       username: "<user>"
       password: <pw>
     static_configs:
     - targets:
       - http://get-this-custom-metrics.com/metrics

Custom Scrape Relabel Configs

groundcover exposes two configuration options to customize how targets and metrics are handled during the Prometheus scrape pipeline.

Understanding the Two Stages

Prometheus scraping has two distinct relabeling stages, and groundcover exposes both:

Stage	Helm Value	When It Runs	What It Operates On
Target relabeling	customRelabelConfigs	Before the scrape	Service discovery metadata (__meta_* labels)
Metric relabeling	customMetricRelabelConfigs	After the scrape	The actual scraped metric samples

Choosing the wrong stage is a common mistake. For example, filtering by metric name (__name__) only works in customMetricRelabelConfigs because __name__ is only available after metrics have been scraped. Using it in customRelabelConfigs will silently have no effect.

customRelabelConfigs (Pre-Scrape)

Use customRelabelConfigs to control which targets are scraped. These rules are appended to the default relabel_configs of the built-in kubernetes-pods scrape job.

At this stage, you have access to Kubernetes service discovery labels such as:

• __meta_kubernetes_pod_annotation_*

• __meta_kubernetes_pod_label_*

• __meta_kubernetes_namespace

• __meta_kubernetes_pod_name

• __meta_kubernetes_pod_node_name

Common Use Cases

Only scrape pods with a specific label:

agent:
  sensor:
    receivers:
      metricsscraper:
        customRelabelConfigs:
          - source_labels: [__meta_kubernetes_pod_label_app]
            action: keep
            regex: 'my-important-app'

Drop targets from a specific namespace:

agent:
  sensor:
    receivers:
      metricsscraper:
        customRelabelConfigs:
          - source_labels: [__meta_kubernetes_namespace]
            action: drop
            regex: 'noisy-namespace'

Add a custom label from a pod annotation:

agent:
  sensor:
    receivers:
      metricsscraper:
        customRelabelConfigs:
          - source_labels: [__meta_kubernetes_pod_annotation_team]
            target_label: team
            action: replace

customMetricRelabelConfigs (Post-Scrape)

Use customMetricRelabelConfigs to filter or modify metrics after they have been scraped. These rules create a metric_relabel_configs block on the built-in kubernetes-pods scrape job.

At this stage, you have access to the actual metric labels, including:

• __name__ (the metric name)

• Any labels present on the scraped metrics (e.g., namespace, pod, container)

Common Use Cases

Drop all metrics matching a prefix (e.g., high-cardinality envoy metrics):

agent:
  sensor:
    receivers:
      metricsscraper:
        customMetricRelabelConfigs:
          - source_labels: [__name__]
            regex: 'envoy_.*'
            action: drop

Keep only specific metrics:

agent:
  sensor:
    receivers:
      metricsscraper:
        customMetricRelabelConfigs:
          - source_labels: [__name__]
            regex: 'up|process_cpu_seconds_total|http_requests_total'
            action: keep

Drop metrics from a specific namespace:

agent:
  sensor:
    receivers:
      metricsscraper:
        customMetricRelabelConfigs:
          - source_labels: [namespace]
            regex: 'kube-system'
            action: drop

Combining Both Stages

You can use both options together. For example, to only scrape pods with a specific label and drop noisy metrics from the results:

agent:
  sensor:
    receivers:
      metricsscraper:
        customRelabelConfigs:
          - source_labels: [__meta_kubernetes_pod_label_team]
            action: keep
            regex: 'platform'
        customMetricRelabelConfigs:
          - source_labels: [__name__]
            regex: 'envoy_.*'
            action: drop

For more details on Prometheus relabeling actions and syntax, see the Prometheus relabel_config documentation.

Scraping Prometheus CRDs

groundcover supports scraping Prometheus metrics based on CRDs like PodMonitor and ServiceMonitor. This is achieved using the VictoriaMetrics Operator.

Enable VictoriaMetrics Operator

Deploy the victoria-metrics-operator before enabling the built-in vmagent.

Deploy the operator first. Enable builtinVMAgent only in a separate step to prevent issues (race between the operator and the custom resource deployment).

Step 1 - Enable only the operator:

victoria-metrics-operator:
  enabled: true
# Do not enable builtinVMAgent at this stage.

Step 2 - Enable builtinVMAgent with a second deployment:

victoria-metrics-operator:
  enabled: true
  builtinVMAgent:
    enabled: true
    # By default, all ServiceMonitor, PodMonitor, PrometheusRule, and Probe CRDs
    # in all namespaces are scraped automatically.
    # To limit scope, override selectors as documented:
    # https://docs.victoriametrics.com/operator/resources/vmagent/#scraping
    # spec:
    #   selectAllByDefault: true
    #   serviceScrapeNamespaceSelector: {}
    #   podScrapeNamespaceSelector: {}
    #   podScrapeSelector: {}
    #   serviceScrapeSelector: {}
    #   nodeScrapeSelector: {}
    #   nodeScrapeNamespaceSelector: {}
    #   staticScrapeSelector: {}
    #   staticScrapeNamespaceSelector: {}

ArgoCD Integration

If deploying monitoring CRDs via ArgoCD, add this override to prevent sync conflicts:

victoria-metrics-operator:
  operator:
    prometheus_converter_add_argocd_ignore_annotations: true

This instructs the operator to add ArgoCD ignore annotations for Prometheus CRDs.

Deploy Monitoring CRDs

The operator automatically discovers and scrapes any deployed Prometheus CRDs (ServiceMonitor, PodMonitor, PrometheusRule, Probe).

Example - PodMonitor

Create my-test-podmonitor.yaml:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: scrape-demo
spec:
  selector:
    matchLabels:
      # <your app labels>
  podMetricsEndpoints:
    - port: <metrics port>
  namespaceSelector:
    matchNames:
      - <namespace>

Apply it:

kubectl apply -n groundcover -f my-test-podmonitor.yaml

The vmagent will automatically reload and begin scraping the new target. Metrics should appear in groundcover’s Grafana dashboard soon after.

VictoriaMetrics Operator also supports its own CRDs; details are available here.

Setting up Additional Scrape Jobs

groundcover supports setting up additional Prometheus metric scrapingvia a built in vmagent component (VictoriaMetrics). vmagent supports standard Prometheus scrape job configs.

The jobs are defined as an array under the custom-metrics section of the helm chart.

custom-metrics:
  enabled: true
  extraScrapeConfigs: []

Example - Scraping cadvisor metrics

By default, only key metrics are collected to limit metric cardinality. Edit the regex to include additional metrics as needed

custom-metrics:
  enabled: true
  extraScrapeConfigs:
    - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      job_name: kubernetes-nodes-cadvisor
      scrape_interval: 1m
      kubernetes_sd_configs:
        - role: node
      relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - replacement: kubernetes.default.svc:443
          target_label: __address__
        - regex: (.+)
          replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
          source_labels:
            - __meta_kubernetes_node_name
          target_label: __metrics_path__
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      metric_relabel_configs:
      - source_labels: [__name__]
        action: keep
        regex: '(container_cpu_usage_seconds_total|container_memory_working_set_bytes)'

Example - Scraping Full KSM Metrics

By default, groundcover exports a curated set of kube-state-metrics (KSM) metrics that power native dashboards and screens. These metrics are prefixed with groundcover_ to differentiate them from other KSM deployments.

To collect the complete set of kube-state-metrics (KSM) metrics in groundcover, follow these steps:

1. Enable all KSM collectors in the ksm deployment.

2. Enable custom metrics scraping.

3. Add a scrape job for the KSM metrics.

kube-state-metrics:
  collectors:
    - certificatesigningrequests
    - configmaps
    - cronjobs
    - daemonsets
    - deployments
    - endpoints
    - horizontalpodautoscalers
    - ingresses
    - jobs
    - leases
    - limitranges
    - mutatingwebhookconfigurations
    - namespaces
    - networkpolicies
    - nodes
    - persistentvolumeclaims
    - persistentvolumes
    - poddisruptionbudgets
    - pods
    - replicasets
    - replicationcontrollers
    - resourcequotas
    - secrets
    - services
    - statefulsets
    - storageclasses
    - validatingwebhookconfigurations
    - volumeattachments

custom-metrics:
  enabled: true
  extraScrapeConfigs:
    - job_name: groundcover-kube-state-metrics
      honor_timestamps: true
      honor_labels: true
      scrape_interval: 1m
      scrape_timeout: 1m
      metrics_path: /metrics
      scheme: http
      static_configs:
      - targets:
        - groundcover-kube-state-metrics.groundcover.svc.cluster.local:8080

Metrics Cardinality Limits

groundcover limits the cardinality of the metrics collected from custom jobs.

Defaults:

remoteWrite.maxDailySeries: "5000000" # Max daily series churn
remoteWrite.maxHourlySeries: "1000000" # Max active series

To raise these limits:

Higher values need more CPU/memory. Monitor resource usage and adjust custom-metrics resources if increased.

custom-metrics:
  enabled: true
  extraArgs:
    remoteWrite.maxDailySeries: "<desired value>"
    remoteWrite.maxHourlySeries: "<desired value>"

Resource Configuration

Defaults:

custom-metrics:
  resources:
    limits:
      memory: 1024Mi
    requests:
      cpu: 100m
      memory: 256Mi

custom-metrics:
  resources:
    limits:
      cpu: <>
      memory: <>
    requests:
      cpu: <>
      memory: <>

Common Questions

How often are my metrics scraped?

Metrics are automatically scraped every 10 seconds unless explicitly specified in the scrape job.

Where can I find scraped metrics?

All metrics in the platform can be found in the metrics exploration page: https://app.groundcover.com/explore/data-explorer