# Scraping Metrics in Kubernetes

## Scraping Pods using Prometheus Annotations

groundcover supports standard Prometheus annotations instructing scrapers to periodically fetch metrics from pods.

Once enabled groundcover will automatically discover every pod with those annotations and periodically scrape the metrics path specified.

### Annotating Your Pods

Add the following annotations to your Kubernetes pods exposing Prometheus metrics

{% hint style="warning" %}
Ensure the relevant port is specified in the container's `containerPort`.
{% endhint %}

```yaml
annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "<port>"
  prometheus.io/path: "<metrics-path>" # Optional, defaults to /metrics
```

### Scraping from the groundcover Sensor

{% hint style="warning" %}
To avoid duplicate metric collection, you must disable the custom-metrics default jobs.

If your existing custom-metrics setup includes additional scrape jobs defined via `extraScrapeConfigs` please refer to the steps below.
{% endhint %}

In this configuration, each Sensor scrapes metrics locally from its own node. This reduces overall load and minimizes cross-availability-zone traffic.

```yaml
agent:
  sensor:
    receivers:
      metricsscraper: 
        enabled: true
custom-metrics:
  config:
    scrape_configs: []
```

### Additional Sensor Scrape Jobs

For targets that don't carry standard Prometheus annotations — or to scrape pods that the default job filters out — you can append additional scrape jobs to the sensor's metrics scraper using `extraScrapeConfigs`.

Each entry is an independent scrape job rendered as a sibling of the built-in `kubernetes-pods` job, with its own relabel pipeline. The default job's drop rules do not apply to your custom jobs.

Like the built-in sensor scrape, **`role: pod` discovery is automatically scoped to the local sensor's node**, so each sensor scrapes only the pods on its own node — no cross-availability-zone traffic and no duplicate metrics.

```yaml
agent:
  sensor:
    receivers:
      metricsscraper:
        enabled: true
        extraScrapeConfigs:
          - job_name: my-extra-job
            kubernetes_sd_configs:
              - role: pod
            relabel_configs:
              - source_labels: [__meta_kubernetes_pod_label_app]
                action: keep
                regex: my-app
              - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
                action: replace
                regex: ([^:]+)(?::\d+)?;(\d+)
                replacement: $1:$2
                target_label: __address__
```

{% hint style="warning" %}
Only `role: pod` discovery is node-scoped automatically. Other roles (`service`, `endpoints`, `node`) will run on every sensor in the cluster and produce duplicate metrics. Prefer `role: pod` unless you have a specific reason to use another role.
{% endhint %}

#### When to use this instead of `custom-metrics.extraScrapeConfigs`

|                  | `agent.sensor.receivers.metricsscraper.extraScrapeConfigs` | `custom-metrics.extraScrapeConfigs`                     |
| ---------------- | ---------------------------------------------------------- | ------------------------------------------------------- |
| Runs on          | Sensor DaemonSet (per-node)                                | A separate vmagent Deployment (cluster-wide)            |
| Discovery        | Local node only (for `role: pod`)                          | Cluster-wide                                            |
| Extra deployment | None — reuses the sensor                                   | Adds a vmagent pod                                      |
| Use case         | Per-node pod targets                                       | External targets, full-cluster discovery, non-pod roles |

#### Scraping from the sensors while keeping extraScrapeConfigs on the custom-metrics agent

{% hint style="info" %}
Below is only relevant for setups where `custom-metrics` is used with additional scrape jobs.
{% endhint %}

```yaml
agent: 
  sensor:
    receivers:
      metricsscraper: 
        enabled: true
custom-metrics:
  enabled: true
  config:
    scrape_configs: []
  extraScrapeConfigs:
   - job_name: extra-job
     scrape_interval: 10s
     scrape_timeout: 10s
     honor_timestamps: true
     basic_auth:
       username: "<user>"
       password: <pw>
     static_configs:
     - targets:
       - http://get-this-custom-metrics.com/metrics
```

## Custom Scrape Relabel Configs

groundcover exposes two configuration options to customize how targets and metrics are handled during the Prometheus scrape pipeline.

### Understanding the Two Stages

Prometheus scraping has two distinct relabeling stages, and groundcover exposes both:

| Stage                 | Helm Value                   | When It Runs          | What It Operates On                            |
| --------------------- | ---------------------------- | --------------------- | ---------------------------------------------- |
| **Target relabeling** | `customRelabelConfigs`       | **Before** the scrape | Service discovery metadata (`__meta_*` labels) |
| **Metric relabeling** | `customMetricRelabelConfigs` | **After** the scrape  | The actual scraped metric samples              |

{% hint style="warning" %}
Choosing the wrong stage is a common mistake. For example, filtering by metric name (`__name__`) only works in `customMetricRelabelConfigs` because `__name__` is only available after metrics have been scraped. Using it in `customRelabelConfigs` will silently have no effect.
{% endhint %}

### customRelabelConfigs (Pre-Scrape)

Use `customRelabelConfigs` to control **which targets** are scraped. These rules are appended to the default `relabel_configs` of the built-in `kubernetes-pods` scrape job.

At this stage, you have access to Kubernetes service discovery labels such as:

* `__meta_kubernetes_pod_annotation_*`
* `__meta_kubernetes_pod_label_*`
* `__meta_kubernetes_namespace`
* `__meta_kubernetes_pod_name`
* `__meta_kubernetes_pod_node_name`

#### Common Use Cases

**Only scrape pods with a specific label:**

```yaml
agent:
  sensor:
    receivers:
      metricsscraper:
        customRelabelConfigs:
          - source_labels: [__meta_kubernetes_pod_label_app]
            action: keep
            regex: 'my-important-app'
```

**Drop targets from a specific namespace:**

```yaml
agent:
  sensor:
    receivers:
      metricsscraper:
        customRelabelConfigs:
          - source_labels: [__meta_kubernetes_namespace]
            action: drop
            regex: 'noisy-namespace'
```

**Add a custom label from a pod annotation:**

```yaml
agent:
  sensor:
    receivers:
      metricsscraper:
        customRelabelConfigs:
          - source_labels: [__meta_kubernetes_pod_annotation_team]
            target_label: team
            action: replace
```

### customMetricRelabelConfigs (Post-Scrape)

Use `customMetricRelabelConfigs` to filter or modify **metrics after they have been scraped**. These rules create a `metric_relabel_configs` block on the built-in `kubernetes-pods` scrape job.

At this stage, you have access to the actual metric labels, including:

* `__name__` (the metric name)
* Any labels present on the scraped metrics (e.g., `namespace`, `pod`, `container`)

#### Common Use Cases

**Drop all metrics matching a prefix (e.g., high-cardinality envoy metrics):**

```yaml
agent:
  sensor:
    receivers:
      metricsscraper:
        customMetricRelabelConfigs:
          - source_labels: [__name__]
            regex: 'envoy_.*'
            action: drop
```

**Keep only specific metrics:**

```yaml
agent:
  sensor:
    receivers:
      metricsscraper:
        customMetricRelabelConfigs:
          - source_labels: [__name__]
            regex: 'up|process_cpu_seconds_total|http_requests_total'
            action: keep
```

**Drop metrics from a specific namespace:**

```yaml
agent:
  sensor:
    receivers:
      metricsscraper:
        customMetricRelabelConfigs:
          - source_labels: [namespace]
            regex: 'kube-system'
            action: drop
```

### Combining Both Stages

You can use both options together. For example, to only scrape pods with a specific label **and** drop noisy metrics from the results:

```yaml
agent:
  sensor:
    receivers:
      metricsscraper:
        customRelabelConfigs:
          - source_labels: [__meta_kubernetes_pod_label_team]
            action: keep
            regex: 'platform'
        customMetricRelabelConfigs:
          - source_labels: [__name__]
            regex: 'envoy_.*'
            action: drop
```

For more details on Prometheus relabeling actions and syntax, see the [Prometheus relabel\_config documentation](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config).

## Scraping Prometheus CRDs

groundcover supports scraping Prometheus metrics based on CRDs like PodMonitor and ServiceMonitor. This is achieved using the VictoriaMetrics Operator.

### Enable VictoriaMetrics Operator

Deploy the `victoria-metrics-operator` before enabling the built-in vmagent.

{% hint style="warning" %}
Deploy the operator first. Enable `builtinVMAgent` only in a separate step to prevent issues (race between the operator and the custom resource deployment).
{% endhint %}

#### Step 1 - Enable only the operator:

```yaml
victoria-metrics-operator:
  enabled: true
# Do not enable builtinVMAgent at this stage.
```

#### Step 2 - Enable builtinVMAgent with a second deployment:

```yaml
victoria-metrics-operator:
  enabled: true
  builtinVMAgent:
    enabled: true
    # By default, all ServiceMonitor, PodMonitor, PrometheusRule, and Probe CRDs
    # in all namespaces are scraped automatically.
    # To limit scope, override selectors as documented:
    # https://docs.victoriametrics.com/operator/resources/vmagent/#scraping
    # spec:
    #   selectAllByDefault: true
    #   serviceScrapeNamespaceSelector: {}
    #   podScrapeNamespaceSelector: {}
    #   podScrapeSelector: {}
    #   serviceScrapeSelector: {}
    #   nodeScrapeSelector: {}
    #   nodeScrapeNamespaceSelector: {}
    #   staticScrapeSelector: {}
    #   staticScrapeNamespaceSelector: {}
```

#### ArgoCD Integration

If deploying monitoring CRDs via ArgoCD, add this override to prevent sync conflicts:

```yaml
victoria-metrics-operator:
  operator:
    prometheus_converter_add_argocd_ignore_annotations: true
```

This instructs the operator to add ArgoCD ignore annotations for Prometheus CRDs.

### Deploy Monitoring CRDs

The operator automatically discovers and scrapes any deployed Prometheus CRDs (`ServiceMonitor`, `PodMonitor`, `PrometheusRule`, `Probe`).

### Example - PodMonitor

Create `my-test-podmonitor.yaml`:

```yaml
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: scrape-demo
spec:
  selector:
    matchLabels:
      # <your app labels>
  podMetricsEndpoints:
    - port: <metrics port>
  namespaceSelector:
    matchNames:
      - <namespace>
```

Apply it:

```bash
kubectl apply -n groundcover -f my-test-podmonitor.yaml
```

The vmagent will automatically reload and begin scraping the new target. Metrics should appear in groundcover’s Grafana dashboard soon after.

VictoriaMetrics Operator also supports its own CRDs; details are [available here](https://docs.victoriametrics.com/operator/VictoriaMetrics-Operator.html#overview).

## Setting up Additional Scrape Jobs

groundcover supports setting up additional Prometheus metric scrapingvia a built in [vmagent](https://docs.victoriametrics.com/vmagent.html) component (VictoriaMetrics). vmagent supports standard [Prometheus scrape job configs](https://docs.victoriametrics.com/sd_configs.html#supported-service-discovery-configs).

The jobs are defined as an array under the `custom-metrics` section of the helm chart.

```yaml
custom-metrics:
  enabled: true
  extraScrapeConfigs: []
```

### Example - Scraping cadvisor metrics

{% hint style="info" %}
By default, only key metrics are collected to limit metric cardinality. Edit the `regex` to include additional metrics as needed
{% endhint %}

```yaml
custom-metrics:
  enabled: true
  extraScrapeConfigs:
    - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      job_name: kubernetes-nodes-cadvisor
      scrape_interval: 1m
      kubernetes_sd_configs:
        - role: node
      relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - replacement: kubernetes.default.svc:443
          target_label: __address__
        - regex: (.+)
          replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
          source_labels:
            - __meta_kubernetes_node_name
          target_label: __metrics_path__
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      metric_relabel_configs:
      - source_labels: [__name__]
        action: keep
        regex: '(container_cpu_usage_seconds_total|container_memory_working_set_bytes)'


```

### Example - Scraping Full KSM Metrics

By default, groundcover exports a curated set of kube-state-metrics (KSM) metrics that power native dashboards and screens. These metrics are prefixed with `groundcover_` to differentiate them from other KSM deployments.

To collect the complete set of kube-state-metrics (KSM) metrics in groundcover, follow these steps:

1. Enable all KSM collectors in the ksm deployment.
2. Enable custom metrics scraping.
3. Add a scrape job for the KSM metrics.

```yaml
kube-state-metrics:
  collectors:
    - certificatesigningrequests
    - configmaps
    - cronjobs
    - daemonsets
    - deployments
    - endpoints
    - horizontalpodautoscalers
    - ingresses
    - jobs
    - leases
    - limitranges
    - mutatingwebhookconfigurations
    - namespaces
    - networkpolicies
    - nodes
    - persistentvolumeclaims
    - persistentvolumes
    - poddisruptionbudgets
    - pods
    - replicasets
    - replicationcontrollers
    - resourcequotas
    - secrets
    - services
    - statefulsets
    - storageclasses
    - validatingwebhookconfigurations
    - volumeattachments

custom-metrics:
  enabled: true
  extraScrapeConfigs:
    - job_name: groundcover-kube-state-metrics
      honor_timestamps: true
      honor_labels: true
      scrape_interval: 1m
      scrape_timeout: 1m
      metrics_path: /metrics
      scheme: http
      static_configs:
      - targets:
        - groundcover-kube-state-metrics.groundcover.svc.cluster.local:8080

```

### Metrics Cardinality Limits

groundcover limits the cardinality of the metrics collected from custom jobs.

Defaults:

```yaml
remoteWrite.maxDailySeries: "5000000" # Max daily series churn
remoteWrite.maxHourlySeries: "1000000" # Max active series
```

To raise these limits:

{% hint style="warning" %}
Higher values need more CPU/memory. Monitor resource usage and adjust `custom-metrics` resources if increased.
{% endhint %}

```yaml
custom-metrics:
  enabled: true
  extraArgs:
    remoteWrite.maxDailySeries: "<desired value>"
    remoteWrite.maxHourlySeries: "<desired value>"
```

### Resource Configuration

Defaults:

```yaml
custom-metrics:
  resources:
    limits:
      memory: 1024Mi
    requests:
      cpu: 100m
      memory: 256Mi
```

```yaml
custom-metrics:
  resources:
    limits:
      cpu: <>
      memory: <>
    requests:
      cpu: <>
      memory: <>
```

## Common Questions

### How often are my metrics scraped?

Metrics are automatically scraped every `10 seconds` unless explicitly specified in the scrape job.

### Where can I find scraped metrics?

All metrics in the platform can be found in the metrics exploration page:\
<https://app.groundcover.com/explore/data-explorer>


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.groundcover.com/integrations/data-sources/prometheus/scrape-using-sensor.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.