> For the complete documentation index, see [llms.txt](https://docs.groundcover.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.groundcover.com/~/revisions/ETrLpNk6KtHjyaVUTLoE/use-groundcover/scraping-custom-metrics.md).

# Scraping custom metrics

groundcover can scrape your custom metrics by deploying a metrics scraper ([vmagent](https://docs.victoriametrics.com/vmagent.html) by Victoria Metrics) that will automatically scrape prometheus targets.

vmagent is fully compatible with prometheus scrape job syntax - more can be found [here](https://docs.victoriametrics.com/sd_configs.html#supported-service-discovery-configs).

#### Enabling custom metrics scraping

The following helm override enables custom metrics scraping

```yaml
custom-metrics:
  enabled: true
```

## Using CLI

Scrape your custom metrics using groundcover CLI (using default scrape jobs):

```bash
groundcover deploy --custom-metrics
```

## Using Helm

#### Using Helm (Upgrading existing installation)

Either create a new custom-values.yaml or edit your existing groundcover `values.yaml`

```bash
helm upgrade groundcover groundcover/groundcover -n groundcover -f values.yaml --reuse-values
```

## Autodiscovery

Ensure that the Kubernetes resources that contain your Prometheus exporters have been deployed with the following annotations to enable scraping

```yaml
annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "<port>"
  prometheus.io/path: "<metrics-path>" // Optional, if path is not /metrics
```

By default, the following scrape jobs are deployed when enabling custom-metrics:

{% hint style="warning" %}
Starting November 25th 2024, `kubernetes-pods` scraping job will be the only scrape job that is enabled out of the box when activating custom metrics scraping.\
You can add back the legacy scrape jobs under `extraScrapeConfigs` section as described in [Scraping custom metrics](/~/revisions/ETrLpNk6KtHjyaVUTLoE/use-groundcover/scraping-custom-metrics.md#using-custom-scrape-jobs)
{% endhint %}

<pre class="language-yaml"><code class="lang-yaml">custom-metrics:
  enabled: true
  config:
    scrape_configs:
      - job_name: "kubernetes-apiservers"
        honor_labels: true  
        kubernetes_sd_configs:
          - role: endpoints
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        relabel_configs:
          - action: drop
            source_labels:
              - __meta_kubernetes_pod_label_app_kubernetes_io_part_of
            regex: groundcover
          - source_labels:
              [
                __meta_kubernetes_namespace,
                __meta_kubernetes_service_name,
                __meta_kubernetes_endpoint_port_name,
              ]
            action: keep
            regex: default;kubernetes;https
      - job_name: "kubernetes-nodes"
        honor_labels: true
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
          - role: node
        relabel_configs:
          - action: drop
            source_labels:
              - __meta_kubernetes_pod_label_app_kubernetes_io_part_of
            regex: groundcover
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
          - target_label: __address__
            replacement: kubernetes.default.svc:443
          - source_labels: [__meta_kubernetes_node_name]
            regex: (.+)
            target_label: __metrics_path__
            replacement: /api/v1/nodes/$1/proxy/metrics
      - job_name: "kubernetes-service-endpoints"
        honor_labels: true
        kubernetes_sd_configs:
          - role: endpointslices
        relabel_configs:
          - action: drop
            source_labels:
              - __meta_kubernetes_pod_label_app_kubernetes_io_part_of
            regex: groundcover
          - action: drop
            source_labels: [__meta_kubernetes_pod_container_init]
            regex: true
          - source_labels:
              [__meta_kubernetes_service_annotation_prometheus_io_scrape]
            action: keep
            regex: true
          - source_labels:
              [__meta_kubernetes_service_annotation_prometheus_io_scheme]
            action: replace
            target_label: __scheme__
            regex: (https?)
          - source_labels:
              [__meta_kubernetes_service_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels:
              [
                __address__,
                __meta_kubernetes_service_annotation_prometheus_io_port,
              ]
            action: replace
            target_label: __address__
            regex: ([^:]+)(?::\d+)?;(\d+)
            replacement: $1:$2
          - action: labelmap
            regex: __meta_kubernetes_service_label_(.+)
          - source_labels: [__meta_kubernetes_pod_name]
            target_label: pod
          - source_labels: [__meta_kubernetes_pod_container_name]
            target_label: container
          - source_labels: [__meta_kubernetes_namespace]
            target_label: namespace
          - source_labels: [__meta_kubernetes_service_name]
            target_label: service
          - source_labels: [__meta_kubernetes_service_name]
            target_label: job
            replacement: ${1}
          - source_labels: [__meta_kubernetes_pod_node_name]
            action: replace
            target_label: node
      - job_name: "kubernetes-service-endpoints-slow"
        honor_labels: true
        scrape_interval: 5m
        scrape_timeout: 30s
        kubernetes_sd_configs:
          - role: endpointslices
        relabel_configs:
          - action: drop
            source_labels:
              - __meta_kubernetes_pod_label_app_kubernetes_io_part_of
            regex: groundcover
          - action: drop
            source_labels: [__meta_kubernetes_pod_container_init]
            regex: true
          - source_labels:
              [__meta_kubernetes_service_annotation_prometheus_io_scrape_slow]
<strong>            action: keep
</strong>            regex: true
          - source_labels:
              [__meta_kubernetes_service_annotation_prometheus_io_scheme]
            action: replace
            target_label: __scheme__
            regex: (https?)
          - source_labels:
              [__meta_kubernetes_service_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels:
              [
                __address__,
                __meta_kubernetes_service_annotation_prometheus_io_port,
              ]
            action: replace
            target_label: __address__
            regex: ([^:]+)(?::\d+)?;(\d+)
            replacement: $1:$2
          - action: labelmap
            regex: __meta_kubernetes_service_label_(.+)
          - source_labels: [__meta_kubernetes_pod_name]
            target_label: pod
          - source_labels: [__meta_kubernetes_pod_container_name]
            target_label: container
          - source_labels: [__meta_kubernetes_namespace]
            target_label: namespace
          - source_labels: [__meta_kubernetes_service_name]
            target_label: service
          - source_labels: [__meta_kubernetes_service_name]
            target_label: job
            replacement: ${1}
          - source_labels: [__meta_kubernetes_pod_node_name]
            action: replace
            target_label: node
      - job_name: "kubernetes-services"
        honor_labels: true
        metrics_path: /probe
        params:
          module: [http_2xx]
        kubernetes_sd_configs:
          - role: service
        relabel_configs:
          - action: drop
            source_labels:
              - __meta_kubernetes_pod_label_app_kubernetes_io_part_of
            regex: groundcover
          - source_labels:
              [__meta_kubernetes_service_annotation_prometheus_io_probe]
            action: keep
            regex: true
          - source_labels: [__address__]
            target_label: __param_target
          - target_label: __address__
            replacement: blackbox
          - source_labels: [__param_target]
            target_label: instance
          - action: labelmap
            regex: __meta_kubernetes_service_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            target_label: namespace
          - source_labels: [__meta_kubernetes_service_name]
            target_label: service
      - job_name: "kubernetes-pods"
        honor_labels: true
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - action: drop
            source_labels:
              - __meta_kubernetes_pod_label_app_kubernetes_io_part_of
            regex: groundcover
          - action: drop
            source_labels: [__meta_kubernetes_pod_container_init]
            regex: true
          - source_labels:
              [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
            action: keep
            regex: true
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels:
              [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
            action: replace
            regex: ([^:]+)(?::\d+)?;(\d+)
            replacement: $1:$2
            target_label: __address__
          - action: labelmap
            regex: __meta_kubernetes_pod_label_(.+)
          - source_labels: [__meta_kubernetes_pod_name]
            target_label: pod
          - source_labels: [__meta_kubernetes_pod_container_name]
            target_label: container
          - source_labels: [__meta_kubernetes_namespace]
            target_label: namespace
          - source_labels: [__meta_kubernetes_service_name]
            target_label: service
          - source_labels: [__meta_kubernetes_service_name]
            target_label: job
            replacement: ${1}
          - source_labels: [__meta_kubernetes_pod_node_name]
            action: replace
            target_label: node

</code></pre>

### Disable autodiscovery scrape jobs

In case you're interested in disabling autodiscovery scrape jobs, provide the below override

Disabling custom-metrics scrape jobs allows you to scale the custom-metrics deployment horizontally.

```yaml
custom-metrics:
  enabled: true
  config:
    scrape_configs: []
```

### Using custom scrape jobs

in case you're interested in deploying custom scrape jobs, create/add the following override

```yaml
custom-metrics:
  enabled: true
  extraScrapeConfigs:
  - job_name: "custom-scrape-job"
  ...
```

## Cardinality limits

In order to safeguard groundcover's performance, there are default limitations on metrics ingestion in place.

```yaml
remoteWrite.maxDailySeries: "1000000" # limiting daily churn rate.
remoteWrite.maxHourlySeries: "100000" # limiting the number of active time series.
```

### Increasing metrics cardinality

In order to increase metrics resolution, you can implement the following overrides

{% hint style="warning" %}
Increasing cardinality parameters will increase memory/cpu consumption and might cause OOMKills/CPU Throttling.

Please use with caution and increase the custom metrics agent/metrics server resources accordingly
{% endhint %}

```yaml
custom-metrics:
  enabled: true
  extraArgs:
    remoteWrite.maxDailySeries: "<custom value>"
    remoteWrite.maxHourlySeries: "<custom value>"
```

In case you wish to increase metrics server / custom metrics resources, use the following overrides:

```yaml
custom-metrics:
  resources:
    limits:
      cpu: <>
      memory: <>
    requests:
      cpu: <>
      memory: <>
```


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.groundcover.com/~/revisions/ETrLpNk6KtHjyaVUTLoE/use-groundcover/scraping-custom-metrics.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.