Scraping Metrics with Sensor

The groundcover sensors spread across your environment can be used to scrape metrics, to refrain from having a large load on a single scraper and to minimize cross-AZ traffic.

Enabling Metrics-Scraping on Sensor

To enable scraping using the sensor, use the following values:

agent:
  sensor:
    metricIngestor:
      scraperEnabled: true

Notice that if custom-metrics pod-specific metrics scraping was previously in use, it should be disabled in order to refrain from having the same metrics scraped twice. To do that, remove the scrape jobs that sensor performs from the dedicated custom-metrics pod configuration.

Scraping custom metrics may increase sensor's resource consumption, depending on the configuration, the cardinality of the scraped metrics, and the number of scraped pods.

Since metrics are usually spread out across all nodes, this is usually negligible when compared to a global custom-metrics scraper.

Configuring Scrape Jobs

Sensor's scrape jobs are configured in its sensor-scrape-configuration configmap, under metrics-scrape-config.yaml. If you're running sensor on a node, it can be found at /opt/groundcover/scrape-config/metrics-scrape-config.yaml.

This configuration file is fully compatible with prometheus scrape job syntax - more can be found here.

It's best practice to use the sensor to scrape pod-specific metrics, to minimize cross-AZ traffic.

Sensor automatically adds a node selector to the discovery process, resulting in every sensor scraping metrics only from pods on the same node. This is only true for metrics under the pod role.

Using the sensor to scrape non-pod-specific metrics is discouraged, because it will result in multiple sensors scraping the same resource.

The default configuration contains discovery and scraping of pod-specific metrics:

global:
  scrape_interval: 10s
scrape_configs:
  - job_name: "kubernetes-pods"
    honor_labels: true
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - action: drop
        source_labels:
          - __meta_kubernetes_pod_label_app_kubernetes_io_part_of
        regex: groundcover
      - action: drop
        source_labels:
          - __meta_kubernetes_pod_label_app_kubernetes_io_name
          - __meta_kubernetes_pod_label_app_groundcover_com_owner
        regex: kube-state-metrics;groundcover
      - action: drop
        source_labels: [__meta_kubernetes_pod_container_init]
        regex: true
      - source_labels:
          [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels:
          [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod
      - source_labels: [__meta_kubernetes_pod_container_name]
        target_label: container
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_service_name]
        target_label: service
      - source_labels: [__meta_kubernetes_service_name]
        target_label: job
        replacement: ${1}
      - source_labels: [__meta_kubernetes_pod_node_name]
        action: replace
        target_label: node