Scraping Metrics with Sensor
The groundcover sensors spread across your environment can be used to scrape metrics, to refrain from having a large load on a single scraper and to minimize cross-AZ traffic.
Enabling Metrics-Scraping on Sensor
To enable scraping using the sensor, use the following values:
agent:
sensor:
metricIngestor:
scraperEnabled: true
Notice that if custom-metrics pod-specific metrics scraping was previously in use, it should be disabled in order to refrain from having the same metrics scraped twice. To do that, remove the scrape jobs that sensor performs from the dedicated custom-metrics pod configuration.
Configuring Scrape Jobs
Sensor's scrape jobs are configured in its sensor-scrape-configuration
configmap, under metrics-scrape-config.yaml
. If you're running sensor on a node, it can be found at /opt/groundcover/scrape-config/metrics-scrape-config.yaml
.
This configuration file is fully compatible with prometheus scrape job syntax - more can be found here.
It's best practice to use the sensor to scrape pod-specific metrics, to minimize cross-AZ traffic.
Using the sensor to scrape non-pod-specific metrics is discouraged, because it will result in multiple sensors scraping the same resource.
The default configuration contains discovery and scraping of pod-specific metrics:
global:
scrape_interval: 10s
scrape_configs:
- job_name: "kubernetes-pods"
honor_labels: true
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: drop
source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_part_of
regex: groundcover
- action: drop
source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_name
- __meta_kubernetes_pod_label_app_groundcover_com_owner
regex: kube-state-metrics;groundcover
- action: drop
source_labels: [__meta_kubernetes_pod_container_init]
regex: true
- source_labels:
[__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels:
[__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- source_labels: [__meta_kubernetes_pod_name]
target_label: pod
- source_labels: [__meta_kubernetes_pod_container_name]
target_label: container
- source_labels: [__meta_kubernetes_namespace]
target_label: namespace
- source_labels: [__meta_kubernetes_service_name]
target_label: service
- source_labels: [__meta_kubernetes_service_name]
target_label: job
replacement: ${1}
- source_labels: [__meta_kubernetes_pod_node_name]
action: replace
target_label: node
Last updated