Operator based metrics

guide on how to enable crd-based scraping targets

Step 1 - Enable custom metrics and victoria-metrics-operator

1. Enable custom metrics scraping and victoria-metrics-operator

Please note that due to operator limitations, it's necessary to deploy it prior to deploying he builtinVMAgent

• First, enable only the operator

custom-metrics:
  enabled: true
victoria-metrics-operator:
# Do not enable the operator and the builtinVMAgent at the same time
# Deploy groundcover once with the operator enabled, and then add the builtinVMAgent section
  enabled: true

• Then, if enabling builtinVMAgent - enable it in a separate deployment step

custom-metrics:
  enabled: true
victoria-metrics-operator:
  enabled: true
   builtinVMAgent:
    enabled: true
# Optional:  uncomment this section to auto discover Monitor objects from the entire cluster
# Default behavior is to reconcile objects in the groundcover namespace
#    spec:
#      nodeScrapeSelector: null
#      podScrapeSelector: null
#      probeSelector: null
#      selectAllByDefault: true
#      serviceScrapeSelector: null
#      staticScrapeSelector: null

Step 2 - Deploy Monitoring CRDs

By default the vm operator will identify the Prometheus CRDs (ServiceMonitor, PodMonitor, PrometheusRule and Probe ) that are already deployed and will scrape them automatically.

In case you want to deploy a test monitor object, here is an example using PodMonitor

1. Create the following my-test-podmonitor.yaml

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: scrape-demo
spec:
  selector:
    matchLabels:
      {app labels}
  podMetricsEndpoints:
  - port: {app metrics port}
  namespaceSelector:
      matchNames:
        - {pod namespace}

1. Deploy it

kubectl apply -n groundcover -f my-test-podmonitor.yaml

1. The vmagent will reload its configuration and will start scraping the target, metrics should appear in groundcover's grafana shortly after.

Other then supporting the standard Prometheus CRDs, VictoriaMetrics operator has their own proprietary CRDs that can be used, more about them can be found here.