Operator based metrics

guide on how to enable crd-based scraping targets

Step 1 - Enable victoria-metrics-operator

1. Enable victoria-metrics-operator

Please note that due to operator limitations, it's necessary to deploy it prior to deploying the builtinVMAgent

• First, enable only the operator

victoria-metrics-operator:
# Do not enable the operator and the builtinVMAgent at the same time
# Deploy groundcover once with the operator enabled, and then add the builtinVMAgent section
  enabled: true

• Then, if enabling builtinVMAgent - enable it in a separate deployment step

victoria-metrics-operator:
  enabled: true
  builtinVMAgent:
    enabled: true
# By default, all scrape CRDs (ServiceMonitor, PodMonitor, PrometheusRule, Probe) 
# from all namespaces are reconciled automatically
# To control the scope of discovery, see: https://docs.victoriametrics.com/operator/resources/vmagent/#scraping and override the following accordingly.
    # spec:  
    #   selectAllByDefault: true
    #   serviceScrapeNamespaceSelector: {}
    #   podScrapeNamespaceSelector: {}
    #   podScrapeSelector: {}
    #   serviceScrapeSelector: {}
    #   nodeScrapeSelector: {}
    #   nodeScrapeNamespaceSelector: {}
    #   staticScrapeSelector: {}
    #   staticScrapeNamespaceSelector: {}

ArgoCD Deployment Considerations

If you are using ArgoCD to deploy your monitoring CRDs, add the following override to avoid syncing issues:

victoria-metrics-operator:
  operator:
    prometheus_converter_add_argocd_ignore_annotations: true

This configuration ensures that the VictoriaMetrics operator adds the necessary ArgoCD ignore annotations to prevent conflicts during synchronization.

Step 2 - Deploy Monitoring CRDs

By default the vm operator will identify the Prometheus CRDs (ServiceMonitor, PodMonitor, PrometheusRule and Probe ) that are already deployed and will scrape them automatically.

In case you want to deploy a test monitor object, here is an example using PodMonitor

1. Create the following my-test-podmonitor.yaml

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: scrape-demo
spec:
  selector:
    matchLabels:
      {app labels}
  podMetricsEndpoints:
  - port: {app metrics port}
  namespaceSelector:
      matchNames:
        - {pod namespace}

1. Deploy it

kubectl apply -n groundcover -f my-test-podmonitor.yaml

1. The vmagent will reload its configuration and will start scraping the target, metrics should appear in groundcover's grafana shortly after.

Other then supporting the standard Prometheus CRDs, VictoriaMetrics operator has their own proprietary CRDs that can be used, more about them can be found here.