Operator based metrics

guide on how to enable crd-based scraping targets

Step 1 - Enable custom metrics

1. Enabling custom metrics scraping

# Make sure custom-metrics service object is also enabled in the helm overrides
custom-metrics:
  enabled: true
  service:
    enabled: true

1. Fetch the service name of the custom metrics deployment

kubectl get svc -n groundcover | grep "custom-metrics"

Step 2 - Install the VictoriaMetrics operator

#add the repo
helm repo add vm https://victoriametrics.github.io/helm-charts/
#install the operator
helm install groundcover-vmoperator vm/victoria-metrics-operator \ 
-n groundcover

By default VMAgent will discover CRDs from all the namespaces in the cluster, in case you're interested in binding it to a specific namespace, this can be achieved by passing --set watchNamespace={namespace name}

Step 2 - Install the VMAgent CRD

1. Create the following groundcover-vmagent.yaml

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
  name: groundcover-vmagent
spec:
  serviceScrapeNamespaceSelector: {}
  podScrapeNamespaceSelector: {}
  podScrapeSelector: {}
  serviceScrapeSelector: {}
  nodeScrapeSelector: {}
  nodeScrapeNamespaceSelector: {}
  staticScrapeSelector: {}
  staticScrapeNamespaceSelector: {}
  replicaCount: 1
  remoteWrite:
    - url: "http://{custom-metrics-svc-name-from-step-1}.groundcover.svc.cluster.local:8429/api/v1/write"

2. Deploy it

kubectl apply -f groundcover-vmagent.yaml -n groundcover

Step 3 - Deploy Monitoring CRDs

By default the vm operator will identify the Prometheus CRDs (ServiceMonitor, PodMonitor, PrometheusRule and Probe ) that are already deployed and scrape them automatically.

In case you want to deploy a test monitor object, here is an example using PodMonitor

1. Create the following my-test-podmonitor.yaml

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: scrape-demo
spec:
  selector:
    matchLabels:
      {app labels}
  podMetricsEndpoints:
  - port: {app metrics port}
  namespaceSelector:
      matchNames:
        - {pod namespace}

2. Deploy it

kubectl apply -n groundcover -f my-test-podmonitor.yaml

3. The vmagent will reload its configuration and will start scrape the target, metrics should appear in groundcover's grafana shortly after.

Other then supporting the standard Prometheus CRDs, VictoriaMetrics operator has its own proprietary CRDs that can be used, more about them can be found here.