Configuring sensor Deployment on Kubernetes

groundcover's sensors are required to be running on every node for it to be monitored. By default, the sensor will be included across all installed clusters with the exception of fargate nodes.

When installing groundcover using the CLI, detected taints will be displayed and a prompt for adding appropriate tolerations will be displayed.

Tolerations

When installing groundcover using the CLI, detected taints will be displayed and a prompt for adding appropriate tolerations will be displayed.

Otherwise, the following configuration values are used by default, which allow our sensor to run on all nodes.

agent:
  tolerations: 
   - operator: "Exists"

This allows sensors to tolerate all taints, which may sometimes be problematic. For example, tolerating node.kubernetes.io/not-ready may cause sensors to restart until the node becomes ready. If this is a problem in your case, you may replace this configuration with a list of specific tolerations that match your environment. To do so, consult these docs.

You can control where other groundcover components are scheduled by configuring nodeSelector and tolerations in your values.yaml.

This is useful when you want to run observability workloads on dedicated nodes.

vector:
  nodeSelector:
    <label-key>: <label-value>

  tolerations:
    - key: "<taint-key>"
      operator: "Equal"   # or "Exists"
      value: "<taint-value>"   # omit if operator is "Exists"
      effect: "NoSchedule"     # or "PreferNoSchedule", "NoExecute"

The relevant sections for the non-backend components of groundcover that can be configured in a similar way are: vector, k8sWatcher, kube-state-metrics, custom-metrics, metrics-ingester, victoria-metrics-agent

Example:

k8sWatcher:
  nodeSelector:
    node-role: observability

  tolerations:
    - key: "dedicated"
      operator: "Equal"
      value: "observability"
      effect: "NoSchedule"

Affinity

By default, we prevent the sensor from attempting to start on fargate nodes, with the affinity configuration:

agent:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions: # prevent sensor from starting on fargate
          - key: eks.amazonaws.com/compute-type
            operator: NotIn
            values:
            - fargate
        - matchExpressions: # prevent sensor from starting on control plane nodes
          - key: "node-role.kubernetes.io/control-plane"
            operator: "DoesNotExist"

Priority Class

It's recommended for sensor to have a high priority class so that it's scheduled before other pods, providing full visibility into the startup of the entire environment.

The default priority class is relatively high (yet lower than default node and cluster critical priority class), and can be configured. The default is shown below:

priorityClass:
  create: true
  fullname:
  value: 1000000000
  preemptionPolicy: PreemptLowerPriority

Exceptions can be set by overriding the above values.