# Backup & Restore Metrics

groundcover uses [VictoriaMetrics](https://docs.victoriametrics.com/) as its underlying metrics storage solution. As such, groundcover integrates seamlessly with VictoriaMetrics [vmbackup](https://docs.victoriametrics.com/vmbackup.html) and [vmrestore](https://docs.victoriametrics.com/vmrestore.html) tools.

## Doing incremental backups

* [Install vmutils](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)
* port-forward groundcover's VictoriaMetrics service object

```bash
kubectl get svc -n groundcover | grep "victoria-metrics"
# Identify the victoria-metrics service object name
kubectl port-forward svc/{victoria-metrics-service-object-name} \
-n groundcover 8428:8428
```

* Run the `vmbackup` utility, in this example we'll set the destination to an AWS S3 bucket, [but more providers are supported](https://docs.victoriametrics.com/vmbackup.html#supported-storage-types)

```bash
./vmbackup -credsFilePath={aws credentials path} \
-storageDataPath=</path/to/victoria-metrics-data> \
-snapshot.createURL=http://localhost:8428/snapshot/create \
-dst=s3://<bucket>/<path/to/backup>
```

{% hint style="info" %}
vmbackup automatically uses incremental backup strategy if the destination contains an existing backup
{% endhint %}

## Restoring from backup

* Scale down VictoriaMetrics statefulSet (VictoriaMetrics must be offline during restorations)

```bash
kubectl scale sts {release name}-victoria-metrics --replicas=0
```

* Get the VictoriaMetrics PVC name

```bash
kubectl get pvc -n groundcover | grep victoria-metrics
```

* Create the following Kubernetes Job manifest `vm-restore.yaml`

{% hint style="info" %}
Make sure you replace {VICTORIA METRICS PVC NAME} with the fetched pvc name
{% endhint %}

<pre class="language-yaml"><code class="lang-yaml">apiVersion: v1
kind: ServiceAccount
metadata:
  name: vm-restore
  annotations:
    eks.amazonaws.com/role-arn: XXXXX # role with permissions to write to the bucket
---
<strong>apiVersion: batch/v1
</strong>kind: Job
metadata:
  name: vm-restore
spec:
  ttlSecondsAfterFinished: 600
  template:
    spec:
      serviceAccountName: vm-restore
      restartPolicy: OnFailure
      volumes:
      - name: vmstorage-volume
        persistentVolumeClaim:
          claimName: "{VICTORIA METRICS PVC NAME}"
      containers:
      - name: vm-restore
        image: victoriametrics/vmrestore
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: /storage
          name: vmstorage-volume
        command:
        - /bin/sh
        - -c
        - /vmrestore-prod -src=s3://&#x3C;bucket>/&#x3C;path/to/backup> -storageDataPath=/storage
</code></pre>

* Deploy the job and wait for completion

```bash
kubectl apply -f vm-restore.yaml -n groundcover
```

* Once completed, scale up groundcover's VictoriaMetrics instance

```
kubectl scale sts {release name}-victoria-metrics --replicas=1
```