LogoLogo
Log in|Playground
  • Welcome
    • Introduction
    • FAQ
  • Capabilities
    • Log Management
    • Infrastructure Monitoring
    • Application Performance Monitoring (APM)
      • Application Metrics
      • Traces
      • Supported Technologies
    • Real User Monitoring (RUM)
  • Getting Started
    • Requirements
      • Kubernetes requirements
      • Kernel requirements for eBPF sensor
      • CPU architectures
      • ClickHouse resources
    • Installation & updating
    • Connect Linux hosts
    • Connect RUM
    • 5 quick steps to get you started
    • groundcover MCP
      • Configure groundcover's MCP Server
      • Getting-started Prompts
      • Real-world Use Cases
  • Use groundcover
    • Monitors
      • Create a new Monitor
      • Issues page
      • Monitor List page
      • Silences page
      • Monitor Catalog page
      • Monitor YAML structure
      • Embedded Grafana Alerts
        • Create a Grafana alert
    • Dashboards
      • Create a dashboard
      • Embedded Grafana Dashboards
        • Create a Grafana dashboard
        • Build alerts & dashboards with Grafana Terraform provider
        • Using groundcover datasources in a Self-hosted Grafana
    • Insights
    • Explore & Monitors query builder
    • Workflows
      • Create a new Workflow
      • Workflow Examples
      • Alert Structure
    • Search & Filter
    • Issues
    • Role-Based Access Control (RBAC)
    • Service Accounts
    • API Keys
    • APIs
    • Log Patterns
    • Drilldown
    • Scraping custom metrics
      • Operator based metrics
      • kube-state-metrics
      • cadvisor metrics
    • Backup & Restore Metrics
    • Metrics & Labels
    • Add custom environment labels
    • Configuring Pipelines
      • Writing Remap Transforms
      • Logs Pipeline Examples
      • Traces Pipeline Examples
      • Logs to Events Pipeline Examples
      • Logs/Traces Sensitive Data Obfuscation
      • Sensitive Data Obfuscation using OTTL
      • Log Filtering using OTTL
    • Querying your groundcover data
      • Query your logs
        • Example queries
        • Logs alerting
      • Query your metrics
      • Querying you data using an API
      • Using KEDA autoscaler with groundcover
  • Log Parsing with OpenTelemetry Pipelines
  • Log and Trace Correlation
  • RUM
  • Customization
    • Customize deployment
      • Agents in host network mode
      • API Key Secret
      • Argo CD
      • On-premise deployment
      • Quay.io registry
      • Configuring sensor deployment coverage
      • Enabling SSL Tracing in Java Applications
    • Customize usage
      • Filtering Kubernetes entities
      • Custom data retention
      • Sensitive data obfuscation
      • Custom storage
      • Custom logs collection
      • Custom labels and annotations
        • Enrich logs and traces with pod labels & annotations
        • Enrich metrics with node labels
      • Disable tracing for specific protocols
      • Tuning resources
      • Controlling the eBPF sampling mechanism
  • Integrations
    • Overview
    • Workflow Integrations
      • Slack Webhook Integration
      • Opsgenie Integration
      • Webhook Integration
        • Incident.io
      • PagerDuty Integration
      • Jira Webhook Integration
      • Send groundcover Alerts to Email via Zapier
    • Data sources
      • OpenTelemetry
        • Traces & Logs
        • Metrics
      • Istio
      • AWS
        • Ingest CloudWatch Metrics
        • Ingest CloudWatch Logs
        • Ingest Logs Stored on S3
        • Integrate CloudWatch Grafana Datasource
      • GCP
        • Ingest Google Cloud Monitoring Metrics
        • Stream Logs using Pub/Sub
        • Integrate Google Cloud Monitoring Grafana Datasource
      • Azure
        • Ingest Azure Monitor Metrics
      • DataDog
        • Traces
        • Metrics
      • FluentBit
      • Fluentd
      • JSON Logs
    • 3rd-party metrics
      • ActiveMQ
      • Aerospike
      • Cassandra
      • CloudFlare
      • Consul
      • CoreDNS
      • Etcd
      • HAProxy
      • Harbor
      • JMeter
      • K6
      • Loki
      • Nginx
      • Pi-hole
      • Postfix
      • RabbitMQ
      • Redpanda
      • SNMP
      • Solr
      • Tomcat
      • Traefik
      • Varnish
      • Vertica
      • Zabbix
    • Source control (Gitlab/Github)
  • Architecture
    • Overview
    • inCloud Managed
      • Setup inCloud Managed with AWS
        • AWS PrivateLink Setup
        • EKS add-on
      • Setup inCloud Managed with GCP
      • Setup inCloud Managed with Azure
      • High Availability
      • Disaster Recovery
      • Ingestion Endpoints
      • Deploying in Sensor-Only mode
    • Security considerations
      • Okta SSO - onboarding
    • Service endpoints inside the cluster
  • Product Updates
    • What's new?
    • Earlier updates
      • 2025
        • Mar 2025
        • Feb 2025
        • Jan 2025
      • 2024
        • Dec 2024
        • Nov 2024
        • Oct 2024
        • Sep 2024
        • Aug 2024
        • July 2024
        • May 2024
        • Apr 2024
        • Mar 2024
        • Feb 2024
        • Jan 2024
      • 2023
        • Dec 2023
        • Nov 2023
        • Oct 2023
Powered by GitBook
On this page
  • Doing incremental backups
  • Restoring from backup
Export as PDF
  1. Use groundcover

Backup & Restore Metrics

Learn how to backup and restore metrics into groundcover metrics storage

Last updated 1 year ago

groundcover uses as its underlying metrics storage solution. As such, groundcover integrates seamlessly with VictoriaMetrics and tools.

Doing incremental backups

  • port-forward groundcover's VictoriaMetrics service object

kubectl get svc -n groundcover | grep "victoria-metrics"
# Identify the victoria-metrics service object name
kubectl port-forward svc/{victoria-metrics-service-object-name} \
-n groundcover 8428:8428
  • Run the vmbackup utility, in this example we'll set the destination to an AWS S3 bucket,

./vmbackup -credsFilePath={aws credentials path} \
-storageDataPath=</path/to/victoria-metrics-data> \
-snapshot.createURL=http://localhost:8428/snapshot/create \
-dst=s3://<bucket>/<path/to/backup>

vmbackup automatically uses incremental backup strategy if the destination contains an existing backup

Restoring from backup

  • Scale down VictoriaMetrics statefulSet (VictoriaMetrics must be offline during restorations)

kubectl scale sts {release name}-victoria-metrics --replicas=0
  • Get the VictoriaMetrics PVC name

kubectl get pvc -n groundcover | grep victoria-metrics
  • Create the following Kubernetes Job manifest vm-restore.yaml

Make sure you replace {VICTORIA METRICS PVC NAME} with the fetched pvc name

apiVersion: v1
kind: ServiceAccount
metadata:
  name: vm-restore
  annotations:
    eks.amazonaws.com/role-arn: XXXXX # role with permissions to write to the bucket
---
apiVersion: batch/v1
kind: Job
metadata:
  name: vm-restore
spec:
  ttlSecondsAfterFinished: 600
  template:
    spec:
      serviceAccountName: vm-restore
      restartPolicy: OnFailure
      volumes:
      - name: vmstorage-volume
        persistentVolumeClaim:
          claimName: "{VICTORIA METRICS PVC NAME}"
      containers:
      - name: vm-restore
        image: victoriametrics/vmrestore
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: /storage
          name: vmstorage-volume
        command:
        - /bin/sh
        - -c
        - /vmrestore-prod -src=s3://<bucket>/<path/to/backup> -storageDataPath=/storage
  • Deploy the job and wait for completion

kubectl apply -f vm-restore.yaml -n groundcover
  • Once completed, scale up groundcover's VictoriaMetrics instance

kubectl scale sts {release name}-victoria-metrics --replicas=1

VictoriaMetrics
vmbackup
vmrestore
Install vmutils
but more providers are supported