Custom data retention

Data Retention Strategies

There are two ways to define retention in groundcover:

1. Simple - each type of data has a global retention period

2. Advanced - data is retained based on various criteria such as cluster, log level, namespace, etc.

For metrics, only the simple strategy is supported.

Controlling Retention

For managed inCloud deployments, the values will need to be set by the groundcover team. Please contact us to perform any retention changes.

To customize the retention on the groundcover platform, either create a new custom-values.yaml or edit your existing values.yaml with the overrides defined below and redeploy groundcover.

Retention Field Format

Retention value format is: {amount}[h(ours), d(ays), w(eeks), y(ears)] .

For example: 4h, 30d, 6w, 1y

Simple Data Retention

The most common and simple way to configure retention in groundcover. Based solely on data type, without exceptions.

Below is an example configuration for setting data retention values:

global:
  traces:
    retention: 24h # default traces retention
  logs:
    retention: 3d # default logs retention
  events:
    retention: 7d # default events retention
victoria-metrics-single:
  server:  
    retentionPeriod: 7d # default metrics retention

Default retention values

Standard

• Traces - 24 hours

• Metrics - 7 days

• Logs - 3 days

• Events - 7 days

inCloud Managed

• Traces - 48 hours

• Metrics - 30 days

• Logs - 30 days

• Events - 7 days

Advanced Data Retention

groundcover allows you to customize retention policies for your data to better manage storage and compliance requirements. You can define specific retention periods for logs, traces, and events based on various criteria such as cluster, log level, namespace and more.

Custom Retention Overrides

The custom_retention_overrides list allows you to define specific retention periods for data based on conditions. Each override has a retention field and a conditions field.

• Retention: Specifies the duration for which the data should be retained.

• Conditions: Specify the criteria that the retention policy applies to. When multiple conditions are set, they are connected by an AND condition, meaning all conditions must be met.

Default Retention

The retention field under each data type (traces, logs, events) specifies the default retention period for that data type, meaning, anything that doesn't match the custom conditions set.

If a default retention value is not set for a certain datatype, groundcover will apply its own default retention from the default list described above.

In instances of overlapping overrides, the override with the shorter retention interval will be used.

Example Advanced Configuration File

The configuration below implements the following logic:

• Traces

  • Traces with labels cluster: prod, namespace: app will be retained for 7d

  • Traces with label env: staging will be retained for 14d

  • Other traces will be retained for 24h

• Logs

  • Logs with labels cluster: prod, level: info will be retained for 20d

  • Logs with labels cluster: prod, level: error will be retained for 30d

  • Other logs will be retained for 3d

• Events

  • Events with labels cluster: dev, type: Warning will be retained for 15d

  • Other events will be retained for 15d

global:
  traces:
    retention: 24h # default traces retention
    custom_retention_overrides:
      - retention: 7d
        conditions:
          cluster: 'prod'
          namespace: 'app'
      - retention: 14d
        conditions:
          env: 'staging'
  logs:
    retention: 3d # default logs retention
    custom_retention_overrides:
      - retention: 20d
        conditions:
          cluster: 'prod'
          level: 'info'
      - retention: 30d
        conditions:
          cluster: 'prod'
          level: 'error'
  events:
    retention: 7d # default events retention
    custom_retention_overrides:
      - retention: 15d
        conditions:
          type: 'Warning'
          cluster: 'dev'

victoria-metrics-single:
  server:
    retentionPeriod: 7d # {amount}[h(ours), d(ays), w(eeks), y(ears)]

Available Retention Fields

Logs

• cluster

• source

• env

• env_type

• workload

• namespace

• level

Traces

• cluster

• source

• env

• env_type

• protocol_type

Events

• cluster

• source

• env_name

• entity_workload

• entity_namespace

• type