Configuring Pipelines

groundcover supports the configuration of logs and traces pipelines, to further process and customize the data being collected, using Vector transforms. This enables full flexibility to manipulate the data as it flows into the platform.

See this page for more information about how Vector is being used in the groundcover platform's architecture.

Using Vector transforms

groundcover uses Vector as an aggregator and transformer deployed into each monitored environment. It is an open-source, highly performant service, capable of supporting many manipulations on the data flowing into groundcover's backend.

Pipelines are configured using Vector transforms, where each transform defines one step in the pipeline. There are many types of transforms, and all of them can be natively used within the groundcover deployment to achieve full flexibility.

The most common transform is the remap transform - allowing to write arbitrary logic using Vector's VRL syntax. There are many pre-defined functions to parse, filter and enrich data, and we recommend experimenting with it to fit your needs.

For testing out VRL before deployment we recommend the VRL playground.

Deploying groundcover with Pipelines

groundcover's deployment supports adding a list of transforms for logs and traces independently. These steps will be automatically appended into the default pipeline, eliminating the need to understand the inner workings of grouncover's setup. Instead, you only need to configure the steps you wish to execute, and after redeploying groundcover you will see them take effect immediately.

Each step requires two attributes:

• name: must be unique across all pipelines

• transform: the transform itself, passed as-is to Vector.

Logs Pipeline

The following is a template for a logs pipeline with two remap stages:

vector:
  logsPipeline:
    extraSteps: 
    - name: stepA
      transform:
        type: remap
        source: |-
          ...
    - name: stepB
      transform:
        type: remap
        source: |-
          ...

View an example with real inputs ->

Traces Pipeline

The following is an template for a traces pipeline with one filter stage:

vector:
  tracesPipeline:
    extraSteps: 
    - name: stepA
      transform:
        type: filter
        condition: |- 
           ...

View an example with real inputs ->