Configuring Pipelines

groundcover supports the configuration of logs and traces pipelines, to further process and customize the data being collected, using Vector transforms. This enables full flexibility to manipulate the data as it flows into the platform.

See this page for more information about how Vector is being used in the groundcover platform's architecture.

Using Vector transforms

groundcover uses Vector as an aggregator and transformer deployed into each monitored environment. It is an open-source, highly performant service, capable of supporting many manipulations on the data flowing into groundcover's backend.

Pipelines are configured using Vector transforms, where each transform defines one step in the pipeline. There are many types of transforms, and all of them can be natively used within the groundcover deployment to achieve full flexibility.

The most common transform is the remap transform - allowing to write arbitrary logic using Vector's VRL syntax. There are many pre-defined functions to parse, filter and enrich data, and we recommend experimenting with it to fit your needs.

For testing out VRL before deployment we recommend the VRL playground.

Deploying groundcover with Pipelines

groundcover's deployment supports adding a list of transforms for logs and traces independently. These steps will be automatically appended into the default pipeline, eliminating the need to understand the inner workings of grouncover's setup. Instead, you only need to configure the steps you wish to execute, and after redeploying groundcover you will see them take effect immediately.

Each step requires two attributes:

• name: must be unique across all pipelines

• transform: the transform itself, passed as-is to Vector.

Logs Pipeline

The following is a template for a logs pipeline with two remap stages:

vector:
  logsPipeline:
    extraSteps: 
    - name: stepA
      transform:
        type: remap
        source: |-
          ...
    - name: stepB
      transform:
        type: remap
        source: |-
          ...

View an example with real inputs ->

Traces Pipeline

The following is a template for a traces pipeline with one filter stage:

vector:
  tracesPipeline:
    extraSteps: 
    - name: stepA
      transform:
        type: filter
        condition: |- 
           ...

View an example with real inputs ->

Custom Logs to Events Pipeline

Logs to Events pipelines allow creating of custom events from incoming logs. Unlike the logs and traces pipelines, they do not affect the original logs, and are meant to create parallel, distinguished events for future analytics.

The following is a template for a custom event pipeline with a filter stage and an extraction step.

The inputs fields below will connect the events pipeline with the default incoming logs pipelines.

vector:
  eventsPipelines:
    my_event_name:
      inputs:
        - logs_from_logs
        - json_logs
      extraSteps:
        - name: filter_step
          transform:
            type: filter
            condition: |-
              ...
        - name: extraction_step
          transform:
            type: remap
            source: |-
              ...
          

View an example with real inputs