Configuring Pipelines
groundcover supports the configuration of logs and traces pipelines, to further process and customize the data being collected, using Vector transforms. This enables full flexibility to manipulate the data as it flows into the platform.
See this page for more information about how Vector is being used in the groundcover platform's architecture.
Using Vector transforms
groundcover uses Vector as an aggregator and transformer deployed into each monitored environment. It is an open-source, highly performant service, capable of supporting many manipulations on the data flowing into groundcover's backend.
Pipelines are configured using Vector transforms, where each transform defines one step in the pipeline. There are many types of transforms, and all of them can be natively used within the groundcover deployment to achieve full flexibility.
Deploying groundcover with Pipelines
groundcover's deployment supports adding a list of transforms for logs and traces independently. These steps will be automatically appended into the default pipeline, eliminating the need to understand the inner workings of grouncover's setup. Instead, you only need to configure the steps you wish to execute, and after redeploying groundcover you will see them take effect immediately.
Each step requires two attributes:
name: must be unique across all pipelines
transform: the transform itself, passed as-is to Vector.
Logs Pipeline
The following is a template for a logs pipeline with two remap stages:
vector:
logsPipeline:
extraSteps:
- name: stepA
transform:
type: remap
source: |-
...
- name: stepB
transform:
type: remap
source: |-
...Traces Pipeline
The following is a template for a traces pipeline with one filter stage:
vector:
tracesPipeline:
extraSteps:
- name: stepA
transform:
type: filter
condition: |-
...Custom Logs to Events Pipeline
Logs to Events pipelines allow creating of custom events from incoming logs. Unlike the logs and traces pipelines, they do not affect the original logs, and are meant to create parallel, distinguished events for future analytics.
The following is a template for a custom event pipeline with a filter stage and an extraction step.
vector:
eventsPipelines:
my_event_name:
inputs:
- logs_from_logs
- json_logs
extraSteps:
- name: filter_step
transform:
type: filter
condition: |-
...
- name: extraction_step
transform:
type: remap
source: |-
...
Last updated