Logs Pipeline Examples

We strongly advise reading the intro guide to working with remap transforms in order to fully understand the functionalities of writing pipelines steps.

Attributes parsed from logs or traces can be accessed under the .string_attributes or .float_attributes maps - see here for more information.

Parsing an arbitrary format using regex

The following example attempts to match the contents of a log line with a given regex pattern, extracting named groups if successful. We recommend using named groups in the regex pattern for best experience, automatically creating named attributes which will appear in the system.

For example, this transform will create new timestamp and pid fields if they are successfully extracted from the content.

Note that we are only performing the parsing if the format attribute equals "unknown" - otherwise it means groundcover has already parsed the log format and extracted the fields beforehand.

The custom-format value is up to you, and will appear in the UI under the format filter.

For more regex documentation see this page. Vector natively supports parsing many known formats - it's always worth checking if the format is already natively supported!

We are using the drop_on_abort attribute to instruct vector to keep forwarding the event down the pipeline when encountering errors. For more information see this section.

vector:
  logsPipeline:
    extraSteps: 
    - name: parseRegex
      transform:
        type: remap
        drop_on_error: false
        source: |-
          if .format == "unknown" {
            regex_pattern = r'(?<timestamp>.*) (?<pid>\d+)'
            .string_attributes = object!(.string_attributes) | parse_regex!(.content, regex_pattern)
            .format = "custom-format"
          }

Renaming an attribute

The following example attempts to rename an attribute called oldName to newName. If it does not exists, no changes are made.

vector:
  logsPipeline:
    extraSteps: 
    - name: renameAttribute
      transform:
        type: remap
        source: |-
          if exists(.string_attributes.oldName) {
            .string_attributes.newName = del(.string_attributes.oldName)
          }