Logs to Events Pipeline Examples

We strongly advise reading the intro guide to working with remap transforms in order to fully understand the functionalities of writing pipelines steps.

The generated events will currently only be available by querying the ClickHouse database directly. Contact us over Slack for additional information.

Detecting a pattern and extracting data

Attributes parsed from logs or traces can be accessed under the .string_attributes or .float_attributes maps - see here for more information.

The following example demonstrates transformation of a log in a specific format to an event, while applying additional filtering and extraction logic.

In this example, we want to create events for when a user consistently fails to login to a system. We base it on logs with this specific format:

login failed for user <username>, attempt number <number>

This pipeline will create events with the type multiple_login_failures for each time a user fails to login for the 5th time or more . It will store the username in .string_attributes and the attempt number in .float_attributes.

vector:
  eventsPipelines:
    multiple_login_failures:
      inputs:
        - logs_from_logs
        - json_logs
      extraSteps:
        - name: multiple_login_failures_filter
          transform:
            type: filter
            condition: |
              .container_name == "loginservice" && contains(string!(.content), "login failed for user")
        - name: multiple_login_failures_extract
          transform:
            type: "remap"
            source: |
              regex_result = parse_regex!(string!(.content), r'login failed for user (?P<username>.*) attempt number (?P<attempt_number>[0-9.]+)')
              if to_int!(regex_result.attempt_number) < 5 {
                abort
              }
              .float_attributes = object!(.float_attributes)
              .float_attributes.attempt_number = to_int!(regex_result.attempt_number)
              .string_attributes = object!(.string_attributes)
              .string_attributes.username = regex_result.username