Logs to Events Pipeline Examples

The generated events will currently only be available by querying the ClickHouse database directly. Contact us over Slack for additional information.

Detecting a pattern and extracting data

Attributes parsed from logs or traces can be accessed under the .string_attributes or .float_attributes maps - see here for more information.

The following example demonstrates transformation of a log in a specific format to an event, while applying additional filtering and extraction logic.

In this example, we want to create events for when a user consistently fails to login to a system. We base it on logs with this specific format:

login failed for user <username>, attempt number <number>

This pipeline will create events with the type multiple_login_failures for each time a user fails to login for the 5th time or more . It will store the username in .string_attributes and the attempt number in .float_attributes.

vector:
  eventsPipelines:
    multiple_login_failures:
      inputs:
        - logs_from_logs
        - json_logs
      extraSteps:
        - name: multiple_login_failures_filter
          transform:
            type: filter
            condition: |
              .container_name == "loginservice" && contains(string!(.content), "login failed for user")
        - name: multiple_login_failures_extract
          transform:
            type: "remap"
            source: |
              regex_result = parse_regex!(string!(.content), r'login failed for user (?P<username>.*) attempt number (?P<attempt_number>[0-9.]+)')
              if to_int!(regex_result.attempt_number) < 5 {
                abort
              }
              .float_attributes = object!(.float_attributes)
              .float_attributes.attempt_number = to_int!(regex_result.attempt_number)
              .string_attributes = object!(.string_attributes)
              .string_attributes.username = regex_result.username

Last updated