Logs to Events Pipeline Examples
We strongly advise reading the intro guide to working with remap transforms in order to fully understand the functionalities of writing pipelines steps.
Detecting a pattern and extracting data
The following example demonstrates transformation of a log in a specific format to an event, while applying additional filtering and extraction logic.
In this example, we want to create events for when a user consistently fails to login to a system. We base it on logs with this specific format:
login failed for user <username>, attempt number <number>
This pipeline will create events with the type multiple_login_failures
for each time a user fails to login for the 5th time or more . It will store the username in .string_attributes
and the attempt number in .float_attributes
.
vector:
eventsPipelines:
multiple_login_failures:
inputs:
- logs_from_logs
- json_logs
extraSteps:
- name: multiple_login_failures_filter
transform:
type: filter
condition: |
.container_name == "loginservice" && contains(string!(.content), "login failed for user")
- name: multiple_login_failures_extract
transform:
type: "remap"
source: |
regex_result = parse_regex!(string!(.content), r'login failed for user (?P<username>.*) attempt number (?P<attempt_number>[0-9.]+)')
if to_int!(regex_result.attempt_number) < 5 {
abort
}
.float_attributes = object!(.float_attributes)
.float_attributes.attempt_number = to_int!(regex_result.attempt_number)
.string_attributes = object!(.string_attributes)
.string_attributes.username = regex_result.username
Last updated