Parsing Logs

Overview

Transform unstructured log messages into structured, queryable data. By extracting fields from log messages, you can unlock powerful filtering, searching, and analysis capabilities in groundcover.

Why Parse Logs?

Raw log messages often contain valuable information buried in unstructured text. Parsing allows you to:

Extract meaningful fields from log messages for better searchability
Structure your data to enable powerful filtering and querying
Enrich logs with additional context and metadata
Standardize formats across different services and applications

Best Practices

Use conditions effectively - Only parse logs from relevant workloads to minimize processing overhead
Test in the Parsing Playground - Always test your parsing rules before deploying
Cache intermediate results - Use the cache variable for temporary storage during multi-step transformations
Be specific with patterns - More specific GROK patterns perform better and are less likely to cause false matches

Common Parsing Use Cases

Parsing JSON Logs

Extract fields from JSON-formatted log messages.

Log

{
  "body": "2025-03-23 14:55:12,456 ERROR {\"event\":\"user_login\",\"user_id\":12345,\"status\":\"failed\",\"ip\":\"192.168.1.10\"}"
}

Rule

ottlRules:
  - ruleName: "parse_json_logs"
    conditions:
      - 'workload == "auth-service"'
    statements:
      - 'set(cache, ExtractGrokPatterns(body, "^%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{LOGLEVEL}%{SPACE}(?<json_body>\\{.*\\})"))'
      - 'set(cache["parsed_json"], ParseJSON(cache["json_body"]))'
      - 'merge_maps(attributes, cache["parsed_json"], "insert")'

Result

{
  "event": "user_login",
  "user_id": 12345,
  "status": "failed",
  "ip": "192.168.1.10"
}

Merging Multiline Logs

Merge multiline log entries (stack traces, system warnings, etc) into a single record.

Log

{
  "body": "2026-01-15 10:30:00 ERROR java.lang.NullPointerException: user is null"
}
{
  "body": "  at com.example.UserService.getUser(UserService.java:42)"
}
{
  "body": "  at com.example.ApiHandler.handle(ApiHandler.java:128)"
}

Rule

ottlRules:
  - ruleName: "java-stacktrace"
    conditions:
      - 'workload == "my-java-app"'
    statements:
      - 'multiline_merge("^\\d{4}-\\d{2}-\\d{2}")'

Result

{
  "body": "2026-01-15 10:30:00 ERROR java.lang.NullPointerException: user is null\n  at com.example.UserService.getUser(UserService.java:42)\n  at com.example.ApiHandler.handle(ApiHandler.java:128)"
}

Extracting Structured Data with GROK Patterns

Use GROK patterns to extract specific fields from formatted log messages.

Log

{
  "body": "2025-03-23 10:30:45 INFO User login attempt from 192.168.1.100"
}

Rule

ottlRules:
  - ruleName: "extract_login_info"
    conditions:
      - 'workload == "auth-service"'
    statements:
      - 'set(cache, ExtractGrokPatterns(body, "^%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{LOGLEVEL:level}%{SPACE}User login attempt from %{IP:source_ip}"))'
      - 'keep_matching_keys(cache, "^[a-z_]+$")'
      - 'merge_maps(attributes, cache, "insert")'

Result

{
  "timestamp": "2025-03-23 10:30:45",
  "level": "INFO",
  "source_ip": "192.168.1.100"
}

Parsing Key-Value Pairs

Extract multiple key-value pairs from bracketed log sections.

Log

{
  "body": "2025-03-23 15:20:12,512 - EventProcessor - DEBUG - Completed event processing [analyzer_name=disk-space-check] [node_id=7f5e9aa8412d4c0003a7b2c5] [service_id=813dd10298f77700029d54e3] [sensor_id=3] [tracking_code=19fd5b6e72c7e94088a9ff3d] [log_id=b'67acfe0c92d43000'] [instance_id=microservice-7894563210]"
}

Rule

ottlRules:
  - ruleName: "parse_event_processor"
    conditions:
      - 'workload == "event-processor"'
    statements:
      - 'set(cache, ExtractGrokPatterns(body, "^%{TIMESTAMP_ISO8601:timestamp}%{SPACE}-%{SPACE}%{NOTSPACE}%{SPACE}%{NOTSPACE}%{SPACE}%{LOGLEVEL:level}%{DATA}(?<kv>\\[%{GREEDYDATA})"))'
      - 'replace_pattern(cache["kv"], "[\\[\\]]", "")'
      - 'merge_maps(attributes, ParseKeyValue(cache["kv"]), "insert")'
      - 'set(attributes["timestamp"], cache["timestamp"])'

Result

{
  "instance_id": "microservice-7894563210",
  "analyzer_name": "disk-space-check",
  "node_id": "7f5e9aa8412d4c0003a7b2c5",
  "service_id": "813dd10298f77700029d54e3",
  "sensor_id": "3",
  "tracking_code": "19fd5b6e72c7e94088a9ff3d",
  "log_id": "b67acfe0c92d43000",
  "timestamp": "2025-03-23 15:20:12,512"
}

Common GROK Patterns

Here are some commonly used GROK patterns:

%{TIMESTAMP_ISO8601} - ISO8601 timestamps (2025-03-23T10:30:45)
%{LOGLEVEL} - Log levels (INFO, ERROR, DEBUG, etc.)
%{IP} - IP addresses
%{NUMBER} - Numeric values
%{WORD} - Single words
%{NOTSPACE} - Non-whitespace characters
%{GREEDYDATA} - Match everything (greedy)
%{DATA} - Match everything (non-greedy)
%{SPACE} - Whitespace characters

Key Functions

ExtractGrokPatterns

Extracts structured data using GROK patterns.

Basic Usage:

- 'set(cache, ExtractGrokPatterns(body, "pattern"))'

Example - Extract timestamp and error code:

ottlRules:
  - ruleName: "extract_error_details"
    conditions:
      - 'level == "error"'
    statements:
      - 'set(cache, ExtractGrokPatterns(body, "^%{TIMESTAMP_ISO8601:timestamp}.*Error %{NUMBER:error_code}:"))'
      - 'merge_maps(attributes, cache, "insert")'

ParseJSON

Parses JSON strings into structured attributes.

Basic Usage:

- 'set(cache["parsed"], ParseJSON(cache["json_string"]))'

Example - Parse embedded JSON:

ottlRules:
  - ruleName: "parse_json_payload"
    conditions:
      - 'format == "json"'
    statements:
      - 'set(attributes["parsed_data"], ParseJSON(body))'
      - 'set(attributes["user_id"], attributes["parsed_data"]["user"]["id"])'

ParseKeyValue

Parses key=value formatted strings.

Basic Usage:

- 'merge_maps(attributes, ParseKeyValue(cache["kv"]), "insert")'

Example - Parse query parameters:

ottlRules:
  - ruleName: "parse_query_params"
    conditions:
      - 'IsMatch(body, "params=")'
    statements:
      - 'set(cache, ExtractGrokPatterns(body, "params=(?<params>.+)$"))'
      - 'merge_maps(attributes, ParseKeyValue(cache["params"], "&", "="), "insert")'

merge_maps

Merges extracted data into the attributes map.

Basic Usage:

- 'merge_maps(attributes, cache, "insert")'

Modes:

insert - Only add keys that don't exist
update - Only update keys that exist
upsert - Add or update keys (default behavior)

Example:

ottlRules:
  - ruleName: "merge_extracted_fields"
    conditions:
      - 'workload == "api-gateway"'
    statements:
      - 'set(cache, ExtractGrokPatterns(body, "user=(?<user>\\w+) action=(?<action>\\w+)"))'
      - 'merge_maps(attributes, cache, "upsert")'

keep_matching_keys

Filters a map to keep only keys matching a regex pattern.

Basic Usage:

- 'keep_matching_keys(cache, "^[a-z_]+$")'

Example - Keep only lowercase field names:

ottlRules:
  - ruleName: "filter_extracted_fields"
    conditions:
      - 'workload == "data-processor"'
    statements:
      - 'set(cache, ExtractGrokPatterns(body, "(?<TIMESTAMP>\\d+) (?<user_id>\\d+) (?<action>\\w+)"))'
      - 'keep_matching_keys(cache, "^[a-z_]+$")'  # Keeps user_id and action, drops TIMESTAMP
      - 'merge_maps(attributes, cache, "insert")'

set

Sets a value to a specific field or attribute.

Basic Usage:

- 'set(attributes["field"], "value")'
- 'set(cache["temp"], body)'

Example - Set computed fields:

ottlRules:
  - ruleName: "add_computed_fields"
    conditions:
      - 'workload == "payment-service"'
    statements:
      - 'set(attributes["environment"], "production")'
      - 'set(attributes["service_type"], "payment")'
      - 'set(attributes["processed_at"], UnixNano())'

replace_pattern

Replaces parts of a string matching a regex pattern.

Basic Usage:

- 'replace_pattern(target_field, "pattern", "replacement")'

Example - Clean up log messages:

ottlRules:
  - ruleName: "clean_log_format"
    conditions:
      - 'container_name == "nginx"'
    statements:
      - 'replace_pattern(body, "\\[|\\]", "")'  # Remove brackets
      - 'replace_pattern(body, "\\s+", " ")'    # Normalize whitespace

delete_key

Removes a specific key from a map.

Basic Usage:

- 'delete_key(attributes, "field_name")'

Example - Remove temporary fields:

ottlRules:
  - ruleName: "cleanup_fields"
    conditions:
      - 'workload == "processor"'
    statements:
      - 'set(cache, ExtractGrokPatterns(body, "(?<temp_field>\\w+) (?<user_id>\\d+)"))'
      - 'merge_maps(attributes, cache, "insert")'
      - 'delete_key(attributes, "temp_field")'

Concat

Concatenates multiple strings or values together.

Basic Usage:

- 'set(attributes["combined"], Concat([value1, value2, value3], " - "))'

Example - Create composite fields:

ottlRules:
  - ruleName: "create_composite_id"
    conditions:
      - 'attributes["user_id"] != nil'
    statements:
      - 'set(attributes["request_id"], Concat([attributes["user_id"], attributes["timestamp"]], "_"))'
      - 'set(attributes["full_path"], Concat([attributes["namespace"], "/", attributes["workload"]], ""))'

ToLowerCase / ToUpperCase

Converts strings to lowercase or uppercase.

Basic Usage:

- 'set(attributes["normalized_level"], ToLowerCase(attributes["level"]))'
- 'set(attributes["region"], ToUpperCase(attributes["region"]))'

Example - Normalize field values:

ottlRules:
  - ruleName: "normalize_values"
    conditions:
      - 'attributes["level"] != nil'
    statements:
      - 'set(attributes["level"], ToLowerCase(attributes["level"]))'
      - 'set(attributes["env"], ToUpperCase(attributes["env"]))'

Substring

Extracts a portion of a string.

Basic Usage:

- 'set(attributes["short_id"], Substring(attributes["request_id"], 0, 8))'

Example - Extract prefixes and suffixes:

ottlRules:
  - ruleName: "extract_id_prefix"
    conditions:
      - 'attributes["transaction_id"] != nil'
    statements:
      - 'set(attributes["tx_prefix"], Substring(attributes["transaction_id"], 0, 6))'
      - 'set(attributes["tx_type"], Substring(attributes["transaction_id"], 0, 3))'

Split

Splits a string into an array based on a delimiter.

Basic Usage:

- 'set(cache["parts"], Split(body, ","))'

Example - Parse comma-separated values:

ottlRules:
  - ruleName: "parse_csv_log"
    conditions:
      - 'format == "csv"'
    statements:
      - 'set(cache["fields"], Split(body, ","))'
      - 'set(attributes["user_id"], cache["fields"][0])'
      - 'set(attributes["action"], cache["fields"][1])'
      - 'set(attributes["timestamp"], cache["fields"][2])'

Len

Returns the length of a string or array.

Basic Usage:

- 'set(attributes["message_length"], Len(body))'

Example - Conditional processing based on length:

ottlRules:
  - ruleName: "truncate_long_messages"
    conditions:
      - 'Len(body) > 1000'
    statements:
      - 'set(attributes["original_length"], Len(body))'
      - 'set(body, Substring(body, 0, 1000))'
      - 'set(attributes["truncated"], true)'

Int / Double / String

Type conversion functions.

Basic Usage:

- 'set(attributes["status_code_int"], Int(attributes["status_code"]))'
- 'set(attributes["response_time_ms"], Double(attributes["response_time"]))'
- 'set(attributes["user_id_str"], String(attributes["user_id"]))'

Example - Convert and compute:

ottlRules:
  - ruleName: "convert_and_calculate"
    conditions:
      - 'attributes["duration"] != nil'
    statements:
      - 'set(attributes["duration_ms"], Int(attributes["duration"]))'
      - 'set(attributes["duration_sec"], Double(attributes["duration"]) / 1000.0)'
      - 'set(attributes["is_slow"], attributes["duration_ms"] > 5000)'

IsMatch

Checks if a field matches a regex pattern.

Basic Usage:

- 'set(attributes["is_error"], IsMatch(body, "ERROR|FATAL"))'

Example - Conditional field extraction:

ottlRules:
  - ruleName: "extract_http_status"
    conditions:
      - 'IsMatch(body, "HTTP/\\d\\.\\d\\s+\\d{3}")'
    statements:
      - 'set(cache, ExtractGrokPatterns(body, "HTTP/%{NUMBER:http_version}\\s+%{NUMBER:status_code}"))'
      - 'merge_maps(attributes, cache, "insert")'
      - 'set(attributes["is_error"], Int(attributes["status_code"]) >= 400)'

multiline_merge

Reassembles multiline log entries into a single merge record.

Basic Usage:

- 'multiline_merge("^\\d{4}-\\d{2}-\\d{2}")'

Arguments:

first_line_pattern (string, required) - Regex matching the first line of a new block. Lines not matching are continuations.
max_lines (int, optional, default: 128)- Maximum lines per block. Buffer is flushed when reached.
max_time (string, optional, default: "800ms")- Maximum time for continuations. Buffer is flushed when reached.

Example - Merge then parse with grok:

ottlRules:
  - ruleName: "merge-and-extract"
    conditions:
      - 'workload == "api-server"'
    statements:
      - 'multiline_merge("^\\d{4}-\\d{2}-\\d{2}", 64, "500ms")'
      - 'set(cache, ExtractGrokPatterns(body, "^%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{LOGLEVEL:level}%{SPACE}%{GREEDYDATA:message}"))'
      - 'merge_maps(attributes, cache, "insert")'

Important notes:

Place multiline_merge as the first statement in the rule. Subsequent statements see the merged content when a flush occurs.
The rule's conditions must capture both first lines and continuations, don't filter on a pattern that only matches first lines.
Buffers are isolated per instance. Logs from different pods. containers or files never mix.

Last updated 41 minutes ago

hashtagOverview

hashtagWhy Parse Logs?

hashtagBest Practices

hashtagCommon Parsing Use Cases

hashtagParsing JSON Logs

hashtagMerging Multiline Logs

hashtagExtracting Structured Data with GROK Patterns

hashtagParsing Key-Value Pairs

hashtagCommon GROK Patterns

hashtagKey Functions

hashtagExtractGrokPatterns

hashtagParseJSON

hashtagParseKeyValue

hashtagmerge_maps

hashtagkeep_matching_keys

hashtagset

hashtagreplace_pattern

hashtagdelete_key

hashtagConcat

hashtagToLowerCase / ToUpperCase

hashtagSubstring

hashtagSplit

hashtagLen

hashtagInt / Double / String

hashtagIsMatch

hashtagmultiline_merge

Overview

Why Parse Logs?

Best Practices

Common Parsing Use Cases

Parsing JSON Logs

Merging Multiline Logs

Extracting Structured Data with GROK Patterns

Parsing Key-Value Pairs

Common GROK Patterns

Key Functions

ExtractGrokPatterns

ParseJSON

ParseKeyValue

merge_maps

keep_matching_keys

set

replace_pattern

delete_key

Concat

ToLowerCase / ToUpperCase

Substring

Split

Len

Int / Double / String

IsMatch

multiline_merge