Log Filtering using OTTL
Filter out noisy, irrelevant, or non-critical logs at ingestion using OpenTelemetry Transformation Language (OTTL) in groundcover. By leveraging simple conditions and the drop flag, you can fine-tune which logs are retained for analysis and which are safely discarded.
Overview
In groundcover, you can apply filtering logic directly to logs using OTTL. This allows you to:
Reduce data volume and storage costs.
Focus on the logs that matter most.
Clean up noisy logs from specific services, containers, or patterns.
Filtering is achieved by setting a special attribute called drop to true based on specific conditions.
How Log Filtering Works
To filter a log, you define a condition (e.g., log message content, container name, log level) and use a set(drop, true) statement. If the condition matches, the log is dropped before ingestion.
Logs are dropped in the sensor - the earliest point possible - achieving maximum savings in terms of networking and ingestion.
Key Concepts:
drop: A reserved field. If set to true, the log is filtered out.where: Used to specify when the filtering should apply.IsMatch(): A function for matching regex or substrings within fields (like body).statementsErrorMode: How to handle transformation errors (propagate, skip, fail).conditionLogicOperator: Used when you define multiple conditions (or, and).
Common Examples
Example 1: Drop health check logs
💡 Filters out logs with requests to the /healthz endpoint from nginx containers.
Example 2: Drop logs with debug level
💡 Filters out debug logs from groundcover-demo service.
Last updated