Custom logs collection
Custom k8s logs filtering / storing
By default, groundcover stores logs from all namespaces and workloads in your cluster. However there are multiple ways to modify this behavior.
Filtering Rules
groundcover allows you to add logs filtering rules using LogQL syntax by creating a custom values.yaml
file.
The available labels to filter are: namespace, workload, pod, level, container.
Example of filtering out all logs coming from namespace demo
with level info
:
{namespace="demo",level="info"}
In addition, we enable the use of the optional log stream pipeline in order filter the log lines.
Example of filtering out all logs coming from container my-container
which contain the word fifo
or handler
:
{container="my-container"} |~ "fifo|handler"
More info on LogQL syntax can be found here.
Rules are applied sequentially and independently. Therefore, rules which are meant to specify multiple values of the same label should be written as one rule with multiple options, and not many rules with one option each.
For example, a rule to drop logs from all namespaces except prod
and dev
should be written as:
{namespace!="prod", namespace!="dev"}
Store Issues Logs Only
Sometimes saving all the logs from your cluster can be heavy on storage and compute, so we've added the complementary storeIssuesLogsOnly
flag can be set to indicate groundcover to collect logs only when issues are detected. Alternatively, this flag can also be enabled directly as a CLI flag (example below).
Usage
values.yaml example
Using CLI on New or Existing Installation
Using CLI on New or Existing Installation with storeIssuesLogsOnly flag
Using Helm on New Installation
Using Helm on Existing Installation
Customize Multi line logs
This enables merging multiple logs lines into a single log block. A new block is defined using a pre-define firstLineRegex
, which should match the line prefix.
A block is terminated when one of the following conditions is met:
A new block is matched
Timeout has occurred (optional config, default is 3 seconds)
Max number of lines-per-block is reached (optional config, default is 1024 lines)
Configuration holds also workload
& namespace
fields, which can be set to .*
in order to use wildcard logic. An additional optional container
field can be added.
Usage
Example
This will merge all exception logs into a single block line.
Customize logs decolorization
This feature enable removing ANSI color codes from logs' body.
Usage
Example
Will be stripped into:
Control logs truncation
Use this customization carefully as it might heavily effect the performance of the groundcover sensors.
Body VS Content:
During log parsing groundcover generates two attributes named content
and body
:
body
- contains the full log linecontent
- contains the message field of structured logs (from msg/message attribute) or the full log line for unstructured logs
In the platform UI the attribute displayed is the content
, while body
is available in the DB.
Example:
Formatted log with message:
{"time": "Jun 09 2023 15:28:14", "severity": "info", "msg": "Hello World"}
Unformatted log:
[Jun 09 2023 15:28:14], Hello World
Truncation Config:
The following values contain the default truncation size for body
and content
respectively:
Last updated