Custom logs collection
Custom k8s logs filtering / storing
By default, groundcover stores logs from all namespaces and workloads in your cluster. However there are multiple ways to modify this behavior.
Filtering Rules
groundcover allows you to add logs filtering rules using LogQL syntax by creating a custom values.yaml
file.
The available labels to filter are: namespace, workload, pod, level, container.
Example of filtering out all logs coming from namespace demo
with level info
:
{namespace="demo",level="info"}
In addition, we enable the use of the optional log stream pipeline in order filter the log lines.
Example of filtering out all logs coming from container my-container
which contain the word fifo
or handler
:
{container="my-container"} |~ "fifo|handler"
More info on LogQL syntax can be found here.
Usage
values.yaml example
logsDropFilters:
- '{namespace="demo-ng",workload="loadgenerator"} |~ ".*GET.*"'
- '{namespace="demo-ng",workload="currencyservice"} !~ "received"'
Using CLI on New or Existing Installation
groundcover deploy --values values.yaml
Using Helm on New Installation
more on getting api-key, see: Using Helm
helm upgrade \
groundcover \
groundcover/groundcover \
-n groundcover \
-i \
--set global.groundcover_token=<api-key>,clusterId=my_cluster
--values values.yaml
Using Helm on Existing Installation
helm upgrade \
groundcover \
groundcover/groundcover \
-n groundcover \
--reuse-values \
--values values.yaml
Configure Journal Logs
groundcover collects kubelet logs on Kubernetes clusters and docker logs on host machines. You can customize this behavior through additional configuration options.
Usage
journalScraper:
serviceNames: ["kubelet", "kernel"] # List of systemd service units to capture logs from
historyDuration: 10m # How far back in time to collect logs when discovering new files
journalFilter: "system.journal" # Journal file pattern to match (e.g., "*.journal")
includeRemoteJournals: true # Enable collection of journals beyond the current machine-id
Configure Log File Targets
groundcover can collect logs from specific files on your host machine. You can define paths to monitor and add custom labels to the collected logs.
Usage
logFileTargets:
- path: "/var/log/syslog.*" # Path pattern to match log files
excludedPath: "*.gz" # Pattern to exclude from matched files
labels: # Custom labels to attach to collected logs
label1: "value1"
workload: "syslog" # Optional: Set a custom workload name
Customize Multi line logs
This enables merging multiple logs lines into a single log block. A new block is defined using a pre-define firstLineRegex
, which should match the line prefix.
A block is terminated when one of the following conditions is met:
A new block is matched
Timeout has occurred (optional config, default is 3 seconds)
Max number of lines-per-block is reached (optional config, default is 1024 lines)
Configuration holds also workload
& namespace
fields, which can be set to .*
in order to use wildcard logic. An additional optional container
field can be added.
Usage
logsMultiLines:
- namespace: "test-namespace"
workload: ".*"
container: "test-container" # optional
firstLineRegex: "^\\[\\d{4}-\\d{2}-\\d{2} \\d{1,2}:\\d{2}:\\d{2}\\]"
maxLines: 100 # optional, default = 1024
maxWaitTime: "10s" # optional, default = 3s
Example
[2023-08-13 12:00:00] Exception!
Traceback (most recent call last):
File "/path/to/example.py", line 4, in <module>
greet('Chad')
File "/path/to/example.py", line 2, in greet
print('Hello, ' + someon)
NameError: name 'someon' is not defined
[2023-08-13 12:00:01] Next log line
This will merge all exception logs into a single block line.
Custom parsing with OTTL pipelines
groundcover supports using OTTL pipelines for custom parsing such as Grok patterns, key-value extraction and more.
Instructions for setting up OTTL rules can be found here.
Customize logs decolorization
This feature enable removing ANSI color codes from logs' body.
Usage
decolorizeLogs: true
Example
[2023-10-31 20:38:57.123] \033[0;32HTTP\033[0m: GET /_healthz (1 ms) 204
Will be stripped into:
[2023-10-31 20:38:57.123] HTTP: GET /_healthz (1 ms) 204
Control logs truncation
Use this customization carefully as it might heavily effect the performance of the groundcover sensors.
Body VS Content:
During log parsing groundcover generates two attributes named content
and body
:
body
- contains the full log linecontent
- contains the message field of structured logs (from msg/message attribute) or the full log line for unstructured logs
In the platform UI the attribute displayed is the content
, while body
is available in the DB.
Example:
Formatted log with message:
{"time": "Jun 09 2023 15:28:14", "severity": "info", "msg": "Hello World"}
Body: {"time": "Jun 09 2023 15:28:14", "severity": "info", "msg": "Hello World"}
Content: "Hello World"
Unformatted log:
[Jun 09 2023 15:28:14], Hello World
Body: "[Jun 09 2023 15:28:14], Hello World"
Content: "[Jun 09 2023 15:28:14], Hello World"
Truncation Config:
The following values contain the default truncation size for body
and content
respectively:
maxLogSize: 102400 # 100KB
maxLogContentSize: 5120 # 5KB
Last updated