Custom logs collection

Custom k8s logs filtering / storing

By default, groundcover stores logs from all namespaces and workloads in your cluster. However there are multiple ways to modify this behavior.

Filtering Rules

groundcover allows you to add logs filtering rules using LogQL syntax by creating a custom values.yaml file. The available labels to filter are: namespace, workload, pod, level, container.Example of filtering out all logs coming from namespace demo with level info: {namespace="demo",level="info"}In addition, we enable the use of the optional log stream pipeline in order filter the log lines. Example of filtering out all logs coming from container my-container which contain the word fifo or handler: {container="my-container"} |~ "fifo|handler"

More info on LogQL syntax can be found here.

Rules are applied sequentially and independently. Therefore, rules which are meant to specify multiple values of the same label should be written as one rule with multiple options, and not many rules with one option each. For example, a rule to drop logs from all namespaces except prod and dev should be written as: {namespace!="prod", namespace!="dev"}

Store Issues Logs Only

Sometimes saving all the logs from your cluster can be heavy on storage and compute, so we've added the complementary storeIssuesLogsOnly flag can be set to indicate groundcover to collect logs only when issues are detected. Alternatively, this flag can also be enabled directly as a CLI flag (example below).

Usage

values.yaml example

storeIssuesLogsOnly: true

logsDropFilters:
 - '{namespace="demo-ng",workload="loadgenerator"} |~ ".*GET.*"'
 - '{namespace="demo-ng",workload="currencyservice"} !~ "received"'

Using CLI on New or Existing Installation

groundcover deploy --values values.yaml

Using CLI on New or Existing Installation with storeIssuesLogsOnly flag

groundcover deploy --values values.yaml --store-issues-logs-only

Using Helm on New Installation

Using Helm on Existing Installation

helm upgrade \
    groundcover \
    groundcover/groundcover \
    -n groundcover \
    --reuse-values \
    --values values.yaml

Customize Multi line logs

This enables merging multiple logs lines into a single log block. A new block is defined using a pre-define firstLineRegex , which should match the line prefix. A block is terminated when one of the following conditions is met:

A new block is matched
Timeout has occurred (optional config, default is 3 seconds)
Max number of lines-per-block is reached (optional config, default is 1024 lines)

Configuration holds also workload & namespace fields, which can be set to .* in order to use wildcard logic. An additional optional container field can be added.

Usage

logsMultiLines:
  - namespace: "test-namespace"
    workload: ".*"
    container: "test-container"    # optional
    firstLineRegex: "^\\[\\d{4}-\\d{2}-\\d{2} \\d{1,2}:\\d{2}:\\d{2}\\]"
    maxLines: 100                  # optional, default = 1024
    maxWaitTime: "10s"             # optional, default = 3s

Example

[2023-08-13 12:00:00] Exception!
Traceback (most recent call last):
  File "/path/to/example.py", line 4, in <module>
    greet('Chad')
  File "/path/to/example.py", line 2, in greet
    print('Hello, ' + someon)
NameError: name 'someon' is not defined
[2023-08-13 12:00:01] Next log line

This will merge all exception logs into a single block line.

Adding Grok Patterns for Custom Formats

groundcover supports providing custom Grok patterns to parse logs with unique formats that don’t conform to standard types.

To add a new grok format, you need to specify a pattern and a ruleName which categories the parsed logs as a specific sub-format. Additionally, namespaces, workloads, and containers can be used as filters to determine where the patterns should be applied.

Each attribute parsed will be automatically appended to the attributes of the log, making it searchable and filterable in the platform.

We strongly advise applying namespaces, workloadsand containersfilters to make the matching as tight as possible, deducing unneeded CPU overhead during parsing.

Usage

This example adds a custom Grok rule for parsing postgresql logs:

grokParserConfig:
  - containers:
      - "postgresql"
      - "postgres-db"
    pattern: '%{TIMESTAMP_ISO8601:timestamp} %{WORD:timezone} \[%{DATA:user_id}\] +%{GREEDYDATA:message}'
    ruleName: "psql"

Example

PostgreSQL error log:

2023-12-25 19:31:10.042 GMT [130] FATAL: terminating connection due to unexpected postmaster exit

"timestamp": "2023-12-25 19:31:10.042",
"timezone": "GMT",
"user_id": 130,
"message": "FATAL: terminating connection due to unexpected postmaster exit"

Customize logs decolorization

This feature enable removing ANSI color codes from logs' body.

Usage

decolorizeLogs: true

Example

[2023-10-31 20:38:57.123] \033[0;32HTTP\033[0m: GET /_healthz (1 ms) 204

Will be stripped into:

[2023-10-31 20:38:57.123] HTTP: GET /_healthz (1 ms) 204

Control logs truncation

Use this customization carefully as it might heavily effect the performance of the groundcover sensors.

Body VS Content:

During log parsing groundcover generates two attributes named content and body:

body - contains the full log line
content - contains the message field of structured logs (from msg/message attribute) or the full log line for unstructured logs

In the platform UI the attribute displayed is the content, while body is available in the DB.

Example:

Formatted log with message: {"time": "Jun 09 2023 15:28:14", "severity": "info", "msg": "Hello World"}

Body: {"time": "Jun 09 2023 15:28:14", "severity": "info", "msg": "Hello World"}
Content: "Hello World"

Unformatted log: [Jun 09 2023 15:28:14], Hello World

Body: "[Jun 09 2023 15:28:14], Hello World"
Content: "[Jun 09 2023 15:28:14], Hello World"

Truncation Config:

The following values contain the default truncation size for body and content respectively:

maxLogSize: 102400 # 100KB
maxLogContentSize: 5120 # 5KB

Last updated 10 days ago