Custom logs collection

Custom k8s logs filtering / storing

By default, groundcover store logs from all namespaces and workloads in your cluster.

groundcover allows you to add logs filtering rules using LogQL syntax by creating a custom values.yaml file. The available labels to filter are: namespace, workload, pod, level, container. Example of filtering out all logs coming from namespace demo with level info: {namespace="demo",level="info"} In addition, we enable the use of the optional log stream pipeline in order filter the log lines. Example of filtering out all logs coming from container my-container which contain the word fifo or handler: {container="my-container"} |~ "fifo|handler"

More info on LogQL syntax can be found here.

Sometimes saving all the logs from your cluster can be heavy on storage and compute, so we've added the complementary storeIssuesLogsOnly flag can be set to indicate groundcover to collect logs only when issues are detected. Alternatively, this flag can also be enabled directly as a CLI flag (example below).

Usage

values.yaml example

storeIssuesLogsOnly: true

logsDropFilters:
 - '{namespace="demo-ng",workload="loadgenerator"} |~ ".*GET.*"'
 - '{namespace="demo-ng",workload="currencyservice"} !~ "received"'

Using CLI on New or Existing Installation

groundcover deploy --values values.yaml

Using CLI on New or Existing Installation with storeIssuesLogsOnly flag

groundcover deploy --values values.yaml --store-issues-logs-only

Using Helm on New Installation

more on getting api-key, see: Using Helm

helm upgrade \
    groundcover \
    groundcover/groundcover \
    -n groundcover \
    -i \
    --set global.groundcover_token=<api-key>,clusterId=my_cluster
    --values values.yaml

Using Helm on Existing Installation

helm upgrade \
    groundcover \
    groundcover/groundcover \
    -n groundcover \
    --reuse-values \
    --values values.yaml

Customize Multi line logs

This enables merging multiple logs lines into a single log block. A new block is defined using a pre-define firstLineRegex , which should match the line prefix. A block is terminated when one of the following conditions is met:

• A new block is matched

• Timeout has occurred (optional config, default is 3 seconds)

• Max number of lines-per-block is reached (optional config, default is 1024 lines)

Configuration holds also workload & namespace fields, which can be set to .* in order to use wildcard logic. An additional optional container field can be added.

Usage

logsMultiLines:
  - namespace: "test-namespace"
    workload: ".*"
    container: "test-container"    # optional
    firstLineRegex: "^\[\d{4}-\d{2}-\d{2} \d{1,2}:\d{2}:\d{2}\]"
    maxLines: 100                  # optional, default = 1024
    maxWaitTime: "10s"             # optional, default = 3s

Example

[2023-08-13 12:00:00] Exception!
Traceback (most recent call last):
  File "/path/to/example.py", line 4, in <module>
    greet('Chad')
  File "/path/to/example.py", line 2, in greet
    print('Hello, ' + someon)
NameError: name 'someon' is not defined
[2023-08-13 12:00:01] Next log line

This will merge all exception logs into a single block line.

Customize logs decolorization

This feature enable removing ANSI color codes from logs' body.

Usage

decolorizeLogs: true

Example

[2023-10-31 20:38:57.123] \033[0;32HTTP\033[0m: GET /_healthz (1 ms) 204

Will be stripped into:

[2023-10-31 20:38:57.123] HTTP: GET /_healthz (1 ms) 204