LogoLogo
Log in|Playground
  • Welcome
    • Introduction
    • FAQ
  • Capabilities
    • Log Management
    • Infrastructure Monitoring
    • Application Performance Monitoring (APM)
      • Application Metrics
      • Traces
      • Supported Technologies
    • Real User Monitoring (RUM)
  • Getting Started
    • Requirements
      • Kubernetes requirements
      • Kernel requirements for eBPF sensor
      • CPU architectures
      • ClickHouse resources
    • Installation & updating
    • Connect Linux hosts
    • Connect RUM
    • 5 quick steps to get you started
    • groundcover MCP
      • Configure groundcover's MCP Server
      • Getting-started Prompts
      • Real-world Use Cases
  • Use groundcover
    • Monitors
      • Create a new Monitor
      • Issues page
      • Monitor List page
      • Silences page
      • Monitor Catalog page
      • Monitor YAML structure
      • Embedded Grafana Alerts
        • Create a Grafana alert
    • Dashboards
      • Create a dashboard
      • Embedded Grafana Dashboards
        • Create a Grafana dashboard
        • Build alerts & dashboards with Grafana Terraform provider
        • Using groundcover datasources in a Self-hosted Grafana
    • Insights
    • Explore & Monitors query builder
    • Workflows
      • Create a new Workflow
      • Workflow Examples
      • Alert Structure
    • Search & Filter
    • Issues
    • Role-Based Access Control (RBAC)
    • Service Accounts
    • API Keys
    • APIs
    • Log Patterns
    • Drilldown
    • Scraping custom metrics
      • Operator based metrics
      • kube-state-metrics
      • cadvisor metrics
    • Backup & Restore Metrics
    • Metrics & Labels
    • Add custom environment labels
    • Configuring Pipelines
      • Writing Remap Transforms
      • Logs Pipeline Examples
      • Traces Pipeline Examples
      • Logs to Events Pipeline Examples
      • Logs/Traces Sensitive Data Obfuscation
      • Sensitive Data Obfuscation using OTTL
      • Log Filtering using OTTL
    • Querying your groundcover data
      • Query your logs
        • Example queries
        • Logs alerting
      • Query your metrics
      • Querying you data using an API
      • Using KEDA autoscaler with groundcover
  • Log Parsing with OpenTelemetry Pipelines
  • Log and Trace Correlation
  • RUM
  • Customization
    • Customize deployment
      • Agents in host network mode
      • API Key Secret
      • Argo CD
      • On-premise deployment
      • Quay.io registry
      • Configuring sensor deployment coverage
      • Enabling SSL Tracing in Java Applications
    • Customize usage
      • Filtering Kubernetes entities
      • Custom data retention
      • Sensitive data obfuscation
      • Custom storage
      • Custom logs collection
      • Custom labels and annotations
        • Enrich logs and traces with pod labels & annotations
        • Enrich metrics with node labels
      • Disable tracing for specific protocols
      • Tuning resources
      • Controlling the eBPF sampling mechanism
  • Integrations
    • Overview
    • Workflow Integrations
      • Slack Webhook Integration
      • Opsgenie Integration
      • Webhook Integration
        • Incident.io
      • PagerDuty Integration
      • Jira Webhook Integration
      • Send groundcover Alerts to Email via Zapier
    • Data sources
      • OpenTelemetry
        • Traces & Logs
        • Metrics
      • Istio
      • AWS
        • Ingest CloudWatch Metrics
        • Ingest CloudWatch Logs
        • Ingest Logs Stored on S3
        • Integrate CloudWatch Grafana Datasource
      • GCP
        • Ingest Google Cloud Monitoring Metrics
        • Stream Logs using Pub/Sub
        • Integrate Google Cloud Monitoring Grafana Datasource
      • Azure
        • Ingest Azure Monitor Metrics
      • DataDog
        • Traces
        • Metrics
      • FluentBit
      • Fluentd
      • JSON Logs
    • 3rd-party metrics
      • ActiveMQ
      • Aerospike
      • Cassandra
      • CloudFlare
      • Consul
      • CoreDNS
      • Etcd
      • HAProxy
      • Harbor
      • JMeter
      • K6
      • Loki
      • Nginx
      • Pi-hole
      • Postfix
      • RabbitMQ
      • Redpanda
      • SNMP
      • Solr
      • Tomcat
      • Traefik
      • Varnish
      • Vertica
      • Zabbix
    • Source control (Gitlab/Github)
  • Architecture
    • Overview
    • inCloud Managed
      • Setup inCloud Managed with AWS
        • AWS PrivateLink Setup
        • EKS add-on
      • Setup inCloud Managed with GCP
      • Setup inCloud Managed with Azure
      • High Availability
      • Disaster Recovery
      • Ingestion Endpoints
      • Deploying in Sensor-Only mode
    • Security considerations
      • Okta SSO - onboarding
    • Service endpoints inside the cluster
  • Product Updates
    • What's new?
    • Earlier updates
      • 2025
        • Mar 2025
        • Feb 2025
        • Jan 2025
      • 2024
        • Dec 2024
        • Nov 2024
        • Oct 2024
        • Sep 2024
        • Aug 2024
        • July 2024
        • May 2024
        • Apr 2024
        • Mar 2024
        • Feb 2024
        • Jan 2024
      • 2023
        • Dec 2023
        • Nov 2023
        • Oct 2023
Powered by GitBook
On this page
  • Filtering Rules
  • Usage
  • Configure Journal Logs
  • Usage
  • Configure Log File Targets
  • Usage
  • Customize Multi line logs
  • Usage
  • Example
  • Custom parsing with OTTL pipelines
  • Customize logs decolorization
  • Usage
  • Example
  • Control logs truncation
  • Body VS Content:
  • Truncation Config:
Export as PDF
  1. Customization
  2. Customize usage

Custom logs collection

Custom k8s logs filtering / storing

Last updated 23 days ago

By default, groundcover stores logs from all namespaces and workloads in your cluster. However there are multiple ways to modify this behavior.

Filtering Rules

groundcover allows you to add logs filtering rules using LogQL syntax by creating a custom values.yaml file. The available labels to filter are: namespace, workload, pod, level, container. Example of filtering out all logs coming from namespace demo with level info: {namespace="demo",level="info"} In addition, we enable the use of the optional log stream pipeline in order filter the log lines. Example of filtering out all logs coming from container my-container which contain the word fifo or handler: {container="my-container"} |~ "fifo|handler"

More info on LogQL syntax can be found .

Rules are applied sequentially and independently. Therefore, rules which are meant to specify multiple values of the same label should be written as one rule with multiple options, and not many rules with one option each. For example, a rule to drop logs from all namespaces except prod and dev should be written as: {namespace!="prod", namespace!="dev"}

Usage

values.yaml example

logsDropFilters:
 - '{namespace="demo-ng",workload="loadgenerator"} |~ ".*GET.*"'
 - '{namespace="demo-ng",workload="currencyservice"} !~ "received"'

Using CLI on New or Existing Installation

groundcover deploy --values values.yaml

Using Helm on New Installation

more on getting api-key, see:

helm upgrade \
    groundcover \
    groundcover/groundcover \
    -n groundcover \
    -i \
    --set global.groundcover_token=<api-key>,clusterId=my_cluster
    --values values.yaml

Using Helm on Existing Installation

helm upgrade \
    groundcover \
    groundcover/groundcover \
    -n groundcover \
    --reuse-values \
    --values values.yaml

Configure Journal Logs

groundcover collects kubelet logs on Kubernetes clusters and docker logs on host machines. You can customize this behavior through additional configuration options.

Usage

journalScraper:
  serviceNames: ["kubelet", "kernel"]  # List of systemd service units to capture logs from
  historyDuration: 10m                 # How far back in time to collect logs when discovering new files
  journalFilter: "system.journal"      # Journal file pattern to match (e.g., "*.journal")
  includeRemoteJournals: true          # Enable collection of journals beyond the current machine-id
/etc/opt/groundcover/overrides.yaml
logs:
  scraper:
    journalConfig:
      - serviceNames: ["docker", "sshd", "kernel", "groundcover-sensor"]
        journalFilter: "*.journal"
        includeRemoteJournals: false

Configure Log File Targets

groundcover can collect logs from specific files on your host machine. You can define paths to monitor and add custom labels to the collected logs.

Usage

logFileTargets:
  - path: "/var/log/syslog.*" # Path pattern to match log files
    excludedPath: "*.gz"      # Pattern to exclude from matched files
    labels:                   # Custom labels to attach to collected logs
      label1: "value1"
    workload: "syslog"        # Optional: Set a custom workload name
/etc/opt/groundcover/overrides.yaml
logs:
  scraper:
    logFileTargets:
      - path: /var/log/auth.log

Customize Multi line logs

This enables merging multiple logs lines into a single log block. A new block is defined using a pre-define firstLineRegex , which should match the line prefix. A block is terminated when one of the following conditions is met:

  • A new block is matched

  • Timeout has occurred (optional config, default is 3 seconds)

  • Max number of lines-per-block is reached (optional config, default is 1024 lines)

Configuration holds also workload & namespace fields, which can be set to .* in order to use wildcard logic. An additional optional container field can be added.

Usage

logsMultiLines:
  - namespace: "test-namespace"
    workload: ".*"
    container: "test-container"    # optional
    firstLineRegex: "^\\[\\d{4}-\\d{2}-\\d{2} \\d{1,2}:\\d{2}:\\d{2}\\]"
    maxLines: 100                  # optional, default = 1024
    maxWaitTime: "10s"             # optional, default = 3s
logs:
  pipeline:
    multiLineConfigs: 
    - namespace: ".*" 
      workload: ".*"
      firstLineRegex: "^\\[\\d{4}-\\d{2}-\\d{2} \\d{1,2}:\\d{2}:\\d{2}\\]"
      maxLines: 100                  # optional, default = 1024
      maxWaitTime: "10s"             # optional, default = 3s

Example

[2023-08-13 12:00:00] Exception!
Traceback (most recent call last):
  File "/path/to/example.py", line 4, in <module>
    greet('Chad')
  File "/path/to/example.py", line 2, in greet
    print('Hello, ' + someon)
NameError: name 'someon' is not defined
[2023-08-13 12:00:01] Next log line

This will merge all exception logs into a single block line.

Custom parsing with OTTL pipelines

Customize logs decolorization

This feature enable removing ANSI color codes from logs' body.

Usage

decolorizeLogs: true

Example

[2023-10-31 20:38:57.123] \033[0;32HTTP\033[0m: GET /_healthz (1 ms) 204

Will be stripped into:

[2023-10-31 20:38:57.123] HTTP: GET /_healthz (1 ms) 204

Control logs truncation

Use this customization carefully as it might heavily effect the performance of the groundcover sensors.

Body VS Content:

During log parsing groundcover generates two attributes named content and body:

  1. body - contains the full log line

  2. content - contains the message field of structured logs (from msg/message attribute) or the full log line for unstructured logs

In the platform UI the attribute displayed is the content, while body is available in the DB.

Example:

Formatted log with message: {"time": "Jun 09 2023 15:28:14", "severity": "info", "msg": "Hello World"}

Body: {"time": "Jun 09 2023 15:28:14", "severity": "info", "msg": "Hello World"}
Content: "Hello World"

Unformatted log: [Jun 09 2023 15:28:14], Hello World

Body: "[Jun 09 2023 15:28:14], Hello World"
Content: "[Jun 09 2023 15:28:14], Hello World"

Truncation Config:

The following values contain the default truncation size for body and content respectively:

maxLogSize: 102400 # 100KB
maxLogContentSize: 5120 # 5KB

groundcover supports using for custom parsing such as Grok patterns, key-value extraction and more.

Instructions for setting up OTTL rules can be found .

OTTL pipelines
here
here
Using Helm