LogoLogo
Log in|Playground
  • Welcome
    • Introduction
    • FAQ
  • Capabilities
    • Log Management
    • Infrastructure Monitoring
    • Application Performance Monitoring (APM)
      • Application Metrics
      • Traces
      • Supported Technologies
    • Real User Monitoring (RUM)
  • Getting Started
    • Requirements
      • Kubernetes requirements
      • Kernel requirements for eBPF sensor
      • CPU architectures
      • ClickHouse resources
    • Installation & updating
    • Connect Linux hosts
    • Connect RUM
    • 5 quick steps to get you started
    • groundcover MCP
      • Configure groundcover's MCP Server
      • Getting-started Prompts
      • Real-world Use Cases
  • Use groundcover
    • Monitors
      • Create a new Monitor
      • Issues page
      • Monitor List page
      • Silences page
      • Monitor Catalog page
      • Monitor YAML structure
      • Embedded Grafana Alerts
        • Create a Grafana alert
    • Dashboards
      • Create a dashboard
      • Embedded Grafana Dashboards
        • Create a Grafana dashboard
        • Build alerts & dashboards with Grafana Terraform provider
        • Using groundcover datasources in a Self-hosted Grafana
    • Insights
    • Explore & Monitors query builder
    • Workflows
      • Create a new Workflow
      • Workflow Examples
      • Alert Structure
    • Search & Filter
    • Issues
    • Role-Based Access Control (RBAC)
    • Service Accounts
    • API Keys
    • APIs
    • Log Patterns
    • Drilldown
    • Scraping custom metrics
      • Operator based metrics
      • kube-state-metrics
      • cadvisor metrics
    • Backup & Restore Metrics
    • Metrics & Labels
    • Add custom environment labels
    • Configuring Pipelines
      • Writing Remap Transforms
      • Logs Pipeline Examples
      • Traces Pipeline Examples
      • Logs to Events Pipeline Examples
      • Logs/Traces Sensitive Data Obfuscation
      • Sensitive Data Obfuscation using OTTL
      • Log Filtering using OTTL
    • Querying your groundcover data
      • Query your logs
        • Example queries
        • Logs alerting
      • Query your metrics
      • Querying you data using an API
      • Using KEDA autoscaler with groundcover
  • Log Parsing with OpenTelemetry Pipelines
  • Log and Trace Correlation
  • RUM
  • Customization
    • Customize deployment
      • Agents in host network mode
      • API Key Secret
      • Argo CD
      • On-premise deployment
      • Quay.io registry
      • Configuring sensor deployment coverage
      • Enabling SSL Tracing in Java Applications
    • Customize usage
      • Filtering Kubernetes entities
      • Custom data retention
      • Sensitive data obfuscation
      • Custom storage
      • Custom logs collection
      • Custom labels and annotations
        • Enrich logs and traces with pod labels & annotations
        • Enrich metrics with node labels
      • Disable tracing for specific protocols
      • Tuning resources
      • Controlling the eBPF sampling mechanism
  • Integrations
    • Overview
    • Workflow Integrations
      • Slack Webhook Integration
      • Opsgenie Integration
      • Webhook Integration
        • Incident.io
      • PagerDuty Integration
      • Jira Webhook Integration
      • Send groundcover Alerts to Email via Zapier
    • Data sources
      • OpenTelemetry
        • Traces & Logs
        • Metrics
      • Istio
      • AWS
        • Ingest CloudWatch Metrics
        • Ingest CloudWatch Logs
        • Ingest Logs Stored on S3
        • Integrate CloudWatch Grafana Datasource
      • GCP
        • Ingest Google Cloud Monitoring Metrics
        • Stream Logs using Pub/Sub
        • Integrate Google Cloud Monitoring Grafana Datasource
      • Azure
        • Ingest Azure Monitor Metrics
      • DataDog
        • Traces
        • Metrics
      • FluentBit
      • Fluentd
      • JSON Logs
    • 3rd-party metrics
      • ActiveMQ
      • Aerospike
      • Cassandra
      • CloudFlare
      • Consul
      • CoreDNS
      • Etcd
      • HAProxy
      • Harbor
      • JMeter
      • K6
      • Loki
      • Nginx
      • Pi-hole
      • Postfix
      • RabbitMQ
      • Redpanda
      • SNMP
      • Solr
      • Tomcat
      • Traefik
      • Varnish
      • Vertica
      • Zabbix
    • Source control (Gitlab/Github)
  • Architecture
    • Overview
    • inCloud Managed
      • Setup inCloud Managed with AWS
        • AWS PrivateLink Setup
        • EKS add-on
      • Setup inCloud Managed with GCP
      • Setup inCloud Managed with Azure
      • High Availability
      • Disaster Recovery
      • Ingestion Endpoints
      • Deploying in Sensor-Only mode
    • Security considerations
      • Okta SSO - onboarding
    • Service endpoints inside the cluster
  • Product Updates
    • What's new?
    • Earlier updates
      • 2025
        • Mar 2025
        • Feb 2025
        • Jan 2025
      • 2024
        • Dec 2024
        • Nov 2024
        • Oct 2024
        • Sep 2024
        • Aug 2024
        • July 2024
        • May 2024
        • Apr 2024
        • Mar 2024
        • Feb 2024
        • Jan 2024
      • 2023
        • Dec 2023
        • Nov 2023
        • Oct 2023
Powered by GitBook
On this page
  • Examples
  • Patterns Library
Export as PDF
  1. Use groundcover
  2. Configuring Pipelines

Logs/Traces Sensitive Data Obfuscation

groundcover’s pipelines can be used to protect sensitive data in your Logs and Traces using Vector's redact function. Mask or remove sensitive information while preserving the usefulness of your data.

Last updated 6 months ago

groundcover’s pipelines can be used to protect sensitive data in your Logs and Traces. With Vector's redact function, you can mask or remove sensitive information while preserving the usefulness of your observability data.

We highly recommend using Vector's built-in function redact for Logs/Traces obfuscation. This powerful function allows you to configure simple yet effective redaction rules to protect sensitive information in your logs and traces.

With redact, you can:

  • Mask or remove sensitive data from strings, arrays, or objects

  • Replace text matching specified patterns (like regex) with a placeholder, custom text, or a hash (SHA-2 or SHA-3)

Please refer to the redact function's for more details.

On this page, we'll explore how to leverage the redact function and VRL's capabilities to obfuscate PII in Logs and Traces. At the end of this page, you'll find a handy list of regex patterns to save you time and effort.

Trace obfuscation can also be. Can't find what you're looking for?

Examples

In the examples below, we redact both the log contents (.content) and any attributes derived from the structued logs (.string_attributes).

Obfuscate credit card numbers from Logs

In this example, we'll obfuscate Visa credit card numbers from logs using the Visa credit card regex pattern from the library. By not specifying a redactor type, the redact function will default to full redaction, replacing detected numbers with the string “[REDACTED].”

vector:
  logsPipeline:
    extraSteps: 
    - name: obfuscateVisaCards
      transform:
        type: remap
        source: |-
          pattern = r'(?:4\d{3}){4}|4\d{7}\d{8}|4\d{12}(?:\d{3})?'
          
          # Redact content. Cast to string and redact
          contentAsString, err = string(.content)
          if err == null {
              .content = redact(contentAsString, filters: [pattern])
          }
          
          # Redact all attributes. Iterate, cast to string, redact
          .string_attributes = map_values(object!(.string_attributes), recursive:true) -> |value| { 
            asString, err = string(value)
            if err == null {
                redact(asString, filters: [pattern])
            } else {
                value
            }
          }

Here's an example of how Logs appear before and after obfuscation:

# Before
{
  "name": "Amit Shamun",
  "creditCard": {
    "issuer": "Visa",
    "number": "4111111111111111"
  }
}

# After
{
    "name": "Amit Shamun",
    "creditCard": {
        "issuer": "Visa",
        "number": "[REDACTED]"
    }
}

Hash US SSNs in Logs

In this example we’ll hash of all US Social Security Numbers hidden in logs. We’ll pass the sha2 parameter to the redactor to hash the sensitive values.

vector:
  logsPipeline:
    extraSteps: 
    - name: redactSSN
      transform:
        type: remap
        source: |- 
          # Redact content. Cast to string and redact
          contentAsString, err = string(.content)
          if err == null {
              .content = redact(contentAsString, filters: ["us_social_security_number"], redactor: {"type": "sha2", "variant": "SHA-256", "encoding": "base16"})
          }
          
          # Redact all attributes. Iterate, cast to string, redact
          .string_attributes = map_values(object!(.string_attributes), recursive:true) -> |value| { 
            asString, err = string(value)
            if err == null {
                redact(asString, filters: ["us_social_security_number"], redactor: {"type": "sha2", "variant": "SHA-256", "encoding": "base16"})
            } else {
                value
            }
          }

Here's how logs appear before and after obfuscation:

# Before
{
  "name": "Amit Shamun",
  "ssn": "123-45-6789"
}

# After
{
    "name": "Amit Shamun",
    "ssn": "01a54629efb952287e554eb23ef69c52097a75aecc0e3a93ca0855ab6d7a31a0"
}

Obfuscate IPs with Two Stages from Logs This example demonstrates how to obfuscate IP addresses in logs using a two-stage approach:

vector:
  logsPipeline:
    extraSteps: 
    - name: obfuscateIPv4
      transform:
        type: remap
        source: |-
          ipv4Pattern = r'\b(?:\d{1,3}\.){3}\d{1,3}\b'
          
          # Redact content. Cast to string and redact
          contentAsString, err = string(.content)
          if err == null {
              .content = redact(contentAsString, filters: [ipv4Pattern])
          }
          
          # Redact all attributes. Iterate, cast to string, redact
          .string_attributes = map_values(object!(.string_attributes), recursive:true) -> |value| { 
            asString, err = string(value)
            if err == null {
                redact(asString, filters: [ipv4Pattern])
            } else {
                value
            }
          }
  
    - name: obfuscateIPv6
      transform:
        type: remap
        source: |-
          ipv6Pattern = r'([0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}'
          
          # Redact content. Cast to string and redact
          contentAsString, err = string(.content)
          if err == null {
              .content = redact(contentAsString, filters: [ipv6Pattern])
          }
          
          # Redact all attributes. Iterate, cast to string, redact
          .string_attributes = map_values(object!(.string_attributes), recursive:true) -> |value| { 
            asString, err = string(value)
            if err == null {
                redact(asString, filters: [ipv6Pattern])
            } else {
                value
            }
          }

Here's an example of how logs appear before and after this obfuscation:

# Before
{
    "server": "test-server",
    "ipv4": "1.2.3.4",
    "ipv6": "2001:0db8:85a3:0000:0000:8a2e:0370:7334",
}

# After
{
    "server": "test-server",
    "ipv4": "[REDACTED]",
    "ipv6": "[REDACTED]",
}

Patterns Library

Credit Card Scanners

Description
Pattern

Maestro Card (16 digits)

5[0678]\d{14}|5[0678]\d{7}\d{8}

Discover Card (16 digits)

(?:6(?:011|5\d{2}))\d{12}|6(?:011|5\d{2})\d{7}\d{8}

Diners Club (14 digits)

3(?:0[0-5]|[68]\d)\d{11}|3(?:0[0-5]|[68]\d)\d{4}\d{4}\d{2}|3(?:0[0-5]|[68]\d)\d{7}\d{6}

American Express (15 digits)

3[47]\d{2}\d{6}\d{5}|3[47]\d{2}\d{4}\d{4}\d{3}|3[47]\d{7}\d{6}|3[47]\d{13}

JCB Card (16 digits)

(?:2131|1800|35\d{3})\d{11}|(?:2131|1800|35\d{3})\d{7}\d{8}

MasterCard (16 digits)

(?:5[1-5]\d{2}){4}|5[1-5]\d{7}\d{8}|5[1-5]\d{14}

Visa Card (16 or 19 digits)

(?:4\d{3}){4}|4\d{7}\d{8}|4\d{12}(?:\d{3})?

API Key and Token Scanners

Description
Pattern

AWS Access Key ID and Secret Access Key

AKIA[0-9A-Z]{16}|(?:[A-Za-z0-9/+=]{40})

Google API Key and OAuth Access Token

AIza[0-9A-Za-z-]{35}|ya29.[0-9A-Za-z-]{35,}

Mailchimp API Key

(?i)(?:[a-f0-9]{32}-us\d{1,2})

Social Media Tokens (Facebook, Slack, Twitter, Instagram, LinkedIn)

EAACEdEose0cBA[0-9A-Za-z]{0,}|xox[baprs]-[0-9]{12}-[0-9]{12}-[a-zA-Z0-9]{24}|T[a-zA-Z0-9]{8}/B[a-zA-Z0-9]{8}/[a-zA-Z0-9]{24}|AAAAA[0-9A-Za-z]{35}|IGQVJW[0-9A-Za-z]{16,}|[A-Z0-9]{16}

Azure Personal Access Token

eyJ[0-9A-Za-z-_=]+

Azure SQL Connection String

Server=tcp:[A-Za-z0-9-.]+,1433;Database=[A-Za-z0-9-]+;User Id=[A-Za-z0-9-]+;Password=[A-Za-z0-9-]+;Encrypt=true;

Azure Subscription Key

Ocp-Apim-Subscription-Key: [A-Za-z0-9]{32}

GitHub Access Token and Refresh Token

ghp_[0-9A-Za-z]{36}

Shopify Access Token and Shared Secret

shpat_[A-Za-z0-9]{32}

Okta API Token

00[0-9A-Fa-f]{8}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{12}

JSON Web Token (JWT)

eyJ[0-9A-Za-z-*=]+\.[0-9A-Za-z-*=]+\.[A-Za-z0-9-_=]+

RSA Private Key

-----BEGIN RSA PRIVATE KEY-----[\s\S]+-----END RSA PRIVATE KEY-----

PGP Private Key

-----BEGIN PGP PRIVATE KEY BLOCK-----[\s\S]+-----END PGP PRIVATE KEY BLOCK-----

GitLab Token

glpat-[A-Za-z0-9-]{20}

Amazon Marketplace Web Services Auth Token

amzn\.mws\.[a-zA-Z0-9]{64}

Bearer Token

Bearer [A-Za-z0-9-*=]+\.[A-Za-z0-9-*=]+\.[A-Za-z0-9-_=]+

JIRA API Token

jira\.api\.token\.[A-Za-z0-9-_]+

Other Scanners

Description
Pattern

Standard Email Address

[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+

Standard IBAN Code

[A-Z]{2}\d{2}[A-Z0-9]{1,30}

Standard MAC Address

([0-9A-Fa-f]{2}[:-]){5}([0-9A-Fa-f]{2})

IPv4 Address

\b(?:\d{1,3}\.){3}\d{1,3}\b

IPv6 Address

([0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}

HTTP(S) URL

https?:\/\/(?:www\.)?[a-zA-Z0-9\-\.]+\.[a-zA-Z]{2,6}(\/[a-zA-Z0-9\&%_\.\/~\-]*)?

HTTP Basic Authentication Header

Basic\s[a-zA-Z0-9=:_-]+

HTTP Cookie

Set-Cookie:\s[a-zA-Z0-9_\-]+=\S+

US Passport Number

[a-zA-Z]\d{9}

US Vehicle Identification Number (VIN)

[A-HJ-NPR-Z0-9]{17}

UK National Insurance Number

[A-CEGHJ-PR-TW-Za-ceghj-pr-tw-z]\d{2}\d{6}[A-Za-z]{1}

Canadian Social Insurance Number (SIN)

\d{3}-\d{3}-\d{3}

US Social Security Number (SSN)

\b\d{3}-\d{2}-\d{4}\b

documentation
configured directly in the sensor
Let us know over Slack.