Logs/Traces Sensitive Data Obfuscation

groundcover’s pipelines can be used to protect sensitive data in your Logs and Traces using Vector's redact function. Mask or remove sensitive information while preserving the usefulness of your data.

groundcover’s pipelines can be used to protect sensitive data in your Logs and Traces. With Vector's redact function, you can mask or remove sensitive information while preserving the usefulness of your observability data.

We highly recommend using Vector's built-in function redact for Logs/Traces obfuscation. This powerful function allows you to configure simple yet effective redaction rules to protect sensitive information in your logs and traces.

With redact, you can:

• Mask or remove sensitive data from strings, arrays, or objects

• Replace text matching specified patterns (like regex) with a placeholder, custom text, or a hash (SHA-2 or SHA-3)

Please refer to the redact function's documentation for more details.

On this page, we'll explore how to leverage the redact function and VRL's capabilities to obfuscate PII in Logs and Traces. At the end of this page, you'll find a handy list of regex patterns to save you time and effort.

Trace obfuscation can also be configured directly in the sensor. Can't find what you're looking for? Let us know over Slack.

Examples

In the examples below, we redact both the log contents (.content) and any attributes derived from the structued logs (.string_attributes).

Obfuscate credit card numbers from Logs

In this example, we'll obfuscate Visa credit card numbers from logs using the Visa credit card regex pattern from the library. By not specifying a redactor type, the redact function will default to full redaction, replacing detected numbers with the string “[REDACTED].”

vector:
  logsPipeline:
    extraSteps: 
    - name: obfuscateVisaCards
      transform:
        type: remap
        source: |-
          pattern = r'(?:4\d{3}){4}|4\d{7}\d{8}|4\d{12}(?:\d{3})?'
          
          # Redact content. Cast to string and redact
          contentAsString, err = string(.content)
          if err == null {
              .content = redact(contentAsString, filters: [pattern])
          }
          
          # Redact all attributes. Iterate, cast to string, redact
          .string_attributes = map_values(object!(.string_attributes), recursive:true) -> |value| { 
            asString, err = string(value)
            if err == null {
                redact(asString, filters: [pattern])
            } else {
                value
            }
          }

Here's an example of how Logs appear before and after obfuscation:

# Before
{
  "name": "Amit Shamun",
  "creditCard": {
    "issuer": "Visa",
    "number": "4111111111111111"
  }
}

# After
{
    "name": "Amit Shamun",
    "creditCard": {
        "issuer": "Visa",
        "number": "[REDACTED]"
    }
}

Hash US SSNs in Logs

In this example we’ll hash of all US Social Security Numbers hidden in logs. We’ll pass the sha2 parameter to the redactor to hash the sensitive values.

vector:
  logsPipeline:
    extraSteps: 
    - name: redactSSN
      transform:
        type: remap
        source: |- 
          # Redact content. Cast to string and redact
          contentAsString, err = string(.content)
          if err == null {
              .content = redact(contentAsString, filters: ["us_social_security_number"], redactor: {"type": "sha2", "variant": "SHA-256", "encoding": "base16"})
          }
          
          # Redact all attributes. Iterate, cast to string, redact
          .string_attributes = map_values(object!(.string_attributes), recursive:true) -> |value| { 
            asString, err = string(value)
            if err == null {
                redact(asString, filters: ["us_social_security_number"], redactor: {"type": "sha2", "variant": "SHA-256", "encoding": "base16"})
            } else {
                value
            }
          }

Here's how logs appear before and after obfuscation:

# Before
{
  "name": "Amit Shamun",
  "ssn": "123-45-6789"
}

# After
{
    "name": "Amit Shamun",
    "ssn": "01a54629efb952287e554eb23ef69c52097a75aecc0e3a93ca0855ab6d7a31a0"
}

Obfuscate IPs with Two Stages from Logs This example demonstrates how to obfuscate IP addresses in logs using a two-stage approach:

vector:
  logsPipeline:
    extraSteps: 
    - name: obfuscateIPv4
      transform:
        type: remap
        source: |-
          ipv4Pattern = r'\b(?:\d{1,3}\.){3}\d{1,3}\b'
          
          # Redact content. Cast to string and redact
          contentAsString, err = string(.content)
          if err == null {
              .content = redact(contentAsString, filters: [ipv4Pattern])
          }
          
          # Redact all attributes. Iterate, cast to string, redact
          .string_attributes = map_values(object!(.string_attributes), recursive:true) -> |value| { 
            asString, err = string(value)
            if err == null {
                redact(asString, filters: [ipv4Pattern])
            } else {
                value
            }
          }
  
    - name: obfuscateIPv6
      transform:
        type: remap
        source: |-
          ipv6Pattern = r'([0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}'
          
          # Redact content. Cast to string and redact
          contentAsString, err = string(.content)
          if err == null {
              .content = redact(contentAsString, filters: [ipv6Pattern])
          }
          
          # Redact all attributes. Iterate, cast to string, redact
          .string_attributes = map_values(object!(.string_attributes), recursive:true) -> |value| { 
            asString, err = string(value)
            if err == null {
                redact(asString, filters: [ipv6Pattern])
            } else {
                value
            }
          }

Here's an example of how logs appear before and after this obfuscation:

# Before
{
    "server": "test-server",
    "ipv4": "1.2.3.4",
    "ipv6": "2001:0db8:85a3:0000:0000:8a2e:0370:7334",
}

# After
{
    "server": "test-server",
    "ipv4": "[REDACTED]",
    "ipv6": "[REDACTED]",
}

Patterns Library

Credit Card Scanners

Description	Pattern
Maestro Card (16 digits)	5[0678]\d{14}|5[0678]\d{7}\d{8}
Discover Card (16 digits)	(?:6(?:011|5\d{2}))\d{12}|6(?:011|5\d{2})\d{7}\d{8}
Diners Club (14 digits)	3(?:0[0-5]|[68]\d)\d{11}|3(?:0[0-5]|[68]\d)\d{4}\d{4}\d{2}|3(?:0[0-5]|[68]\d)\d{7}\d{6}
American Express (15 digits)	3[47]\d{2}\d{6}\d{5}|3[47]\d{2}\d{4}\d{4}\d{3}|3[47]\d{7}\d{6}|3[47]\d{13}
JCB Card (16 digits)	(?:2131|1800|35\d{3})\d{11}|(?:2131|1800|35\d{3})\d{7}\d{8}
MasterCard (16 digits)	(?:5[1-5]\d{2}){4}|5[1-5]\d{7}\d{8}|5[1-5]\d{14}
Visa Card (16 or 19 digits)	(?:4\d{3}){4}|4\d{7}\d{8}|4\d{12}(?:\d{3})?

API Key and Token Scanners

Description	Pattern
AWS Access Key ID and Secret Access Key	AKIA[0-9A-Z]{16}|(?:[A-Za-z0-9/+=]{40})
Google API Key and OAuth Access Token	AIza[0-9A-Za-z-]{35}|ya29.[0-9A-Za-z-]{35,}
Mailchimp API Key	(?i)(?:[a-f0-9]{32}-us\d{1,2})
Social Media Tokens (Facebook, Slack, Twitter, Instagram, LinkedIn)	EAACEdEose0cBA[0-9A-Za-z]{0,}|xox[baprs]-[0-9]{12}-[0-9]{12}-[a-zA-Z0-9]{24}|T[a-zA-Z0-9]{8}/B[a-zA-Z0-9]{8}/[a-zA-Z0-9]{24}|AAAAA[0-9A-Za-z]{35}|IGQVJW[0-9A-Za-z]{16,}|[A-Z0-9]{16}
Azure Personal Access Token	eyJ[0-9A-Za-z-_=]+
Azure SQL Connection String	Server=tcp:[A-Za-z0-9-.]+,1433;Database=[A-Za-z0-9-]+;User Id=[A-Za-z0-9-]+;Password=[A-Za-z0-9-]+;Encrypt=true;
Azure Subscription Key	Ocp-Apim-Subscription-Key: [A-Za-z0-9]{32}
GitHub Access Token and Refresh Token	ghp_[0-9A-Za-z]{36}
Shopify Access Token and Shared Secret	shpat_[A-Za-z0-9]{32}
Okta API Token	00[0-9A-Fa-f]{8}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{12}
JSON Web Token (JWT)	eyJ[0-9A-Za-z-*=]+\.[0-9A-Za-z-*=]+\.[A-Za-z0-9-_=]+
RSA Private Key	-----BEGIN RSA PRIVATE KEY-----[\s\S]+-----END RSA PRIVATE KEY-----
PGP Private Key	-----BEGIN PGP PRIVATE KEY BLOCK-----[\s\S]+-----END PGP PRIVATE KEY BLOCK-----
GitLab Token	glpat-[A-Za-z0-9-]{20}
Amazon Marketplace Web Services Auth Token	amzn\.mws\.[a-zA-Z0-9]{64}
Bearer Token	Bearer [A-Za-z0-9-*=]+\.[A-Za-z0-9-*=]+\.[A-Za-z0-9-_=]+
JIRA API Token	jira\.api\.token\.[A-Za-z0-9-_]+

Other Scanners

Description	Pattern
Standard Email Address	[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+
Standard IBAN Code	[A-Z]{2}\d{2}[A-Z0-9]{1,30}
Standard MAC Address	([0-9A-Fa-f]{2}[:-]){5}([0-9A-Fa-f]{2})
IPv4 Address	\b(?:\d{1,3}\.){3}\d{1,3}\b
IPv6 Address	([0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}
HTTP(S) URL	https?:\/\/(?:www\.)?[a-zA-Z0-9\-\.]+\.[a-zA-Z]{2,6}(\/[a-zA-Z0-9\&%_\.\/~\-]*)?
HTTP Basic Authentication Header	Basic\s[a-zA-Z0-9=:_-]+
HTTP Cookie	Set-Cookie:\s[a-zA-Z0-9_\-]+=\S+
US Passport Number	[a-zA-Z]\d{9}
US Vehicle Identification Number (VIN)	[A-HJ-NPR-Z0-9]{17}
UK National Insurance Number	[A-CEGHJ-PR-TW-Za-ceghj-pr-tw-z]\d{2}\d{6}[A-Za-z]{1}
Canadian Social Insurance Number (SIN)	\d{3}-\d{3}-\d{3}
US Social Security Number (SSN)	\b\d{3}-\d{2}-\d{4}\b