# Drop Logs

### Overview

Filter out noisy, irrelevant, or non-critical logs at ingestion. By dropping logs at the sensor level—before they're sent to storage—you can significantly reduce costs while focusing on the logs that matter most.

### Why Drop Logs?

Not all logs provide equal value. Dropping logs allows you to:

* **Reduce data volume and storage costs** by filtering out noise
* **Focus on critical information** without distractions
* **Improve query performance** by reducing the dataset size
* **Save on storage costs** since logs are dropped at the sensor

{% hint style="success" %}
Logs are dropped **in the sensor** - the earliest point possible - achieving maximum savings in terms of networking and ingestion.
{% endhint %}

### How Log Dropping Works

To drop a log, you define a condition (e.g., log message content, container name, log level) and use a `set(drop, true)` statement. If the condition matches, the log is dropped before ingestion.

**Key Concepts:**

* `drop` - A reserved field. If set to `true`, the log is filtered out
* `where` - Used to specify when the dropping should apply
* `IsMatch()` - A function for matching regex or substrings within fields

### Best Practices

1. **Be specific with conditions** - Use precise conditions to avoid accidentally dropping important logs
2. **Test before deploying** - Use the Parsing Playground to verify your drop rules
3. **Start conservative** - Begin by dropping obvious noise, then expand gradually
4. **Monitor the impact** - Track how many logs are being dropped and adjust as needed
5. **Document your rules** - Use clear `ruleName` values to explain what each rule does
6. **Consider performance** - Dropping logs early saves processing, networking, and storage costs
7. **Avoid dropping errors** - Unless absolutely necessary, preserve error and warning logs

### Common Use Cases

#### Drop Health Check Logs

Health check endpoints often generate high-volume, low-value logs.

```yaml
ottlRules:
  - ruleName: "drop_health_checks"
    conditions:
      - 'container_name == "nginx"'
    statements:
      - 'set(drop, true) where IsMatch(body, "GET /healthz")'
```

💡 **What it does:** Filters out logs with requests to the `/healthz` endpoint from nginx containers.

#### Drop Debug Logs from Specific Services

Development or verbose logging can overwhelm production logs.

```yaml
ottlRules:
  - ruleName: "drop_debug_logs"
    conditions:
      - 'level == "debug"'
      - 'workload == "groundcover-demo"'
    conditionLogicOperator: "and"
    statements:
      - 'set(drop, true)'
```

💡 **What it does:** Filters out debug logs from the `groundcover-demo` service.

#### Drop Logs Based on Multiple Patterns

Filter out various monitoring and internal checks.

```yaml
ottlRules:
  - ruleName: "drop_monitoring_logs"
    conditions:
      - 'namespace == "production"'
    statements:
      - 'set(drop, true) where IsMatch(body, "/metrics")'
      - 'set(drop, true) where IsMatch(body, "/readyz")'
      - 'set(drop, true) where IsMatch(body, "/livez")'
```

💡 **What it does:** Drops logs containing metrics, readiness, or liveness probe requests in production.

#### Drop Logs by Status Code

Filter out successful requests to focus on errors.

```yaml
ottlRules:
  - ruleName: "drop_successful_requests"
    conditions:
      - 'workload == "api-gateway"'
      - 'attributes["status_code"] != nil'
    conditionLogicOperator: "and"
    statements:
      - 'set(drop, true) where Int(attributes["status_code"]) >= 200 and Int(attributes["status_code"]) < 300'
```

💡 **What it does:** Drops logs with 2xx status codes from the API gateway to focus on errors and unusual behavior.

#### Drop Logs from Test Environments

Reduce noise from non-production environments.

```yaml
ottlRules:
  - ruleName: "drop_test_logs"
    conditions:
      - 'namespace == "test" or namespace == "staging"'
    conditionLogicOperator: "or"
    statements:
      - 'set(drop, true)'
```

💡 **What it does:** Drops all logs from test and staging namespaces.

#### Conditional Dropping with Complex Logic

Combine multiple conditions for fine-grained control.

```yaml
ottlRules:
  - ruleName: "drop_low_priority"
    conditions:
      - 'workload == "background-worker"'
    statements:
      - 'set(drop, true) where level == "info" and IsMatch(body, "job completed")'
      - 'set(drop, true) where level == "debug"'
```

💡 **What it does:** Drops info-level "job completed" messages and all debug logs from background workers.

### Key Functions

#### set(drop, true)

Marks a log for dropping. Can be used with or without conditions.

```yaml
# Drop all logs matching the rule
- 'set(drop, true)'

# Drop only when specific condition is met
- 'set(drop, true) where IsMatch(body, "pattern")'
```

#### IsMatch()

Checks if a field matches a regex pattern or substring.

```yaml
# Simple substring match
- 'set(drop, true) where IsMatch(body, "/healthz")'

# Regex pattern match
- 'set(drop, true) where IsMatch(body, "^ERROR.*timeout")'
```

#### Combining Conditions

Use logical operators to create complex filtering rules.

```yaml
# Using conditionLogicOperator at rule level
ottlRules:
  - ruleName: "example"
    conditions:
      - 'condition1'
      - 'condition2'
    conditionLogicOperator: "and"  # or "or"
    statements:
      - 'set(drop, true)'

# Using inline conditions in statements
- 'set(drop, true) where condition1 and condition2'
- 'set(drop, true) where condition1 or condition2'
```