# Drop Logs

### Overview

Filter out noisy, irrelevant, or non-critical logs at ingestion. By dropping logs at the sensor level—before they're sent to storage—you can significantly reduce costs while focusing on the logs that matter most.

### Why Drop Logs?

Not all logs provide equal value. Dropping logs allows you to:

* **Reduce data volume and storage costs** by filtering out noise
* **Focus on critical information** without distractions
* **Improve query performance** by reducing the dataset size
* **Save on storage costs** since logs are dropped at the sensor

{% hint style="success" %}
Logs are dropped **in the sensor** - the earliest point possible - achieving maximum savings in terms of networking and ingestion.
{% endhint %}

### How Log Dropping Works

To drop a log, you define a condition (e.g., log message content, container name, log level) and use a `set(drop, true)` statement. If the condition matches, the log is dropped before ingestion.

**Key Concepts:**

* `drop` - A reserved field. If set to `true`, the log is filtered out
* `where` - Used to specify when the dropping should apply
* `IsMatch()` - A function for matching regex or substrings within fields

### Best Practices

1. **Be specific with conditions** - Use precise conditions to avoid accidentally dropping important logs
2. **Test before deploying** - Use the Parsing Playground to verify your drop rules
3. **Start conservative** - Begin by dropping obvious noise, then expand gradually
4. **Monitor the impact** - Track how many logs are being dropped and adjust as needed
5. **Document your rules** - Use clear `ruleName` values to explain what each rule does
6. **Consider performance** - Dropping logs early saves processing, networking, and storage costs
7. **Avoid dropping errors** - Unless absolutely necessary, preserve error and warning logs

### Common Use Cases

#### Drop Health Check Logs

Health check endpoints often generate high-volume, low-value logs.

```yaml
ottlRules:
  - ruleName: "drop_health_checks"
    conditions:
      - 'container_name == "nginx"'
    statements:
      - 'set(drop, true) where IsMatch(body, "GET /healthz")'
```

💡 **What it does:** Filters out logs with requests to the `/healthz` endpoint from nginx containers.

#### Drop Debug Logs from Specific Services

Development or verbose logging can overwhelm production logs.

```yaml
ottlRules:
  - ruleName: "drop_debug_logs"
    conditions:
      - 'level == "debug"'
      - 'workload == "groundcover-demo"'
    conditionLogicOperator: "and"
    statements:
      - 'set(drop, true)'
```

💡 **What it does:** Filters out debug logs from the `groundcover-demo` service.

#### Drop Logs Based on Multiple Patterns

Filter out various monitoring and internal checks.

```yaml
ottlRules:
  - ruleName: "drop_monitoring_logs"
    conditions:
      - 'namespace == "production"'
    statements:
      - 'set(drop, true) where IsMatch(body, "/metrics")'
      - 'set(drop, true) where IsMatch(body, "/readyz")'
      - 'set(drop, true) where IsMatch(body, "/livez")'
```

💡 **What it does:** Drops logs containing metrics, readiness, or liveness probe requests in production.

#### Drop Logs by Status Code

Filter out successful requests to focus on errors.

```yaml
ottlRules:
  - ruleName: "drop_successful_requests"
    conditions:
      - 'workload == "api-gateway"'
      - 'attributes["status_code"] != nil'
    conditionLogicOperator: "and"
    statements:
      - 'set(drop, true) where Int(attributes["status_code"]) >= 200 and Int(attributes["status_code"]) < 300'
```

💡 **What it does:** Drops logs with 2xx status codes from the API gateway to focus on errors and unusual behavior.

#### Drop Logs from Test Environments

Reduce noise from non-production environments.

```yaml
ottlRules:
  - ruleName: "drop_test_logs"
    conditions:
      - 'namespace == "test" or namespace == "staging"'
    conditionLogicOperator: "or"
    statements:
      - 'set(drop, true)'
```

💡 **What it does:** Drops all logs from test and staging namespaces.

#### Conditional Dropping with Complex Logic

Combine multiple conditions for fine-grained control.

```yaml
ottlRules:
  - ruleName: "drop_low_priority"
    conditions:
      - 'workload == "background-worker"'
    statements:
      - 'set(drop, true) where level == "info" and IsMatch(body, "job completed")'
      - 'set(drop, true) where level == "debug"'
```

💡 **What it does:** Drops info-level "job completed" messages and all debug logs from background workers.

### Key Functions

#### set(drop, true)

Marks a log for dropping. Can be used with or without conditions.

```yaml
# Drop all logs matching the rule
- 'set(drop, true)'

# Drop only when specific condition is met
- 'set(drop, true) where IsMatch(body, "pattern")'
```

#### IsMatch()

Checks if a field matches a regex pattern or substring.

```yaml
# Simple substring match
- 'set(drop, true) where IsMatch(body, "/healthz")'

# Regex pattern match
- 'set(drop, true) where IsMatch(body, "^ERROR.*timeout")'
```

#### Combining Conditions

Use logical operators to create complex filtering rules.

```yaml
# Using conditionLogicOperator at rule level
ottlRules:
  - ruleName: "example"
    conditions:
      - 'condition1'
      - 'condition2'
    conditionLogicOperator: "and"  # or "or"
    statements:
      - 'set(drop, true)'

# Using inline conditions in statements
- 'set(drop, true) where condition1 and condition2'
- 'set(drop, true) where condition1 or condition2'
```


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.groundcover.com/use-groundcover/data-pipelines/log-pipelines/drop-logs.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.