> For the complete documentation index, see [llms.txt](https://docs.groundcover.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.groundcover.com/~/revisions/ETrLpNk6KtHjyaVUTLoE/log-parsing-with-opentelemetry-pipelines.md).
# Log Parsing with OpenTelemetry Pipelines
## Overview
groundcover supports the configuration of log pipelines using OpenTelemetry Transformation Language (OTTL) to process and customize your logs.\
With OTTL, you gain full flexibility to transform data as it flows into the platform.
### Transforming Data with OTTL
groundcover uses OTTL to enrich and shape log data inside your monitored environments.\
OTTL pipelines give you a structured way to parse, filter, and modify logs before ingestion.
Each pipeline is made up of transformation steps—each step defines a specific operation (like parsing JSON, extracting key-value pairs, or modifying attributes).\
You can configure these transformations directly in your groundcover deployment.
To test your logic before going live, we recommend using our **Parsing Playground** (click the top right corner when viewing a specific log).
### Required Attributes
To define an OTTL pipeline, make sure to include the following fields:
* `statements` – List of transformations to apply.
* `conditions` – Logic for when the rule should trigger.
* `statementsErrorMode` – How to handle errors (e.g., skip, fail).
* `conditionLogicOperator` – Used when you define multiple conditions.
## Deploying OTTL in groundcover
Rules are defined as a list of steps which are executed one after another. The rules can be viewed and edited by admins in the settings tab, under "Pipelines".
The rules run in groundcover's sensors. This is ideal for cost savings, as the original logs are not sent to or stored in the backend. This is useful particularly when the pipeline is used to drop logs.
{% hint style="info" %}
For the rules to take place, groundcover sensors need to be configured with [Ingestion Keys](/~/revisions/ETrLpNk6KtHjyaVUTLoE/use-groundcover/remote-access-and-apis/ingestion-keys.md) that allow them to pull remote configuration from the [Fleet Manager](/~/revisions/ETrLpNk6KtHjyaVUTLoE/use-groundcover/fleet-manager.md#remote-configuration).
{% endhint %}
The pipeline is stored in `yaml` format and can be edited in the UI. The result `yaml` can be exported to be used with groundcover's terraform provider, if you prefer to use a version control system (e.g. `git`). Notice that the pipeline is a singleton resource, so your terraform script must only define a single one.
{% hint style="info" %}
Each rule must have a unique and non-empty `ruleName`
{% endhint %}
{% hint style="warning" %}
The logs pipeline was previously editable using helm values. Rules that exist in the sensor's values are executed prior to the ones received via remoute configuration, and will not be visible in UI.
Editing the pipeline in sensor's values will be removed in the future, in favor of the UI and terraform.
{% endhint %}
#### Example Structure
```
ottlRules:
- ruleName: "rule1"
conditions:
- 'workload == "service1" or workload == "service2"'
statements:
- statement1
- statement2
- ruleName: "rule2"
conditions:
- 'level == "debug" or container_name == "test"'
statements:
- statement1
- statement2
```
### Setting Conditions
Use `conditions` to apply transformations only when specific attributes match. This ensures your pipeline runs efficiently and only on relevant logs.
Common fields you can use:
* `workload` – Name of the service or app.
* `container_name` – Container where the log originated.
* `level` – Log severity (e.g., info, error).
* `format` – Log format (e.g., JSON, CLF, unknown).
### Writing OTTL Statements
Some commonly used functions in groundcover:
* `ExtractGrokPatterns`
* `ParseJSON`
* `Replace_pattern`
* `Delete_key`
* `ToLowerCase`
* `Concat`
* `ParseKeyValue`
## Examples
### Simple GROK Pattern Extraction
#### **Log**
```json
{
"body": "2025-03-23 10:30:45 INFO User login attempt from 192.168.1.100"
}
```
#### **Statements**
```yaml
- 'set(cache, ExtractGrokPatterns(body, "^%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{LOGLEVEL:level}%{SPACE}User login attempt from %{IP:source_ip}"))'
- 'merge_maps(attributes, cache, "insert")'
```
#### **Results**
```json
{
"timestamp": "2025-03-23 10:30:45",
"level": "INFO",
"source_ip": "192.168.1.100"
}
```
***
### Grok + Replace + ParseKeyValue
#### **Log**
```json
{
"body": "2025-03-23 15:20:12,512 - EventProcessor - DEBUG - Completed event processing [analyzer_name=disk-space-check] [node_id=7f5e9aa8412d4c0003a7b2c5] [service_id=813dd10298f77700029d54e3] [sensor_id=3] [tracking_code=19fd5b6e72c7e94088a9ff3d] [log_id=b'67acfe0c92d43000'] [instance_id=microservice-7894563210]"
}
```
#### **Statements**
```yaml
- 'set(cache, ExtractGrokPatterns(body, "^%{TIMESTAMP_ISO8601:timestamp}%{SPACE}-%{SPACE}%{NOTSPACE}%{SPACE}%{NOTSPACE}%{SPACE}%{LOGLEVEL:level}%{DATA}(?\\[%{GREEDYDATA})"))'
- 'replace_pattern(cache["kv"], "[\\[\\]]", "")'
- 'merge_maps(attributes, ParseKeyValue(cache["kv"]), "insert")'
- 'set(attributes["timestamp"], cache["timestamp"])'
```
#### **Results**
```json
{
"instance_id": "microservice-7894563210",
"analyzer_name": "disk-space-check",
"node_id": "7f5e9aa8412d4c0003a7b2c5",
"service_id": "813dd10298f77700029d54e3",
"sensor_id": "3",
"tracking_code": "19fd5b6e72c7e94088a9ff3d",
"log_id": "b67acfe0c92d43000",
"timestamp": "2025-03-23 15:20:12,512"
}
```
***
### Grok + ToLowerCase + ParseJSON
#### **Log**
```json
{
"body": "2025-03-23 14:55:12,456 ERROR {\"event\":\"user_login\",\"user_id\":12345,\"status\":\"failed\",\"ip\":\"192.168.1.10\"}"
}
```
#### **Statements**
```yaml
- 'set(cache, ExtractGrokPatterns(body, "^%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{LOGLEVEL:level}%{SPACE}(?\\{.*\\})"))'
- 'set(attributes["timestamp"], cache["timestamp"])'
- 'set("level", ToLowerCase(cache["level"]))'
- 'set(cache["parsed_json"], ParseJSON(cache["json_body"]))'
- 'merge_maps(attributes, cache["parsed_json"], "insert")'
```
#### **Results**
```json
{
"timestamp": "2025-03-23 14:55:12,456",
"level": "error",
"event": "user_login",
"user_id": 12345,
"status": "failed",
"ip": "192.168.1.10"
}
```
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://docs.groundcover.com/~/revisions/ETrLpNk6KtHjyaVUTLoE/log-parsing-with-opentelemetry-pipelines.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.