# AI Observability

For an overview of capabilities, supported providers, and cost tracking, see [AI Observability](/capabilities/ai-observability.md).

***

## Getting Started

### eBPF (Automatic)

If your services call a [supported provider](/capabilities/ai-observability.md#supported-providers), groundcover is already capturing AI API calls automatically — no SDKs, no code changes. By default, you get model identification, token counts, cost, latency, and full prompt/response content.

Open [AI Observability](https://app.groundcover.com/ai-observability) in your console to see what's there. If the page is empty, see [Troubleshooting](/use-groundcover/ai-observability/troubleshooting.md).

### SDK (Full Visibility)

To see agent workflows, tool execution chains, conversation threading, and span trees with parent-child relationships, add OpenTelemetry GenAI instrumentation to your services.

groundcover ingests traces using [OpenTelemetry GenAI Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/). Three attributes are all you need to unlock the core experience:

| Attribute               | What it enables                           |
| ----------------------- | ----------------------------------------- |
| `gen_ai.operation.name` | Span appears in AI Observability          |
| `gen_ai.provider.name`  | Provider identification and filtering     |
| `gen_ai.request.model`  | Model identification and cost attribution |

`gen_ai.operation.name` is required for spans to appear in AI Observability. `gen_ai.provider.name` and `gen_ai.request.model` are strongly recommended — without them, you lose provider filtering and cost attribution.

Everything else — messages, agent name, tool definitions, conversation ID — adds depth but isn't required to get started.

***

## SDK Instrumentation

### Approaches

| Approach                                                                         | Best for                                                                                                                                 |
| -------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| **Auto-instrumentation SDKs** (Traceloop/OpenLLMetry, LangSmith, etc.)           | Fastest path to visibility. Add a package, initialize, and your agent workflows appear. Works well for most teams.                       |
| **Native OTel provider libraries** (OpenAI, Anthropic, Google, Claude Agent SDK) | The standard the ecosystem is converging on. Provider-maintained and actively developing.                                                |
| **Manual instrumentation**                                                       | Full control. You emit `gen_ai.*` attributes directly — the data is exactly what you put in. Precise cost attribution, clean span trees. |

**Not sure where to start?** Most teams begin with **Traceloop** — one init call covers OpenAI, Anthropic, and agent frameworks (LangGraph, CrewAI, Pydantic AI) with full content capture out of the box.

```python
# pip install traceloop-sdk opentelemetry-exporter-otlp-proto-http
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from traceloop.sdk import Traceloop

Traceloop.init(
    app_name="my-service",
    exporter=OTLPSpanExporter(endpoint="<COLLECTOR_ENDPOINT>/v1/traces"),
)
# Your existing code runs unchanged — OpenAI, Anthropic, LangGraph, CrewAI all work
```

Replace `<COLLECTOR_ENDPOINT>` with your endpoint from the [**Data Sources**](https://app.groundcover.com/data-sources) page, which also has setup instructions for other SDKs and agent frameworks.

### Adding Depth

Once you have the core attributes above, more attributes unlock more value:

| Add this                                                      | Get this                                  |
| ------------------------------------------------------------- | ----------------------------------------- |
| `gen_ai.usage.input_tokens` + `gen_ai.usage.output_tokens`    | Token counts and cost attribution         |
| `gen_ai.conversation.id`                                      | Multi-turn session grouping               |
| `gen_ai.agent.name`                                           | Per-agent identification and analytics    |
| `gen_ai.input.messages` + `gen_ai.output.messages`            | Prompt/response content in trace drawer   |
| Framework spans (from your agent framework's SDK integration) | Full span trees and behavioral visibility |

For the complete list of supported attributes, see [Span Attributes](/use-groundcover/ai-observability/attributes.md). For query patterns using these attributes, see [Example Queries](/use-groundcover/ai-observability/example-queries.md).

***

## Verify Setup

After deployment, verify data is flowing:

1. **Generate AI traffic** — make a few calls to a [supported provider](/capabilities/ai-observability.md#supported-providers) from a service running on a monitored node
2. **Open AI Observability** — navigate to [AI Observability](https://app.groundcover.com/ai-observability) in your console
3. **Check for spans** — within 1-2 minutes, GenAI spans should appear
4. **Verify data** — confirm `gen_ai.operation.name` is populated on individual spans. You can also run this query in [Data Explorer](/use-groundcover/querying-your-groundcover-data/explore-and-monitors-query-builder.md) to check programmatically:

   ```gcql
   span_type:gen_ai | stats count()
   ```

   If `gen_ai.request.model` and `gen_ai.provider.name` are also present, cost attribution and provider filtering will be available.

If no data appears after several minutes, see [Troubleshooting](/use-groundcover/ai-observability/troubleshooting.md).

***

## OTel GenAI Conventions

The OpenTelemetry GenAI semantic conventions have evolved through three generations. groundcover normalizes all of them automatically — your data displays consistently regardless of which convention your SDK emits.

| Era       | Key Attributes                                                          | Status  | Used By            |
| --------- | ----------------------------------------------------------------------- | ------- | ------------------ |
| **Era 1** | `gen_ai.prompt` `gen_ai.completion` `gen_ai.system`                     | Legacy  | Most SDKs today    |
| **Era 2** | Per-message log events                                                  | Legacy  | Near-zero adoption |
| **Era 3** | `gen_ai.input.messages` `gen_ai.output.messages` `gen_ai.provider.name` | Current | Official OTel      |

{% hint style="info" %}
Most SDKs still emit Era 1 attributes — this is normal. groundcover maps them to the current convention automatically. You don't need to change anything.
{% endhint %}

***

## Data Privacy

By default, groundcover captures full prompt and response content for supported providers. For teams that need to disable AI data collection or strip content from storage, see [Privacy Controls](/use-groundcover/ai-observability/privacy-controls.md).


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.groundcover.com/use-groundcover/ai-observability.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.