# AI Observability

{% hint style="info" %}
AI Observability requires an up-to-date sensor and backend. See [Installation & Updating](https://docs.groundcover.com/getting-started/installation-and-updating).
{% endhint %}

## Overview

AI Observability gives you full visibility into how your services use AI — models, cost, latency, prompts, agent behavior, and tool execution. All data stays in your infrastructure with [BYOC (Bring Your Own Cloud)](https://docs.groundcover.com/architecture/byoc); groundcover never processes your AI data outside your environment.

groundcover captures AI telemetry from two sources: **eBPF auto-detection** for immediate per-call visibility with zero code changes, and **OpenTelemetry instrumentation** for the full agent-level picture.

If your sensor is up to date and your services call a supported provider, you already have data. Open [AI Observability](https://app.groundcover.com/ai-observability) in your console to see what's there.

For instrumentation setup, privacy controls, and troubleshooting, see [Using AI Observability](https://docs.groundcover.com/use-groundcover/ai-observability).

***

## Two Levels of Visibility

eBPF captures every AI API call automatically — model, tokens, cost, latency, and full prompt/response content, with zero code changes. The sensor auto-detects **OpenAI**, **Anthropic**, and **AWS Bedrock** traffic from anything running in your monitored environment: production services, CI/CD pipelines, development pods, staging. If a process runs on a monitored node and makes an HTTPS call to a supported provider, groundcover captures it.

{% hint style="info" %}
For providers not yet auto-detected, use [SDK instrumentation](https://docs.groundcover.com/use-groundcover/ai-observability#sdk-instrumentation) to send GenAI traces directly. [Contact us](https://www.groundcover.com/contact) if you need a specific provider — eBPF support is extended based on customer requests.
{% endhint %}

eBPF gives you the calls. To see the full picture — which agent triggered which call, how tool results shaped the next prompt, how a multi-step workflow reasons from start to finish — add OpenTelemetry instrumentation. SDK spans give you trace trees, agent workflows, tool execution chains, and conversation threading: not just what your AI costs, but how it thinks.

When both are active, the same call may appear twice — once from eBPF and once from the SDK. Both are correct; cost is not double-counted. When groundcover detects an SDK span for the same call, eBPF cost and token data is excluded from aggregations.

See [Using AI Observability](https://docs.groundcover.com/use-groundcover/ai-observability) for setup instructions.

***

## Cost Tracking

groundcover calculates cost per AI span at ingestion time using a maintained model pricing table that updates as providers release new models. Each span includes input tokens, output tokens, and cached tokens.

By default, every AI span is stored — no sampling, no dropping. GenAI calls are high-value and low-volume compared to typical service traffic. Every call matters for cost analysis and debugging. If you need to disable GenAI storage entirely, see [Privacy Controls](https://docs.groundcover.com/use-groundcover/ai-observability/privacy-controls#disable-genai-data-collection).

Cost and token data is available for filtering and sorting in the span list — find your most expensive calls by model, service, or agent. See [Example Queries](https://docs.groundcover.com/use-groundcover/ai-observability/example-queries) for cost analysis patterns.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.groundcover.com/capabilities/ai-observability.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.