AI Observability

For an overview of capabilities, supported providers, and cost tracking, see AI Observability.

---

Getting Started

eBPF (Automatic)

If your services call a supported provider, groundcover is already capturing AI API calls automatically — no SDKs, no code changes. By default, you get model identification, token counts, cost, latency, and full prompt/response content.

Open AI Observability in your console to see what's there. If the page is empty, see Troubleshooting.

SDK (Full Visibility)

To see agent workflows, tool execution chains, conversation threading, and span trees with parent-child relationships, add OpenTelemetry GenAI instrumentation to your services.

groundcover ingests traces using OpenTelemetry GenAI Semantic Conventions. Three attributes are all you need to unlock the core experience:

Attribute	What it enables
gen_ai.operation.name	Span appears in AI Observability
gen_ai.provider.name	Provider identification and filtering
gen_ai.request.model	Model identification and cost attribution

gen_ai.operation.name is required for spans to appear in AI Observability. gen_ai.provider.name and gen_ai.request.model are strongly recommended — without them, you lose provider filtering and cost attribution.

Everything else — messages, agent name, tool definitions, conversation ID — adds depth but isn't required to get started.

---

SDK Instrumentation

Approaches

Approach	Best for
Auto-instrumentation SDKs (Traceloop/OpenLLMetry, LangSmith, etc.)	Fastest path to visibility. Add a package, initialize, and your agent workflows appear. Works well for most teams.
Native OTel provider libraries (OpenAI, Anthropic, Google, Claude Agent SDK)	The standard the ecosystem is converging on. Provider-maintained and actively developing.
Manual instrumentation	Full control. You emit gen_ai.* attributes directly — the data is exactly what you put in. Precise cost attribution, clean span trees.

Not sure where to start? Most teams begin with Traceloop — one init call covers OpenAI, Anthropic, and agent frameworks (LangGraph, CrewAI, Pydantic AI) with full content capture out of the box.

# pip install traceloop-sdk opentelemetry-exporter-otlp-proto-http
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from traceloop.sdk import Traceloop

Traceloop.init(
    app_name="my-service",
    exporter=OTLPSpanExporter(endpoint="<COLLECTOR_ENDPOINT>/v1/traces"),
)
# Your existing code runs unchanged — OpenAI, Anthropic, LangGraph, CrewAI all work

Replace <COLLECTOR_ENDPOINT> with your endpoint from the Data Sources page, which also has setup instructions for other SDKs and agent frameworks.

Adding Depth

Once you have the core attributes above, more attributes unlock more value:

Add this	Get this
gen_ai.usage.input_tokens + gen_ai.usage.output_tokens	Token counts and cost attribution
gen_ai.conversation.id	Multi-turn session grouping
gen_ai.agent.name	Per-agent identification and analytics
gen_ai.input.messages + gen_ai.output.messages	Prompt/response content in trace drawer
Framework spans (from your agent framework's SDK integration)	Full span trees and behavioral visibility

For the complete list of supported attributes, see Span Attributes. For query patterns using these attributes, see Example Queries.

---

Verify Setup

After deployment, verify data is flowing:

1. Generate AI traffic — make a few calls to a supported provider from a service running on a monitored node

2. Open AI Observability — navigate to AI Observability in your console

3. Check for spans — within 1-2 minutes, GenAI spans should appear

4. Verify data — confirm gen_ai.operation.name is populated on individual spans. You can also run this query in Data Explorer to check programmatically:

   span_type:gen_ai | stats count()

   If gen_ai.request.model and gen_ai.provider.name are also present, cost attribution and provider filtering will be available.

If no data appears after several minutes, see Troubleshooting.

---

OTel GenAI Conventions

The OpenTelemetry GenAI semantic conventions have evolved through three generations. groundcover normalizes all of them automatically — your data displays consistently regardless of which convention your SDK emits.

Era	Key Attributes	Status	Used By
Era 1	gen_ai.prompt gen_ai.completion gen_ai.system	Legacy	Most SDKs today
Era 2	Per-message log events	Legacy	Near-zero adoption
Era 3	gen_ai.input.messages gen_ai.output.messages gen_ai.provider.name	Current	Official OTel

Most SDKs still emit Era 1 attributes — this is normal. groundcover maps them to the current convention automatically. You don't need to change anything.

---

Data Privacy

By default, groundcover captures full prompt and response content for supported providers. For teams that need to disable AI data collection or strip content from storage, see Privacy Controls.