AI Observability
AI Observability requires an up-to-date sensor and backend. See Installation & Updating.
Overview
AI Observability gives you visibility into every AI call your services make — what models are running, what they cost, and how they behave in production.
groundcover ingests GenAI telemetry through two methods:
OpenTelemetry Instrumentation — Send traces from any SDK emitting OTel GenAI Semantic Conventions.
eBPF Auto-Detection — The sensor captures AI API calls at the kernel level without SDKs or code changes.
All observability data stays in your cloud. With BYOC, prompt content and model responses are stored in your own infrastructure — groundcover never processes your AI data outside your environment.
Instrumentation
SDK Instrumentation
Add an OTel GenAI SDK to get full visibility into your AI workloads — agent reasoning chains, tool executions, orchestration flows, and cost attribution. groundcover normalizes traces from any SDK that emits OTel GenAI semantic conventions.
Use the Data Sources page in groundcover to generate setup instructions for your specific SDK and environment.
Any SDK that exports standard gen_ai.* attributes over OTLP is compatible.
eBPF Auto-Detection (Zero Instrumentation)
The sensor captures AI API calls at the kernel level — no SDKs, no code changes. Every call includes model, tokens, cost, latency, and full prompt/response content (with configurable redaction).
Supported providers:
OpenAI (Chat Completion API)
Anthropic (Messages API)
AWS Bedrock (InvokeModel API)
More providers are being added. For unsupported providers, use SDK instrumentation via OTLP.
GenAI Span Attributes
groundcover adheres to the OpenTelemetry GenAI Semantic Conventions. Telemetry from all SDK formats and eBPF is normalized to a consistent attribute set.
Core Attributes
The more GenAI attributes your spans carry, the richer the experience — cost analytics, prompt debugging, execution chain visibility.
gen_ai.provider.name
string
GenAI provider
openai, anthropic, aws.bedrock
gen_ai.operation.name
string
Operation type
chat, embeddings, execute_tool, invoke_agent
gen_ai.request.model
string
Model requested
gpt-4o, claude-3-sonnet
gen_ai.response.model
string
Model that responded
gpt-4o-2024-08-06
gen_ai.response.id
string
Completion ID
chatcmpl-abc123
gen_ai.usage.input_tokens
int
Tokens consumed by the prompt
150
gen_ai.usage.output_tokens
int
Tokens generated in the response
83
gen_ai.response.finish_reasons
string[]
Why the model stopped
stop, tool_calls, length
gen_ai.conversation.id
string
Session/thread identifier
conv-abc123
Full Attribute Reference
Below is the complete set of attributes captured and normalized by groundcover. These are grouped by category for reference.
Token Usage
gen_ai.usage.input_tokens
int
Tokens consumed by the prompt
gen_ai.usage.output_tokens
int
Tokens generated in the response
gen_ai.usage.cache_read.input_tokens
int
Input tokens served from provider cache
gen_ai.usage.cache_creation.input_tokens
int
Input tokens written to provider cache
Request Parameters
gen_ai.request.temperature
double
Temperature setting
gen_ai.request.max_tokens
int
Maximum tokens for the response
gen_ai.request.top_p
double
Top-p sampling
gen_ai.request.top_k
double
Top-k sampling
gen_ai.request.frequency_penalty
double
Frequency penalty
gen_ai.request.presence_penalty
double
Presence penalty
gen_ai.request.stop_sequences
string[]
Stop sequences
gen_ai.request.seed
int
Seed for reproducibility
Response Metadata
gen_ai.response.finish_reasons
string[]
Why the model stopped: stop, tool_calls, length
gen_ai.output.type
string
Content type: text, json, image, speech
gen_ai.conversation.id
string
Session/thread identifier
Content (Opt-In)
gen_ai.input.messages
JSON
Structured input messages (role + parts)
gen_ai.output.messages
JSON
Structured output messages (role + parts)
gen_ai.system_instructions
JSON
System prompt, separate from chat history
gen_ai.tool.definitions
JSON
Tool schemas passed to the model
Agent
gen_ai.agent.name
string
Human-readable agent name
gen_ai.agent.id
string
Unique agent identifier
gen_ai.agent.description
string
Agent description
gen_ai.agent.version
string
Agent version
Tool Execution
gen_ai.tool.name
string
Name of the tool
gen_ai.tool.type
string
function, extension, datastore
gen_ai.tool.call.id
string
Tool call identifier
gen_ai.tool.call.arguments
JSON
Parameters passed to the tool
gen_ai.tool.call.result
JSON
Result returned by the tool
Cost Tracking
groundcover calculates cost per GenAI span at ingestion time. Each span includes input tokens, output tokens, and cached tokens. Cost is derived from a versioned model pricing dictionary.
Cost and token data is available for filtering and sorting in the span list — find your most expensive calls by model, service, or agent.
Configuration
Sensitive Data Obfuscation
By default, groundcover collects full AI payloads. You can obfuscate specific fields using the httphandler configuration in your values.yaml.
See sensitive-data-obfuscation.md for full details.
By default groundcover does not obfuscate AI payloads.
Obfuscating Request Prompts
Obfuscating Response Content
OTel GenAI Conventions
The OpenTelemetry GenAI semantic conventions have evolved through three generations. groundcover normalizes all of them — your data displays consistently regardless of which convention your SDK emits.
Era 1
gen_ai.prompt gen_ai.completion gen_ai.system
Deprecated
Most SDKs today
Era 2
Per-message log events
Deprecated
Near-zero adoption
Era 3
gen_ai.input.messages gen_ai.output.messages gen_ai.provider.name
Current
Official OTel instrumentations
If your SDK emits Era 1 attributes, groundcover maps them to the current convention automatically. No action needed.
Using AI Observability
Spans List
The main view lists all GenAI spans with key details like service name, operation type, model, token usage, cost, and latency.
Trace Drawer
Click any span to open the trace drawer. The drawer shows:
Span tree — parent-child relationships between spans in the trace
Overview tab — structured view of messages, responses, and tool inputs/outputs
Top bar summary — total tokens, cost, LLM call count, and model across the trace
Last updated