groundcover is a full stack, cloud-native observability platform, developed to break all industry paradigms - from making instrumentation a thing of the past, to decoupling cost from data volumes
The groundcover platform consolidates all your traces, metrics, logs, and Kubernetes events into a single pane of glass, allowing you to identify issues faster than ever before and conduct granular investigations for quick remediation and long-term prevention.
Our pricing is not impacted by the volume of data generated by the environments you monitor, so you can dare to start monitoring environments that had been blind spots until now - such as your Dev and Staging clusters. This, in turn, provides you visibility into all your environments, making it much more likely to identify issues in the early stages of development, rather than in your live product.
groundcover introduces game-changing concepts to observability:
eBPF sensor
(extended Berkeley Packet Filter) is a groundbreaking technology that has significantly impacted the Linux kernel, offering a new way to safely and efficiently extend its capabilities.
By powering our sensor with eBPF, groundcover unlocks unprecedented granularity on your cloud environment, while also practically eliminating the need for human involvement in the installation and deployment process. Our unique sensor collects data directly from the Linux kernel with near-zero impact on CPU and memory.
Advantages of our eBPF sensor:
Zero instrumentation: groundcover's eBPF sensor gathers granular observability data without the need for integrating an SDK or changing your applications' code in any way. This enables all your logs, metrics, traces, and other observability data to flow automatically into the platform. In minutes, you gain full visibility into application and infrastructure health, performance, resource usage, and more.
Minimal resources footprint: groundcover’s sensor in installed on a dedicated node in each monitored cluster, operating separately from the applications it is monitoring. Without interference with the application's primary functions, the groundcover platform operates with near-zero impact on your resources, maintaining the applications' performance and avoiding unexpected overhead on the infrastructure.
Bring-your-own-cloud (BYOC) architecture
The one-of-a-kind architecture on which groundcover was built eliminates all requirements to stream your logs, metrics, traces, and other monitoring data outside of your environment and into a third-party's cloud. By leveraging integrations with best-of-breed technologies, including ClickHouse and Victoria Metrics, all your observability is stored data locally, with the ability of being fully managed by groundcover.
Advantages of our BYOC architecture:
By separating the data plane from the control plane, you get the advantages of a SaaS solution, without its security and privacy challenges.
With multiple deployment models available, you also get to choose the level of security and privacy your organization needs, up to the highest standards (FedRamp-level).
Automated deployment, maintenance & resource optimization with our deployment option.
This concept is unique to groundcover, and takes a while to grasp. Read about our BYOC architecture more in detail in.
Learn about groundcover (currently available only on a ), which enables you to deploy groundcover's control plane inside your own environment and delegate the entire setup and management of the groundcover platform.
Disruptive pricing model
Enabled by our unique BYOC architecture, groundcover's vision is to revolutionize the industry by offering a pricing model that is unheard of anywhere else. Our fully transparent pricing model is based only on the number of nodes being monitored, and the costs of hosting the groundcover backend in your environment. Volume of logs, metrics, traces, and all other observability data don’t affect your cost. This results in savings of 60-90% compared to SaaS platforms.
In addition, all our subscription tiers never limit your access to features and capabilities.
Advantages of our nodes-based pricing model:
Cost is predictable and transparent, becoming an enabler of growth and expansion.
The ability to deploy groundcover in data-intensive environments enables the monitoring of Dev and Staging clusters, which promotes early identification of issues.
No cardinality or retention limits
Read our latest customer stories to learn how organization of varying sizes all reduce their observability costs dramatically by migrating to groundcover:
Stream processing
groundcover applies a stream processing approach to collect and control the continuous flow of data to gain immediate insights, detect anomalies, and respond to changing conditions. Unlike batch processing, where data is collected over a period and then analyzed, stream processing analyzes the data as it flows through the system.
Our platform uses a distributed stream processing engine that enables it to ingest huge amounts of data (such as logs, traces and Kubernetes events) in real time. It also processes all that data and instantly generates complex insights (such as metrics and context) based on it.
As a result, the volume of raw data stored dramatically decreases which, in turn, further reduces the overall cost of observability.
Capabilities
Log Management
Designed for high scalability and rapid query performance, enabling quick and efficient log analysis from all your environments. Each log is enriched with actionable context and correlated with relevant metrics and traces, providing a comprehensive view for fast troubleshooting.
Infrastructure Monitoring
The groundcover platform provides cloud-native infrastructure monitoring, enabling automatic collection and real-time monitoring of infrastructure health and efficiency.
Application Performance Monitoring (APM)
Gain end-to-end observability into your applications performance, identify and resolve issues instantly, all with zero code changes.
Real User Monitoring (RUM)
Real User Monitoring (RUM) extends groundcover’s observability platform to the client side, providing visibility into actual user interactions and front-end performance. It tracks key aspects of your web application as experienced by real users, then correlates them with backend metrics, logs, and traces for a full-stack view of your system.
FAQ
How much does groundcover cost?
groundcover's unique pricing model is the first to decouple data volumes from cost of owning and operating the solution. For example, subscribing to our costs $30 per node/host per month.
Overall, the cost of owning and operating groundcover is based on two factors:
Application Performance Monitoring (APM)
Gain end-to-end observability into your applications performance, identify and resolve issues instantly - all with zero code changes.
Overview
The groundcover platform collects data all across your stack using the power of eBPF instrumentation. Our is installed in seconds and provides 100% coverage into application metrics and traces with zero code changes or configurations.
Resolve faster - By seamlessly correlating traces with application metrics, logs, and infrastructure events, groundcover’s APM enables you to detect and resolve root issues faster.
The number of nodes (hosts) you are running in the environment you are monitoring
The costs of hosting groundcover's backend in your environment
Check out our TCO calculator to simulate your total cost of ownership for groundcover.
Can I use groundcover across multiple clusters?
Definitely. As you deploy groundcover each cluster is automatically assigned the unique name it holds inside your cloud environment. You can browse and select all your clusters at one place with our UI experience.
What K8s flavors are supported?
groundcover has been tested and validated on the most common K8s distributions. See full list in the Requirements section.
What protocols are supported?
groundcover supports the most common protocols in most K8s production environments out-of-the-box. See full list here.
What types of data does groundcover collect?
groundcover's kernel-level eBPF sensor automatically collects your logs, application metrics (such as latency, throughput, error rate and much more), infrastructure metrics (such as deployment updates, container crashes etc.), traces, and Kubernetes events. You can control which data is left out of the automatic collection using data obfuscation.
Where is my data being stored?
groundcover stores all the data it collects inside your environment, using the state-of-the-art storage services of ClickHouse and Victoria Metrics, with the option to offload data to object storage such as S3 for long-term retention. See our Architecture section for more details.
Is my data secure?
groundcover stores the data it collects in-cluster, inside your environment without ever leaving the cluster to be stored anywhere else.
Our SaaS UI experience stores only information related to the account, user access and general K8s metadata used for governance (like the number of nodes per cluster, the name given to the cluster etc.).
All the information served to the UI experience is encrypted all the way to the in-cluster data sources. groundcover has no access to your collected data, which is accessible only to an authenticated user from your organization.
groundcover does collect telemetry information (opt-out is of course possible) which includes metrics about the performance of the deployment (e.g. resource consumption metrics) and logs reported from the groundcover components running in the cluster.
All telemetry information is anonymized, and contains no data related to your environment.
Regardless, groundcover is SOC2 and ISO 27001 compliant and follows best practices.
How can I invite my team to my workspace?
If you used your business email to create your groundcover account, you can invite your team to your workspace by clicking on the purple "Invite" button on the upper menu. This will open a pop-up where you can enter the emails of the people you want to invite. You also have an option to copy and share your private link.
Note: The Admin of the account (i.e. the person that created it) can also invite users outside of your email domain. Non-admin users can only invite users that share the same email domain.
If you used a private email, you can only share the link to your workspace by clicking the "Share" button on the top bar.
groundcover's CLI tool is currently Open Source along side more projects like Murre and Caretta. We're working on releasing more parts of our solution to Open Source very soon. Stay tuned in our GitHub page!
What operating system (OS) do I need to use groundcover?
groundcover’s sensor uses eBPF, which means it can only deployed on a Kubernetes cluster that is running on a Linux system.
Installing using the CLI command is currently only supported on Linux and Mac.
You can install using the Helm command from any operating system.
Once installed, accessing the groundcover platform is possible from any web browser, on any operating system.
A new level of insight granularity: With direct access to the Linux kernel, our eBPF sensor enables the collection of data straight from the source. This guarantees that the data is clean, unaltered, and precise. It also offers access to unique insight on your application and infrastructure, such as the ability to view the full traces of payloads, or analyzing network performance over time.
Nobl9 expands monitoring to cover production e2e, including testing and staging environments
Replacing Datadog with groundcover cut Nobl9’s observability costs in half while improving log coverage, providing deeper granularity on traces with eBPF, and enabling operational growth and scalability.
Tracr eliminates blind spots with native-K8s observability and eBPF tracing
Tracr migrates from a fragmented observability stack to groundcover, gaining deep Kubernetes visibility, automated eBPF tracing, and a cost-effective monitoring solution. This transition streamlined troubleshooting, expanded observability across teams, and enhanced the reliability of their blockchain infrastructure.
Improve user experience - Optimize your application performance and resource utilization faster than ever before, avoid downtimes and make poor end-user experience a thing of the past.
Collection
Our revolutionary eBPF sensor, Flora, is deployed as a DaemonSet in your Kubernetes cluster. This approach allows us to inspect every packet that each service is sending or receiving, achieving 100% coverage. No sampling rates or relying on statistical luck - all requests and responses are observed.
This approach would not be feasible without a resource-efficient eBPF-powered sensor. eBPF not only extends the ability to pinpoint issues - it does so with much less overhead than any other method. eBPF can be used to analyze traffic originating from every programming language and SDK - even for encrypted connections!
Click here for a full list of supported technologies
Reconstruction
After being collected by our eBPF code, the traffic is then classified according to its protocol - which is identified directly from the underlying traffic, or the library from which it originated. Connections are reconstructed, and we can generate transactions - HTTP requests and responses, SQL queries and responses etc.
Enrichment
In order to give as much context as possible each transaction is enriched with as much metadata as possible. Some examples might include the pods that took part in this transaction (both client and server), the nodes on which these pods are scheduled, and the state of container at the time of the request.
It is important to emphasize the impressive granularity level with which this process takes place - every single transaction observed is fully enriched. This allows us to perform more advanced aggregations.
Aggregation
After being enriched by as much context as possible, the transactions as grouped together into meaningful aggregations. These could be defined by the workloads involved, the protocols detected and the resources that were accessed in the operations. These aggregations will mostly come into play when displaying golden signals.
Exporting
After collecting the data, contextualizing it and putting it together in meaningful aggregations - we can now create metrics and traces to provide meaningful insights into the services' behaviors.
Stream, store, and query your logs at any scale, for a fixed cost.
Overview
Our Log Management solution is built for high scale and fast query performance so you can analyze logs quickly and effectively from all your cloud environments.
Gain context - Each log data is enriched with actionable context and correlated with relevant metrics and traces in one single view so you can find what you’re looking for and troubleshoot, faster.
Centralize to maximize - The groundcover platform can act as a limitless, centralized log management hub. Your
Monitor Catalog page
Explore and select pre-built Monitors from the catalog to quickly set up observability for your environment. Customize and deploy Monitors in just a few clicks.
Overview
The Monitor Catalog is a library of pre-built templates to efficiently create new Monitors. Browse and select one or more Monitors to quickly configure their environment with a single click. The Catalog groups monitors into "Packs", based on different use cases.
Real-world Use Cases
These are patterns we've seen in the wild. Agents use groundcover to debug, monitor, and close the loop.
Test → Logs → Fix
Cursor generates tests, tags each with a test_id, logs them, and then uses groundcover to instantly fetch related log lines.
Investigate Issues via Cursor
Got a monitor firing? Drop the alert into Cursor. The agent runs a quick RCA, queries groundcover, and even suggests a patch based on recent logs and traces.
Support Workflow
Support rep gets an error ID → uses MCP to query groundcover → jumps straight to the root cause by exploring traces and logs around the error.
The Autonomous Loop
An agent picks up a ticket, writes tests, ships code to staging, monitors it with groundcover, checks logs and traces, and verifies the fix end to end.
Yes, really. Full loop. Almost no hands.
Key Features
Batch Monitor Creation
You can select as many monitors as you wish, and add them all in one click. Select a complete pack or multiple Monitors from different packs, then click "Create Monitor". All Monitors will be automatically created. You can always edit them later.
Single Monitor Creation
You can also create a single Monitor from the Catalog. When hovering over a Monitor, a "Wizard" button will appear. Clicking on it will direct you to the Monitor Creation Wizard where you can review and edit before creation.
Monitors Catalog
CPU architectures
The following architectures are fully supported for all groundcover workloads:
x86
ARM
Service Accounts
A service account is a non-human identity for API access, governed by RBAC and supporting multiple API keys.
Summary
Service accounts in groundcover are non-human identities used for programmatic access to the API. They’re ideal for CI pipelines, automation, and backend services, and are governed by groundcover’s RBAC system.
Identity and Permissions
A service account has a name and email, but it cannot be used to log into the UI or via SSO. Instead, it functions purely for API access. Each account must have at least one RBAC policy assigned, which defines its permission level (Admin, Editor, Viewer) and data scope. Multiple policies can be attached to broaden access; effective permissions are the union of all policies.
Creation and Management
Only Admins can create, update, or delete service accounts. This can be done via the UI (Settings → Access → Service Accounts) or API. During creation, Admins define the name, email, and initial policies. You can edit service account, changing email address and assigned policies, but can't rename.
API Key Association
A service account can have multiple API keys. This makes it easy to rotate credentials or issue distinct keys for different use cases. All keys are tied to the same account and carry its permissions. Any action taken using a key is logged as performed by the associated service account.
Embedded Grafana
are completely unaffected by the amount of logs you choose to store or query. It's entirely up to you to decide.
Collection
Seamless log collection
groundcover ensures a seamless log collection experience with our proprietary eBPF sensor, which automatically collects and aggregates all logs in all formats - including JSON, plain text, NGINX logs, and more. All this without any configuration needed.
This sensor is deployed as a DaemonSet, running a single pod on each node within your Kubernetes cluster. This configuration enables the groundcover platform to automatically collect logs from all of your pods, across all namespaces in your cluster. This means that once you've installed groundcover, no further action is needed on your part for log collection. The logs collected by each sensor instance are then channeled to the OTel Collector.
OTel Collector: A vendor-agnostic way to receive, process and export telemetry data.
Acting as the central processing hub, the OTel Collector is a vendor-agnostic tool that receives logs from various sensor pods. It processes, enriches, and forwards the data into groundcover's ClickHouse database, where all log data from your cluster is securely stored.
Logs Attributes
Logs Attributes enable advanced filtering capabilities and is currently supported for the formats:
JSON
Common Log Format (CLF) - like those from NGNIX and Kong
logfmt
groundcover automatically detects the format of these logs, extracting key:value pairs from the original log records as Attributes.
Each attribute can be added to your filters and search queries.
Example: filtering a log in a supported format with a field of a request path "/status" will look as follows: @request.path:"/status". Syntax can be found here.
Configuration
groundcover offers the flexibility to craft tailored collection filtering rules, you can choose to set up filters and collect only the logs that are essential for your analysis, avoiding unnecessary data noise. For guidance on configuring your filters, explore our Customize Logs Collection section.
You also have the option to define the retention period for your logs in the ClickHouse database. By default, logs are retained for 3 days. To adjust this period to your preferences, visit our Customize Retention section for instructions.
Log Explorer
Once logs are collected and ingested, they are available within the groundcover platform in the Log Explorer, which is designed for quick searches and seamless exploration of your logs data. Using the Log Explorer you can troubleshoot and explore your logs with advanced search capabilities and filters, all within a clear and fast interface.
Search and filter
The Log Explorer integrates dynamic filters and a versatile search functionality that enables you to quickly and easily identify the right data. You can filter out logs by selecting one or multiple criteria, including log-level, workload, namespace and more, and can limit your search to a specific time range.
groundcover natively supports setting up log pipelines using Vector transforms. This allow for full flexibility in the processing and manipulation of logs being collected - parsing additional patterns by regex, renaming attributes, and many more.
groundcover will work out-of-the-box on all protocols, encryption libraries and runtimes below - generating traces and metrics with zero code changes.
We're growing our coverage all the time.
Cant find what you're looking for? let us know over Slack.
Supported protocols
Protocol
Status
Comments
Supported encryption libraries and runtimes
groundcover seamlessly supports APM for encrypted communication - as long as it's listed below.
Encryption Library/Runtime
Status
Comments
Encryption is unsupported for binaries which have been compiled without debug symbols ("stripped"). Known cases:
Crossplane
groundcover MCP
Supercharge your AI agents with o11y superpowers using the groundcover MCP server. Bring logs, traces, metrics, events, K8s resources, and more directly into your agent’s context — and troubleshoot side by side with your AI assistant to solve issues in minutes.
Status: Work in progress. We keep adding tools and polishing the experience. Got an idea or question? Ping us on Slack!
What is MCP?
MCP (Model Context Protocol) is an open standard that enables AI tools to access external data sources like APIs, observability platforms, and documentation directly within their working context. MCP servers expose these resources in a consistent way, so the AI agent can query and use them as needed to streamline workflows.
groundcover’s MCP server brings your live observability data into the picture - making your agents smarter, faster, and more accurate.
How Can groundcover's MCP Help You and Your Agent?
By connecting your agent to groundcover’s MCP server, you enable it to:
Query live logs, traces, and metrics for a workload, container, or issue.
Run root cause analysis (RCA) on issues, right inside your IDE or chat interface.
Auto-debug code with observability context built in.
See examples in our and .
Install groundcover's MCP Server
Set up is quick and agent-friendly. We support both OAuth (recommended) and API key flows.
Head to for setup instructions and client-specific guides.
Installation & Updating
Multiple ways to connect your infrastructure and applications to groundcover
groundcover is designed to support data ingestion from multiple sources, giving you comprehensive observability across your entire stack. Choose the installation method that best fits your infrastructure and monitoring needs.
Available Installation Options
Kubernetes Clusters
Connect your Kubernetes clusters using groundcover's eBPF-based sensor for automatic instrumentation and deep observability.
- Deploy groundcover's sensor to monitor containerized workloads, infrastructure, and applications with zero code changes
Standalone Linux Hosts
Monitor individual Linux servers, virtual machines, or cloud instances outside of Kubernetes.
- Install groundcover on standalone Linux hosts such as EC2 instances, bare metal servers, or VMs
Real User Monitoring (RUM)
Gain visibility into frontend performance and user experience with client-side monitoring.
- Monitor real user interactions, page loads, and frontend performance in web applications
External Data Sources
Integrate with existing observability tools and send data from your current monitoring stack.
- Forward traces, metrics, and logs from existing OpenTelemetry collectors
- Send data from Datadog agents while maintaining your existing setup
Getting Started
- Set up your groundcover account and workspace
Review - Ensure your environment meets the necessary prerequisites
Choose your installation method - Select the option that matches your infrastructure setup
Need Help?
If you're unsure which installation method is right for you, or if you have specific requirements, check our or reach out to our support team.
Requirements
To ensure a seamless experience with groundcover, it's important to confirm that your environment meets the necessary requirements. Please review the detailed requirements for Kubernetes, our eBPF sensor, and the necessary hardware and resources to guarantee optimal performance.
Kubernetes requirements
groundcover supports a wide range of Kubernetes versions and distributions, including popular platforms like EKS, AKS, and GKE.
Our state-of-the-art eBPF sensor leverages advanced kernel features to deliver comprehensive monitoring with minimal overhead, requiring specific Linux kernel versions, permissions, and CO:RE support.
Hardware and resource requirements
groundcover fully supports both x86 and ARM processors, ensuring compatibility across diverse environments.
ClickHouse resources
groundcover operates ClickHouse to support many of its core features. This requires suitable resources given to the deployment, which groundcover takes care of according to your data usage.
Monitor List page
View, filter, and manage all monitors in one place, and quickly identify issues or create new monitors.
The Monitor List is the central hub for managing and monitoring all active and configured Monitors. It provides a clear, filterable table view of your Monitors, with their current status and key details, such as creation date, severity, and live issues. Use this page to review your Monitors performance, identify issues, and take appropriate action.
Key Features
Monitors Table
Displays the following columns:
Name: Title of the monitor.
Creation Date: When the monitor was created.
Create Monitor
You can create a new Monitor by clicking on Create Monitor, then choosing between the different options: Monitor Wizard, Monitor Catalog, or Import. For further guidance, .
Filters Panel
Use filters to narrow down monitors by:
Severity: S1, S2, S3, or custom severity levels.
Status: Alerting or Normal.
Silenced: Exclude silenced monitors.
Tip: Toggle multiple filters to refine your view.
Search Bar
Quickly locate monitors by typing a name, status, category, or other keywords.
Cluster and Environment Filters
Located at the top-right corner, use these to focus on monitors for specific clusters or environments.
Monitors
Monitors offers the ability to define custom alerts, which you can configure using groundcover data and custom metrics.
What is a Monitor
A Monitor defines a set of rules and conditions that track the state of your system. When a monitor's conditions are met, it triggers an issue that is displayed on the Issues page and can be used for alerting using your integrations and workflows.
– Explore more advanced query patterns and techniques for complex scenarios.
5 quick steps to get you started
Once installed, we recommend following these steps to help you quickly gain the most our of groundcover's unique observability platform.
1. Get a complete view of your workloads
The "Home page" of the groundcover app is our Workloads page. From here, you can get a service-centric view,
Full Webhook Examples
This section contains comprehensive examples of webhook integrations with various third-party services. These examples provide step-by-step instructions for setting up complete workflows with external systems.
Available Examples
- Integrate with incident.io for incident management
Migrations
Automated migration from legacy vendors. Bring over your monitors, dashboards, data sources, and all the data you need - with automatic mapping and zero downtime.
Overview
groundcover is the first observability platform to ship a one-click migration tool from legacy vendors. The migration flow automatically discovers, translates, and installs your observability setup into groundcover.
Goal: Move your entire observability stack with zero manual work. We don't just migrate assets - we bring the data and handle all the mapping for you.
Metrics and Logs API
This page describes the available API endpoints for querying logs and metrics in groundcover, including how to authenticate and structure requests for general data retrieval.
Authentication
Authentication is performed using an API key generated in the of the Groundcover console.
All API requests must include the API key in the Authorization header using the following format:
Grafana Service Account Token
Step 1 - generate Grafana Service Account Token
Make sure you have
Dashboards
Learn how to build custom dashboards using groundcover
groundcover’s dashboards are designed to personalize your data visualization and maximize the value of your existing data. Dashboards are perfect for creating investigation flows for critical monitors, displaying the data you care about in a way that suits you and your team, and crafting insights from the data on groundcover.
Easily create a new Dashboard .
Key Features
Insights
Quickly understand your data with groundcover
groundcover insights give you a clear snapshot of notable events in your data. Currently, the platform supports Error Anomalies, with more insight types on the way.
Error Anomalies
Error Anomalies instantly highlight workloads, containers, or environments experiencing unusual spikes in Error or Critical logs, as well as Traces marked with an error status. These anomalies are detected using statistical algorithms, continuously refined through user feedback for accuracy.
Drilldown
The Drilldown view helps you to quickly identify and highlight the most informative attributes - those that stand out and help you pinpoint anomalies or bottlenecks.
Distribution Mode
In this mode, groundcover showcases the top attributes found in your traces or logs data. Each attribute displays up to four values with the highest occurrence across the selected traces.
You can click any value to add or exclude it as a filter and continue drilling down interactively.
Dashboards
groundcover enables access to an embedded Grafana, within the groundcover platform's interface. This enables you to easily import and continue using your existing Grafana dashboards and .
The following guides will help you setup and import your visualizations from Grafana:
Alerts
groundcover enables access to an embedded Grafana, within the groundcover platform's interface. This enables you to easily import and continue using your existing Grafana and alerts.
API Examples
Welcome to the API examples section. Here, you’ll find practical demonstrations of how to interact with our API endpoints using cURL commands. Each example is designed to help you quickly understand h
Structure of the Examples
cURL-based examples: Every example shows the exact cURL command you can copy and run directly in your terminal.
What we migrate
Monitors
Includes alert conditions, thresholds, and evaluation windows.
Dashboards
Complete dashboard migration with preserved layouts:
All widget types groundcover supports
Query translations
Time ranges and filters
Visual settings and arrangements
Data Sources
We detect what you're using in Datadog and help you set it up in groundcover. One-click migration for integrations is coming soon.
Data & Mapping
We don't just copy configurations - we ensure the data flows:
Automatic metric mapping: Datadog metric names translated to groundcover equivalents
Label translation: Tags become labels with intelligent mapping
Query conversion: Datadog query syntax converted to groundcover
Data validation: We verify all referenced metrics and data sources exist
Fetch & discover: Provide API keys. groundcover pulls your monitors, dashboards, and data sources.
Automatic setup: We install data sources, map metrics, and prepare everything.
Migrate assets: Review and migrate monitors and dashboards with one click.
API keys are not stored.
Access
Settings → Migrations (Admin role required)
What's next
The migration flow is structured to support additional asset types:
Data source configurations (available now)
Log pipelines (coming soon)
Advanced metric mappings (coming soon)
How attributes are selected
We use statistical scoring based on:
Entropy: how diverse the values of an attribute are.
Presence ratio: how often the attribute appears across the selected traces.
Attributes that are both common and have high entropy are prioritized.
Endpoint-specific demonstrations: We walk through different API endpoints one by one, highlighting the required parameters and common use cases.
Request & Response clarity: Each section contains both the request (what you send) and the response (what you get back) to illustrate expected behavior.
Prerequisites
Before running any of the examples, make sure you have:
Multi-Mode Query Bar: The Query Bar is central to dashboards and supports multiple modes fully integrated with native pages and Monitors. Currently, the modes include Metrics, Infra Metrics, Logs, and Traces. Learn more in the Query Builder section.
Variables: Built-in variables allow you to filter data quickly based on a predefined list crafted by groundcover.
Widget Types: Two widget types are currently supported:
Chart Widget: Displays data visually.
Textual Widget: Adds context to your dashboards.
Display Types: Five display types are supported for data visualization:
For a complete Prometheus REST API documentation, and available operations, see: Prometheus HTTP API
Logs API
Use the following endpoint to query the groundcover logs API:
To see usage examples on how to query logs in groundcover see: Query logs examples
Legacy API (Clickhouse)
The following configurations are deprecated but may still be in use in older setups.
The legacy datasources us using a different API key than described above. The API key can be obtained by running: groundcover auth get-datasources-api-key
You can query the legacy API to execute SQL statements directly on the Clickhouse database.
Service Account Token are only accessible once, so make sure you keep them somewhere safe, running the command again will generate a new service account token
Only groundcover tenant admins can generate Service Account Tokens
A highly impactful advantage of leveraging eBPF in our proprietary sensor is that it enables visibility on the full payloads of both request and response - including headers! This allows you to very quickly understand issues and provides context.
groundcover enables to very easily build custom dashboards to visualize your data using our intuitive Query Builder as a guide, or using your own queries.
Define custom alerts using our native Monitors, which you can configure using groundcover data and custom metrics. You can also choose from our Monitors Catalog, which contains multiple pre-built Monitors that cover a few of the most common use cases and needs.
Invites lets you share your workspaces with your colleagues in just a couple of clicks. You can find the "Invite Members" option at the bottom of the left navigation bar. Type in the email addresses of the teammates you want to invite, and set their user permissions (Admin, Editor, Read Only), then click "Send Invites".
View and analyze monitor issues with detailed timelines, metadata, and context to quickly identify and resolve problems in your environment.
The Issues page provides a detailed view of active and resolved issues triggered by Monitors. This page helps users investigate, analyze, and resolve problems in their environment by visualizing issue trends and providing in-depth context through an issue drawer.
Issues drawer
Clicking on an issue in the Issues List opens the Issue drawer, which provides an in-depth view of the Monitor and its triggered issue. You can also navigate if possible to related entities like workload, node, pod, etc.
Details tab
Displays metadata about the issue, including:
Monitor Name: Name of the Monitor that triggered the issue, including a link to it.
Description: Explains what the Monitor tracks and why it was triggered.
Severity: Shows the assigned severity level (e.g., S3).
Events tab
Displays the Kubernetes events related to the selected issue within the timeframe selected in the Time Picker dropdown (upper right of the issue drawer).
Traces tab
When creating a Monitor using a traces query, the Traces tab will display the matching traces generated within the timeframe selected in the Time Picker dropdown (upper right of the issue drawer). Click on "View in Traces" to navigate to the Traces section with all relevant filters automatically applied.
Logs tab
When creating a monitor using a log query, the Logs tab will display the matching logs generated within the timeframe selected in the Time Picker dropdown (upper right of the issue drawer). Click on "View in Logs" to navigate to the Logs section with all relevant filters automatically applied.
Map tab
A focused visualization of the interactions between workloads related to the selected issue.
Remote Access & APIs
groundcover has various authentication key types for remotely interacting with our platform, whether to ingest observability data or to automate actions via our APIs:
API Keys- An API key in groundcover provides secure, programmatic access to the API on behalf of a service account. It inherits that account’s permissions and should be stored safely. This is also the key you need when working groundcover’s terraform provider. See:
- Ingestion Keys let sensors, integrations and browsers send observability data to your groundcover backend. These keys are the counterpart of API Keys, which are optimized for reading data or automating dashboards and monitors.
- A key used to connect to groundcover as a datasource, querying Clickhouse and VictoriaMetrics directly.
- Used to remotely configure create Grafana Alerts & Dashboards via Terraform.
\
Workflows
Workflows are YAML-based configurations that are executed whenever a monitor is triggered. They enable you to integrate with third-party systems, apply custom logic to format and transform data, and set different conditions to handle your monitoring alerts intelligently.
Workflow components
Triggers
Triggers apply filtering conditions that determine whether a specific workflow is executed. In groundcover, the trigger type is always set to "alert".
Example: This trigger ensures that only monitors fired with telemetry data from the Prod environment will actually execute the workflow. Note that the "env" attribute needs to be provided as a context label from the monitor:
Note: Workflows are "pull based" which means they will try to match monitors even when these monitors did not explicitly add a specific workflow. Therefore, the filters need to accurately define the condition to be used for a monitor.
Consts
Consts is a section where you can declare predefined attributes based on data provided with the monitor context. A set of functions is available to transform existing data and format it for propagation to third-party integrations. Consts simplify access to data that is needed in the actions section.
Example: The following example shows how to map a predefined set of severity values to the monitor severity as defined in groundcover - here, any potential severity in groundcover is translated into one of P1-P5 values.
The function keep.dictget gets a value from a map (dictionary) using a specific key. In case the key is not found - P3 will be the default severity:
Actions
Actions specify what happens when a workflow is triggered. Actions typically interface with external systems (like sending a Slack message). Actions can be an array of actions, they can be executed conditionally and include the integration in their config part as well as a payload block which is typically dependent on the exact integration used for the notification.
Actions include:
Provider part (provider:) - Configures the integration to be used
Payload part (with:) - Contains the data to submit to the integration based on its actual structure
Example: In this example you can see a typical Slack notification. Note that the actual integration is referenced through the 'providers' context attribute. The integration name is the exact string used to (in this case "groundcover-alerts").
Silences page
Manage and create silences to suppress Monitor notifications during maintenance or specific periods, reducing noise and focusing on critical issues.
Overview
The Silences page lists all Silences you and your team created for your Monitors. In this section, you can also create and manage your Silence rules, to suppress notifications and Issues noise, for a specified period of time. Silences are a great way to reduce alert fatigue, which can lead to missing important issues, and help focus on the most critical issues during specific operational scenarios such as scheduled maintenances.
Create a new Silence
Follow these simple steps to create a new Silence.
Section 1: Schedule
Specify the timeframe for the silence rule. Note that the starting point doesn't have to be now, and can also be any time in the future.
Below the From / Until boxes, you'll see a Silence summary, showing its approximate length (rounded down to full days) and starting date.
Section 2: Matchers
Define the criteria for Monitors or Issues to be silenced.
Click Add Matcher to specify match conditions (e.g., cluster, namespace, span_name).
Combine multiple matchers for more granular control.
Example: Silence all Monitors in the "demo" namespace.
Section 3 - Affected Active Issues
Preview the issues currently affected by the Silence rule, based on any defined Matchers. This list contains only actively firing Issues.
Tip: Us this preview to see the list of impacted issues and adjust your Matchers before finishing to create the Silence.
Section 4: Comment
Add notes or context for the Silence rule. These comments help you and other users understand the purpose of the rule.
API Keys
An API key in groundcover provides secure, programmatic access to the API on behalf of a service account. It inherits that account’s permissions and should be stored safely.
An API key in groundcover provides secure, programmatic access to the API on behalf of a service account. It inherits that account’s permissions and should be stored safely
Binding and Permissions
Each API key is tied to a specific service account. It inherits the permissions defined by that account’s RBAC policies. Optionally, the key can be limited to a subset of those policies for more granular access control. An API key can never exceed the permissions of its parent service account.
Creation and Storage
Only Admins can create or revoke API keys. To create an API key:
Navigate to the Settings page using the settings button located in the bottom left corner
Select "Access" from the sidebar menu
Click on the "API Keys" tab
When a key is created, its value is shown once—store it securely in a secret manager or encrypted environment variable. If lost, a new key must be issued.
Authentication and Usage
To use an API key, send it in the Authorization header as bearer token:
The key authenticates as the service account, and all API permissions are enforced accordingly.
API Key authentication will work using https://api.groundcover.com/ only.
Validity and Revocation
API keys do not expire automatically. Revoking a key immediately disables its access.
Scope of Use
API keys are valid only for requests to https://api.groundcover.com. They do not support data ingestion or Grafana integration—those require dedicated tokens.
API Keys vs Ingestion Keys
Security Best Practices
Store securely: Use secrets managers like AWS Secrets Manager or HashiCorp Vault. Never commit keys to source control.
Follow least privilege: Assign the minimal required policies to service accounts and API keys. Avoid defaulting to admin-level access.
Rotate regularly: Periodically generate new keys, update your systems, and revoke old ones to limit exposure.
Revoke stale keys: Remove keys that are no longer in use or suspected to be compromised.
Create a new Workflow
Creation
Creating new workflows is currently supported through the groundcover app in two ways from the Monitors menu:
1. "Create Notification Workflow" button - The quick way
This provides a guided approach to create a workflow. When creating a notification workflow, you will be asked to give your workflow a name, add filters, and select the specific integration to use.
Filter Rules By Labels - Add key-value attributes to ensure your workflow executes under specific conditions only - for example, env = prod only.
Delivery Destinations - Select one or more integrations to be used for notifications with this workflow.
Scope - When The Workflow Will Run - This setting allows you to limit this workflow execution only to monitors that explicitly select to route their triggers to this workflow, as opposed to "Handle all issues" that catches triggers from any monitor.
Once you create a workflow using this option, you can later edit the workflow to apply any configuration or logic by using the editor option (see next).
2. "Create Workflow" button
Clicking the button will open up a text editor where you can add your workflow definition in YAML format by applying any valid configuration, logic, and functionality.
Note: Make sure to create your integration prior to creating the workflow as it requires using an existing integration.
View
Upon successful workflow creation it will be active immediately, and a new workflow record will appear in the underlying table.
For each existing workflow, we can see the following fields:
Name: Your defined workflow name
Description: If you've added a description of the workflow
Creator: Workflow creator email
Editing
From the right side of each workflow record in the display, you can access the menu (three dots) and click "Edit Workflow". This will open the editor so you can modify the YAML to conform to the available functionality. See examples below.
Create a Grafana dashboard
The following guide explains how to build dashboards within the groundcover platform using our fully integrated Grafana interface. To learn how you can create dashboards using Grafana Terraform, follow this guide.
A dashboard is a great tool for visually tracking, analyzing, and displaying key performance metrics, which enable you to monitor the health of your infrastructure and applications.
Creating a new dashboard
1️⃣ Go to the Dashboards tab in the groundcover app, and click New and then New Dashboard.
2️⃣ Create your first panel by clicking Add a new panel.
3️⃣ In the New panel view, go to the Query tab.
4️⃣ Choose your data source by pressing the -- Grafana -- on the data source selector. You would see your the metrics collected from each of your clusters an a Prometheus data source called Prometheus@<cluster-name>
5️⃣ Create your first Query in the PromQL query interface.
Learn more about and to improve your skills.
For any help in creating your custom dashboard don't hesitate to .
Tips: \
Learn more about the supported metrics you can use to build dashboards in the section under and page.\
groundcover has a set of example dashboards in the
Real User Monitoring (RUM)
Monitor front-end applications and connect it to your backend — all inside your cloud.
This capability is only available to BYOC and on-prem deployments. Check out our for more information about subscription plans and the available deployment modes.
Capture real end-user experiences directly from their browsers and unify these insights with your backend observability data.
➡️ Check out our to your platform.
Get Logs Pipeline Configuration
Retrieve the current logs pipeline configuration.
Endpoint
GET/api/pipelines/logs/config
Login and Create a Workspace
Get started with groundcover
This is the first step to start with groundcover for all types of plans 🚀
Sign up to groundcover
Datasource API Keys
groundcover provides a robust user interface that allows you to view and analyze all your observability data from inside the platform. However, there may be cases in which you need to query the data from outside our platform using API communication.
Our proprietary eBPF sensor automatically captures granular observability data, which is stored via our integrations with two best-of-breed technologies. VictoriaMetrics for metrics storage, and ClickHouse for storage of logs, traces, and Kubernetes events.
Read more about our architecture .
Generate the API key
Update Logs Pipeline Configuration
Update the logs pipeline configuration.
Endpoint
POST/api/pipelines/logs/config
Getting-started Prompts
Once your MCP server is connected, you can dive right in.
Here are a few prompts to try. They work out of the box with agents like Cursor, Claude, or VS Code:
💡 Starting your request with “Use groundcover” is a helpful nudge - it pushes the agent toward MCP tools and context.
Basic Prompts to Try
Delete Workflow
Delete an existing workflow using workflow id
Endpoint
DELETE /api/workflows/{id}
Build alerts & dashboards with Grafana Terraform provider
Configure the Grafana Terraform provider
For instructions on how to generate a Grafana Service Account Token and use it in the Grafana Terraform provider, see: .
Run the following command in your CLI, and select tenant:
groundcover auth get-datasources-api-key
Querying ClickHouse
Example for querying ClickHouse database using POST HTTP Request:
Command parameters
X-ClickHouse-Key (header): API Key you retrieved from the groundcover CLI. Replace ${API_KEY} with your actual API key, or set API_KEY as env parameter.
SELECT count() FROM traces WHERE start_timestamp > now() - interval '15 minutes' (data): The SQL query to execute. This query counts the number of traces where the start_timestamp is within the last 15 minutes.
Learn more about the ClickHouse query language here.
Querying VictoriaMetrics
Example for querying the VictoriaMetrics database using the query_range API:
Command parameters
apikey (header): API Key you retrieved from the groundcover CLI. Replace ${API_KEY} with your actual API key, or set API_KEY as env parameter.
query (data): The promql query to execute. In this case, it calculates the sum of the rate of groundcover_resource_total_counter with the type set to http.
start (data): The start timestamp for the query range in Unix time (seconds since epoch). Example: 1715760000.
end (data): The end timestamp for the query range in Unix time (seconds since epoch). Example: 1715763600.
This endpoint requires API Key authentication via the Authorization header.
Headers
Examples
Basic Request
Update logs pipeline configuration with test pattern:
Response Example
🚨 CRITICAL WARNING: This endpoint COMPLETELY REPLACES the entire pipeline configuration and WILL DELETE ALL EXISTING RULES. Always backup your current configuration first by calling the GET endpoint.
Backup Current Configuration First
ALWAYS get your current configuration before making changes:
Verify Configuration Update
After updating the configuration, verify the patterns were added:
This should return your updated configuration including the new test patterns.
Related Documentation
For detailed information about configuring and writing OTTL transformations, see:
folder which can get you started.
These dashboard are read-only but you can see the PromQL query behind each panel by right-clicking the panel and then
Real User Monitoring (RUM) extends groundcover’s observability platform to the client side, providing visibility into actual user interactions and front-end performance. It tracks key aspects of your web application as experienced by real users, then correlates them with backend metrics, logs, and traces for a full-stack view of your system.
Understand user experience - capture every interaction, page load, and performance metric from the end-user perspective to pinpoint front-end issues in real time.
Resolve issues faster - seamlessly tie front-end events to backend traces and logs in one platform, enabling end-to-end troubleshooting of user journeys.
Privacy first - groundcover’s Bring Your Own Cloud (BYOC) model ensures all RUM data stays in your own cloud environment. Sensitive user data never leaves your infrastructure, ensuring privacy and compliance without sacrificing insight.
Collection
groundcover RUM collects a wide range of data from users’ browsers through a lightweight JavaScript SDK. Once integrated into your web application, the SDK automatically gathers and sends the following telemetry from each user session to the groundcover platform:
Network requests: Every HTTP request initiated by the browser (such as API calls) is captured as a trace. Each client-side request can be linked with its corresponding server-side trace, giving you a complete picture of the request from the user’s click to the backend response.
Front-end logs: Client-side log messages (e.g., console.log outputs, warnings, and errors) are collected and forwarded to groundcover’s log management. This ensures that browser logs are stored alongside your application’s server logs for unified analysis.
Exceptions: Uncaught JavaScript exceptions and errors are automatically captured with full stack traces and contextual data (browser type, URL, etc.). These front-end errors become part groundcover monitors, letting you quickly identify and debug issues in the user’s environment.
Performance metrics (Core Web Vitals): Key performance indicators like page load time and Core Web Vitals like Largest Contentful Paint, Interaction to Next Paint, and Cumulative Layout Shift are measured for each page view. groundcover RUM records these metrics to help you track real-world performance and detect slowdowns affecting users.
User interactions: RUM tracks user interactions such as clicks, keydown, and navigation events. By recording which elements users interact with and when, groundcover helps you reconstruct user flows and understand the sequence of actions leading up to any issue or performance problem.
Custom events: You can instrument your application to send custom events via the RUM SDK. This allows you to capture domain-specific actions or business events (for example, a checkout completion or a specific UI gesture) with associated metadata, providing deeper insight into user behavior beyond automatic captures.
All collected data is streamed securely to your groundcover deployment. Because groundcover runs in your environment, RUM data (including potentially sensitive details from user sessions) is stored in the observability backend within your own cloud. From there, it is aggregated and indexed just like other telemetry, ready to be searched and analyzed in the groundcover UI.
Full-Stack Visibility
One of the core advantages of groundcover RUM is its native integration with backend observability data. Every front-end trace, log, or event captured via RUM is contextualized alongside server-side data:
Trace correlation: Client-side traces (from browser network requests) are automatically correlated with server-side traces captured by groundcover’s eBPF-based instrumentation. This means when a user triggers an API call, you can see the complete distributed trace that spans the browser and the backend services, all in one view.
Unified logging: Front-end log entries and error reports are ingested into the same backend as your server-side logs. In the groundcover Log Explorer, you can filter and search across logs from both client and server, using common fields (like timestamp, session ID, or trace ID) to connect events.
End-to-end troubleshooting: With full-stack data in one platform, you can pivot easily between a user’s session replay, the front-end events, and the backend metrics/traces involved. This end-to-end context significantly reduces the time to isolate whether an issue originated in the frontend (browser/UI) or the backend (services/infrastructure), helping teams pinpoint problems faster across the entire stack.
By bridging the gap between the user’s browser and your cloud infrastructure, groundcover’s RUM capability ensures that no part of the user journey is invisible to your monitoring. This holistic view is critical for optimizing user experience and rapidly resolving issues that span multiple layers of your application.
Sessions Explorer
Once RUM data is collected, it becomes available in the groundcover platform via the Sessions Explorer — a dedicated view for inspecting and troubleshooting user sessions. The Sessions Explorer allows you to navigate through user journeys and understand how your users experience your application.
Clicking on any session opens the Session View, where you can inspect a full timeline of the user’s experience. This view shows every key event captured during the session - including clicks, navigations, network requests, logs, custom events, and errors.
Each event is displayed in sequence with full context like timestamps, URLs, and stack traces. The Session View helps you understand exactly what the user did and what the system reported at each step, making it easier to trace issues and user flows.
The first thing you need to do to start using groundcover is sign up using your email address (no credit card required for the free tier account). Signing up is only possible using a computer and will not be possible using a mobile phone or tablet. It is highly recommended you use your corporate email address, as it will make it easier to use other features such as inviting your colleagues to your workspace. However, signing up using Gmail, Outlook or any other public domains is also possible.
Workspace Selection
When signing in to groundcover for the first time, the platform automatically detects your organization based on the domain you used to sign in. If your organization already has existing workspaces available, the workspace selection screen will be displayed, where you can choose which of the existing workspaces you would like to join, or if you want to create a new workspace.
Available workspaces will be displayed only if either of the following applies:
You have been invited to join existing workspaces and haven't joined them yet
Someone has previously created a workspace that has auto-join enabled for the email domain that you used to sign in (applicable for corporate email domains only)
To join an existing workspace:
Click the Join button next to the desired workspace
You will be added as a user to that workspace with the user privileges that were assigned by default or those that were assigned to you specifically when the invite was sent.
You will automatically be redirected to that workspace.
To create a new workspace:
You will only see the option to create a new workspace if you are the first person from your organization to join groundcover.
Click the Create a new workspace button
Specify a workspace name
Choose whether to enable auto-join (those settings can be changed later)
Click continue
Workspace Auto-joining
Workspace owners and admins can allow teammates that log in with the same email domain as them to join the Workspace they created automatically, without an admin approval. This capability is called "Auto-join". It is disabled by default, but can be switched on during the workspace set up process, or any time in the workspace settings.
If you logged in with a public email domain (Gmail, Yahoo, Proton, etc.) and are creating a new Workspace, you will not be able to switch on Auto-join for that Workspace.
MCP supports complex, multi-step flows, but starting simple is the best way to ramp up.
Pull Logs
Prompt:
Expected behavior:
The agent should call query_logs and show recent logs for that workload.
Get K8s Resource Specs
Prompt:
Expected behavior:
The agent should call get_k8s_object_yaml and return the YAML or a summary of it.
Find Slow Workloads
Prompt:
Expected behavior:
The agent should call get_workloads and return the relevant workloads with their P95 latency.
Investigate Issues
When something breaks, your agent can help investigate and make sense of it.
Paste an Issue Link
Prompt:
Expected behavior:
The agent should use query_monitors_issues, pull issue details, and kick off a relevant investigation using logs, traces, and metadata.
Investigate Multiple Issues
Prompt:
Expected behavior:
The agent should use query_monitors_issues to pull all related issues and start going through them one by one.
Automate Coding & Debugging
groundcover’s MCP can also be your coding sidekick.
Instead of digging through tests and logs manually, deploy your changes and let the agent take over.
Iterate Over Test Results
Prompt:
Expected behavior:
The agent should update the code with log statements, deploy it, and use query_logs to trace and debug.
Deploy & Verify
Prompt:
Expected behavior:
The agent should assist with deployment, then check for issues, error logs, and traces via groundcover.
Dashboard provisioning example
Create a directory for the terraform assets
Create a main.tf file within the directory that contains the terraform provider configuration mentioned in step 2
Create the following dashboards.tf file, this example declares a new Golden Signals folder, and within it a Workload Golden Signals dashboard that will be created
add the workloadgoldensignals.json file to the directory as well
Run terraform init to initialize terraform context
Run terraform plan , you should see a long output describing the assets that are going to be created last line should state
Plan: 2 to add, 0 to change, 0 to destroy.
Run terraform apply to execute the changes, you should now see a new folder in your grafana dashboards screen with the newly created dashboard
Run terraform destroy to revert the changes
Here is a short video to demonstrate the process
You can read more about what you can achieve with the Grafana Terraform provider in the official docs
Traces are a powerful observability pillar, providing granular insights into microservice interactions. Traditionally, they were hard to implement, requiring coordination of multiple teams and constant code changes, making this critical aspect very challenging to maintain.
groundcover's eBPF sensor disrupts the famous tradeoff, empowering developers to gain full visibility into their applications, effortlessly and without any code changes.
The platform supports two kinds of traces:
eBPF traces
These traces are automatically generated for every service in your stack. They are available out-of-the-box and within seconds of installation. These traces always include critical information such as:
All services that took part in the interaction (both client and server)
Accessed resource
Full payloads, including:
3rd-party traces
These can be ingested into the platform, allowing to leverage already existing instrumentation to create a single pane of glass for all of your traces.
Traces are stored in groundcover's ClickHouse deployment, ensuring top notch performance on every scale.
For more details about ingesting 3rd party traces, see the .
Sampling
groundcover further disrupts the customary traces experience by reinventing the concept of sampling. This innovation differs between the different types of traces:
eBPF traces
These are generated by using 100% of the data, always processing every request being made, on every scale. However, the groundcover platform utilizes smart sampling to only store a fraction of the traces, while still generating an accurate picture. In general, sampling is performed according to these rules:
Requests with unusually high or low latencies, measured per resource
Requests which returned an error response (e.g 500 status code for HTTP)
"Normal" requests which form the baseline for each resource
Lastly, is utilized to make the sampling decisions on the node itself, without having to send or save any redundant traces.
Certain aspects of our sampling algorithm are configurable - read more .
3rd-partytraces
Various mechanisms control the sampling performed over 3rd party traces. Read more here:
When integrating 3rd-party traces, it is often wise to configure some sampling mechanism according to the specific use case.
Additional Context
Each trace is enriched with additional information to give as much context as possible for the service which generated the trace. This includes:
Container information - image, environment variables, pod name
Logs generated by the service around the time of the trace
of the resource around the time of the trace
Distributed Tracing
One of the advantages of ingesting 3rd-party traces is the ability to leverage their distributed tracing feature. groundcover natively displays the full trace for ingested traces in the Traces page.
Trace Attributes
Trace Attributes enable advanced filtering and search capabilities. groundcover support attributes across all trace types. This encompasses a diverse range of protocols such as HTTP, MongoDB, PostgreSQL, and others, as well as varied sources including eBPF or manual instrumentations (for example - OpenTelemetry).
groundcover enriches your original traces and generates meaningful metadata as key-value pairs. This metadata includes critical information, such as protocol type, http.path, db.statement, and similar attributes, aligning with OTel conventions. Furthermore, groundcover seamlessly incorporates this metadata from spans received through supported manual instrumentations. For an in-depth understanding of attributes in OTel, please refer to (external link to OpenTelemtry website).
Each attribute can be effortlessly integrated into your filters and search queries. You can add them directly from the trace side-panel with a simple click or input them manually into the search bar.
Example: If you want to filter all HTTP traces that contain the path "/products". The query would be formatted as: @http.path:"/products". For a comprehensive guide on the query syntax, see Syntax table below.
Trace Tags
Trace Tags enable advanced filtering and search capabilities. groundcover support tags across all trace types. This encompasses a diverse range of protocols such as HTTP, MongoDB, PostgreSQL, and others, as well as varied sources including eBPF or manual instrumentations (for example - OpenTelemetry).
Tags are powerful metadata components, structured as key-value pairs. They offer insightful information about the resource generating the span, like: container.image.name ,host.name and more.
Tags include metadata enriched by the our sensor and additional metadata if provided by manual instrumentations (such as OpenTelemetry traces) . Utilizing these Tags enhances understanding and context of your traces, allowing for more comprehensive analysis and easier filtering by the relevant information.
Each tag can be effortlessly integrated into your filters and search queries. You can add them directly from the trace side-panel with a simple click or input them manually into the search bar.
Example: If you want to filter all traces from mysql containers - The query would be formatted as: container.image.name:mysql. For a comprehensive guide on the query syntax, see Syntax table below.
Search and filter
The Trace Explorer integrates dynamic filters and a versatile search functionality, to enhance your trace data analysis. You can filter out traces using specific criteria, including trace-status, workload, namespace and more, as well as limit your search to a specific time range.
Traces Pipelines
groundcover natively supports setting up log pipelines using This allow for full flexibility in the processing and manipulation of traces being collected - parsing additional patterns by regex, renaming attributes, and many more.
Controlling retention
groundcover allows full control over the retention of your traces. to learn more.
Custom Configuration
Tracing can be customized in several ways:
Kubernetes requirements
Kubernetes version
groundcover supports any K8s version from v1.21.
groundcover may work on many other K8s flavors, but we might just didn't get a chance to test it yet. Can't find yours in the list?
Kubernetes distributions
K8s distribution
Status
Comments
Kubernetes RBAC permissions
For the installation to complete successfully, permissions to deploy the following objects are required:
StatefulSet
Deployment
DaemonSet (With privileged containers for loading our )
To learn more about groundcover's architecture and components visit our
Outgoing traffic
groundcover's portal pod sends HTTP requests to the cloud platform app.groundcover.com on port 443.
This unique keeps the the data inside the cluster and fetches it on-demand keeping the data encrypted all the way without the need to open the cluster for incoming traffic via ingresses.
Slack App for Channel Routing
groundcover supports sending notifications to Slack using a Slack App with bot tokens instead of static webhooks. This method allows dynamic routing of alerts to any channel by including the channel ID in the payload. In addition to routing, messages can be enriched with formatting, blocks, and mentions — for example including <@user_id> in the payload to directly notify specific team members. This provides a flexible and powerful alternative to fixed incoming webhooks for alerting.
Use the following workflow as an example. You can later enrich your workflow with additional functionality.
Here are a few tips for using the example workflow:
In the consts section, the channels attribute defines the mapping between Slack channels and their IDs. Use a clear, readable label to identify each channel (for example, the channel’s actual name in Slack), and map it to the corresponding channel ID.
To locate a channel ID, open the channel in Slack, click the channel name at the top, and scroll to the About section. The channel ID is shown at the bottom of this section.
The channel name should be included in the monitor’s Metadata Labels, or you can fall back to a default. See the channel_id attribute in the workflow example.\
Finally, replace the integration name in {{ providers.slack-routing-webhook }} with the actual name of the Webhook integration you created.
MS Teams
To integrate groundcover with MS Teams, follow the steps below. Note that you’ll need at least a Business subscription of MS Teams to be able to create workflows.
Create a webhook workflow for your dedicated Teams channel
Go to Relevant Team -> Specific Channel -> "Workflows", and create a webhook workflow
The webhook workflow is associated a URL which is used to trigger the MS Teams integration on groundcover - make sure to copy this URL
Set Up the Webhook in groundcover
Head out to the integrations section: Settings -> Integrations, to create a new
Start by giving your Webhook integration a name. This name will be used below in the provider block sample .
Set the Webhook URL
Create a Workflow
Go to Monitors --> Workflows --> Create Workflow, and paste the YAML configuration provided below.
Configure the provider Blocks (There are two of them)
In the provider block, replace {{ providers.your-teams-integration-name }} with your actual Webhook integration name (the one you created in step 3)
For example, if you named your integration test-ms-teams, the config reference would be: {{ providers.test-ms-teams }}
The following example shows a pre-configured MS Teams workflow template. You can easily modify workflows to support different formats based on the MS Teams workflow schema.
Sample code for your groundcover workflow:
Search & Filter
Search and filter
To help you slice and dice your data, you can use our dynamic filters (left panel) and/or our powerful filter bar, which supports key:value pairs, as well as free text search. The Query Builder works in tandem with our filters.
To further focus your results, you can also restrict the results to specific time windows using the time picker on the upper right of the screen.
Query Builder
The Query Builder is the default search option wherever search is available. Supporting advanced autocomplete of keys, values, and our discovery mode that across values in your data to teach users the data model.
The following syntaxes are available for you to use in Query Builder:
Syntax
Description
Examples
Sections
How to use filters
Filters are very easy to add and remove, using the filters menu on the left bar. You can combine filters with the Query Builder, and filters applied using the left menu will also be added to the Query Builder in text format.
Select / deselect a single filter - click on the checkbox on the left of the filter. (You can also deselect a filter by clicking the 'x' next to the text format of the filter on the search bar).
Deselect all but one filter (within a filter category, such as 'Level' or 'Format') - hover over the filter you want to leave on, then click on "ONLY".
You can switch between filters you want to leave on by hovering on another filter and clicking "ONLY" again.
incident.io
To integrate groundcover with incident.io, follow the steps below. Note that you’ll need a Pro incident.io account to view your incoming alerts.
Generate an Alerts configuration for groundcover
Log in to your incident.io account. Go to "On-call" -> "Alerts" -> "Configure" and add a new source.
On the "Create Alert Source" screen the answer to the question "Where do your alerts come from?" should be "HTTP". Select this source and give it a unique name. Hit "continue".
incident.io will create your configuration now from which you will need to copy the following items for the Webhook integration\
Set Up the Webhook in groundcover
Head out to the integrations section: Settings -> Integrations, to create a new
Start by giving your Webhook integration a name. This name will be used below in the provider block sample .
Create a Workflow
Go to Monitors --> Workflows --> Create Workflow, and paste the YAML configuration provided below.
Note: The body section is a dictionary of keys that will be sent as a JSON payload to the incident.io platform
Configure the provider Block
In the provider block, replace {{ providers.your-incident-io-integration-name }} with your actual Webhook integration name (the one you created in step 4)
For example, if you named your integration test-incidentio, the config reference would be: {{ providers.test-incidentio }}
Required Parameters for Creating an alert
When triggering an alert, the following keys are required:
title - Alert title that can be pulled from groundcover as seen in the example
status - One of "firing" or "resolved" that can also be pulled from groundcover as the example shows.
You can include additional parameters for richer context (optional):
description
deduplication_key - unique attribute used to group identical alerts, groundcover provides this through the fingerprint attribute
The attributes shown in the yaml block of the metadata section below are an example only! Alert labels can only be attributes used in the group by section of the actual monitor
Example code for your groundcover workflow:
Log Patterns
Log Patterns help you cut through log noise by grouping similar logs based on structure. Instead of digging through thousands of raw lines, you get a clean, high-level view of what’s actually going on
Overview
Log Patterns in groundcover help you make sense of massive log volumes by grouping logs with similar structure. Instead of showing every log line, the platform automatically extracts the static skeleton and replace dynamic values like timestamps, user IDs, or error codes with smart tokens.
This lets you:
Cut through the noise
Spot recurring behaviors
Investigate anomalies faster
How It Works
groundcover automatically detects variable parts of each log line and replace them with placeholders to surface the repeating structure.
Placeholder
Description
Example
Requirements
Log Patterns are collected directly on the sensor.
Example
Raw log:
Patterned:
Viewing Patterns
Go to the Logs section.
Switch from Records to Patterns using the toggle at the top.
Patterns are grouped and sorted by frequency. You’ll see:
Value Distribution
You can hover over any tag in a pattern to preview the distribution of values for that specific token. This feature provides a breakdown of sample values and their approximate frequency, based on sampled log data.
This is especially useful when investigating common IPs, error codes, user identifiers, or other dynamic fields, helping you understand which values dominate or stand out without drilling into individual logs.
For example, hovering over an <IP4> token will show a tooltip listing the most common IP addresses and their respective counts and percentages.
Investigating Patterns
Click a pattern: Filters the Logs view to only show matching entries.
Use filters: Narrow things down by workload, level, format, or custom fields.
Suppress patterns: Hide noisy templates like health checks to stay focused on what matters.
Delete Ingestion Key
Delete an existing ingestion key. This operation permanently removes the key and cannot be undone.
Endpoint
DELETE/api/rbac/ingestion-keys/delete
Authentication
This endpoint requires API Key authentication via the Authorization header.
Headers
Request Body
Parameter
Type
Required
Description
Examples
Delete by Name
Response
The endpoint returns an empty response body with HTTP 200 status code when the key is deleted successfully.
Important Warnings
🚨 PERMANENT DELETION: This operation permanently deletes the ingestion key and cannot be undone.
⚠️ Immediate Impact: Any services using this key will:
Receive 403 PERMISSION_DENIED errors
Stop sending data to groundcover immediately
Lose access to remote configuration (for sensor keys)
Verification
To verify the key was deleted, use the List Ingestion Keys endpoint:
This should return an empty array [] if the key was successfully deleted.
Safe Deletion Workflow
Identify the key to delete using the List endpoint
Update integrations to use a different key first
Test integrations work with the new key
Best Practices
Always have a replacement key ready before deleting production keys
Test your rollover plan in staging environments first
Update all services using the key before deletion
Related Documentation
For comprehensive information about ingestion keys management, see:
Issues
Quickly understand what requires your attention and drive your investigations
The issues page is a useful place to start a troubleshooting or investigation flow from. It gathers together all active issues found in your Kubernetes environment.
Issue Types
HTTP/gRPC Failures
Capturing failed HTTP calls with Response Status Codes of:
5XX — Internal Server Error
429 — Too Many Requests
MySQL/ PostgreSQL Failures
Capturing failed SQL statement executions with Response Errors Codes such as:
1146 — No Such Table
Redis Failures
Capturing any reported Error by the Redis serialization protocol (RESP), such as:
ERR unknown command\
Container Restarts
Capturing all container restart events across the cluster, with Exit Codes such as:
0 — Completed
Deployment Failures
Capturing events such as:
MinimumReplicasUnavailable — Deployment does not have minimum availabiltiy
Issue Aggregation
Issues are auto-detected and aggregated - representing many identical repeating incidents.
Aggregation help cutting through the noise quickly and reach insights like when a new type of issue started to appear, and when it was last seen.
Issues are grouped by:
Type (HTTP, gRPC, Container Restart, etc..)
Status Code / Error Code (e.g HTTP 500, gRPC 13)
Workload name
The smart aggregation mechanism will also identify query parameters, remove them, and group the stripped queries / API URIs into patterns. This allows users to easily identify and isolate the root cause of a problem.
Troubleshooting with Issues
Each issue is assigned a velocity graph showing it's behavior over time (like when it was first seen) and a live counter of its number of incidents.
By clicking on an issue, users can access the specific traces captured around the relevant issue. Each trace is related to the exact resource that was used (e.g. raw API URI, or SQL query), it's latency and Status Code / Error Code.
Further clicking on a selected captured trace allows the user to investigate the root cause of the issue with the entire payload (body and headers) of the request and response, the information around the participating container, the application logs around incident's time and the full context of the metrics around the incident.
Migrating from Issues to Monitors Issues Page
The legacy Issues page is being deprecated in favor of a fully customizable, monitor-based experience that gives you more control over what constitutes an issue in your environment.
While the new page introduces powerful capabilities, no core functionality is being removed, the key change is that the old auto-created issue rules will no longer be automatically generated. Instead, you’ll define your own monitors, or choose from a rich catalog of prebuilt ones.
All the existing issues in the legacy page can be easily added to the monitors via the Monitors Catalog's "Started Pack". See Getting Started below for more info.
Why migrate to the new Issues experience?
The new Issues page is built on top of the Monitors engine, enabling deeper customization and automation:
Define what qualifies as an issue
Use filters in monitor definitions to include or exclude workloads, namespaces, HTTP status codes, clusters, and more tailor it to your context.
Silence issues with precision
Silence issues based on any label, such as status_code, cluster, or workload
What’s Changing?
Aspect
Legacy Issues Page
New Issues Page
Getting Started
All the built-in rules you’re used to are already available in the , you can add them all with a single click.
Adding the monitors in the "Started Pack" will match all the existing issues in the legacy page.
Only users with Editor/Admin roles can create monitors
Learn More
Using groundcover as Prometheus/Clickhouse database in a Self-hosted Grafana
Exposing Data Sources for BYOC installations
groundcover BYOC exposes Prometheus data sources for programmatic access via API, and integration with customer owned Grafana instances.
Different steps are required for On-Prem deployments, contact us for additional info.
Requirements
API Key
The groundcover tenant API KEY is required for configuring the data source connection.
You can obtain your API key from the in the groundcover console.
For this example we will use the key API-KEY-VALUE
Setup
Prometheus
Grafana Data Source Configuration
Configure Grafana prometheus Data Source by following these steps logged in as Grafana Admin.
Connections > Data Sources > + Add new data source
Pick Prometheus
Name: groundcover-prometheus
"Successfully queried the Prometheus API" means the integration was configured correctly.
Legacy Configuration
The following configurations are deprecated but may still be in use in older setups.
Datasources API Key
The legacy datasources API key can be obtained by running: groundcover auth get-datasources-api-key
ClickHouse
ClickHouse datasource integration is deprecated and no longer supported for new installations.
Configure Grafana ClickHouse Data Source by following these steps logged in as Grafana Admin.
Connections > Data Sources > + Add new data source
Pick ClickHouse
Name: groundcover-clickhouse
"Data source is working" means the integration was configured correctly.
Alert Structure
Fields description in the alert you can use in your workflows
Structure
Field Name
Description
Example
labels
Map of key:values derived from monitor definition.
{
"workload": "frontend",
"namespace": "prod"
}
Usage
When crafting your workflows you can use any of the fields above using templating in any workflow field. Encapsulate you fields using double opening and closing curly brackets.
Examples
Using Label Values
You can access label values by alert.labels.*
Migrate from Datadog
Complete guide for migrating your Datadog setup to groundcover.
Prerequisites
Access required:
Admin role in groundcover
Connect RUM
This capability is only available to organizations subscribed to our .
groundcover’s Real User Monitoring (RUM) SDK allows you to capture front-end performance data, user events, and errors from your web applications.
Start capturing RUM data by installing the in your web app.
This guide will walk you through installing the SDK, initializing it, identifying users, sending custom events, capturing exceptions, and configuring optional settings.
Configure groundcover's MCP Server
Set up your agent to talk to groundcover’s MCP server. Use OAuth for a quick login, or an API key for service accounts.
The MCP server supports two methods:
(Recommended for IDEs)
Kernel requirements for eBPF sensor
Intro
groundcover’s eBPF sensor uses state-of-the-art kernel features to provide full coverage at low overhead. In order to do so it requires certain kernel features which are listed below.
Managing Dashboards with Terraform
Create & manage dashboards with Terraform
Use Terraform to create, update, delete, and list groundcover dashboards as code. Managing dashboards with infrastructure‑as‑code (IaC) lets you version changes, review them in pull requests, promote the same definitions across environments, and detect drift between what’s applied and what’s running in your account.
Ingestion Keys
Secure, write‑focused credentials for streaming data into groundcover
Ingestion Keys let sensors, integrations and browsers send observability data to your groundcover backend.
They are the counterpart of API Keys, which are optimized for reading data or automating dashboards and monitors.
Key types
Connect Linux hosts
Linux hosts sensor
Note: Linux host sensor is only available to BYOC and on-prem deployments. Check out our for more information about subscription plans and the available deployment modes.
Supported Environments
Create Workflow
Creates a new workflow for alert handling and notifications. Workflows define how alerts are processed and routed to various integrations like Slack, PagerDuty, webhooks, etc.
Endpoint
POST /api/workflows/create
List Namespaces
Retrieve a list of Kubernetes namespaces within a specified time range.
Endpoint
Authentication
This endpoint requires API Key authentication via the Authorization header.
Use groundcover to get 5 logs from the workload news-service from the past 15 minutes.
Use groundcover to get the spec of the chat-app deployment.
Use groundcover to show the top 5 workloads by P95 latency.
I got an alert for this critical groundcover issue. Can you investigate it?
https://app.groundcover.com/monitors/issues?...
I got multiple alerts in the staging-env namespace. Can you help me look into them using groundcover?
Use groundcover to debug this code. For each test, print relevant logs with test_id, and dive into any error logs.
Please deploy this service and verify everything works using groundcover.
mkdir groundcover-tf-example && cd groundcover-tf-example
Exclude:
Specify terms or filters to omit from your search; applies to each distinct search.
-key:value-term-"search term"
Logs
Traces
K8s Events
API Catalog
Issues
*:value
Search all attributes:
Search any attribute for a value, you can use double quotes for exact match and wildcards.
*:error*:"POST /api/search"*:erro*
Logs
Traces
Issues
To turn all other filters in that filter category back on, hover over the filter again and click "ALL".
Clear all filters within a filters category - click on the funnel icon next to the category name.
Clear all filters currently applied - click on the funnel icon next to the number of results.
key:value
Search attributes:
Both groundcover built-ins custom attributes.
Use * for wildcard search.
Note: Multiple filters for the same key act as 'OR' conditions, whereas multiple filters for different keys act as 'AND' conditions.
namespace:prod-usnamespace:prod-*
Logs
Traces
K8s Events
API Catalog
Issues
term
Free text:
Search for single-word terms.
Tip: Expand your search results by using wildcards.
ExceptionDivisionBy*
Logs
"term"
Phrase Search (case-insensitive):
Enclose terms within double quotes to find results containing the exact phrase.
Note: Using double quotes does not work with * wildcards.
"search term"
Logs
, to reduce noise and keep focus.
Clean, scoped issue view
Only see issues relevant to your environment, based on your configured monitors and silencing rules, no clutter.
Get alerted on new issues
Trigger alerts through your preferred integrations (Slack, PagerDuty, Webhooks, etc.) when a new issue is detected.
Define custom issues using all your data
Build monitors using metrics, traces, logs, and events, and correlate them to uncover complex problems.
Manage everything as code
Use Terraform to manage monitors and issues at scale, ensuring consistency and auditability.
Set the Webhook URL to the url you copied from field (1)
Keep the HTTP method as POST
Under headers add Authorization, and paste the "Bearer <token>" copied from field (2).
metadata - Any additional metadata that you've configured within your monitor in groundcover. Note that these set should actively reflect your monitor definition in groundcover
A dedicated Ingestion Key of type RUM (Settings -> Access -> Ingestion Keys)
dsn
Your public groundcover endpoint in the format of https://example.platform.grcv.io , where example.platform.grcv.io
is your ingress.site installation value.
cluster
Identifier for your cluster; helps filter RUM data by specific cluster.
environment
Environment label (e.g., production, staging) used for filtering data.
appId
Custom application identifier set by you; useful for filtering and segmenting data on a single application level later.
Advanced Configuration
You can customize SDK behavior (event sampling, data masking, enabled events). The following properties are customizable:
You can pass the values by calling the init function:
The request body should contain raw YAML defining the workflow configuration. The YAML structure should include:
id: Unique identifier for the workflow
description: Human-readable description
triggers: Array of trigger conditions
actions: Array of actions to perform when triggered
name: Display name for the workflow
consts (optional): Constants and helper variables
Example Request
Response
Workflow YAML Structure
Basic Structure
Choosing Integration Providers
To route alerts to a specific integration (Slack, PagerDuty, webhook, etc.), use the config field in the provider section to reference your configured integration by name.
Example: Slack Integration
Provider Configuration
config: '{{ providers.integration-name }}' - References a specific integration you've configured in groundcover
type - Specifies the integration type (slack, webhook, pagerduty, opsgenie)
Replace integration-name with your actual integration name.
The integration name must match the name of an integration you've previously configured in your groundcover workspace.
Tip: Migrate all data sources first. This prevents missing data issues when monitors go live.
Migrate monitors
Once data sources are ready, migrate your monitors.
Monitor status indicators
✓ Supported: Fully compatible. Migrate as-is.
⚠ Partial: Migrates with warnings. Review before installing.
✗ Unsupported: Requires manual attention.
Review warnings
For monitors with warnings, click View Warnings:
See what adjustments were made
Understand query translations
Get recommendations for post-migration verification
Warnings don't block migration — they inform you of changes so you can verify behavior.
Migrate monitors
Single monitor:
Preview the monitor
Click Migrate
Monitor installs immediately
Bulk migrate:
Select multiple monitors using checkboxes
Click Migrate Selected
All install in parallel
Migrated monitors appear instantly in Monitors → Monitor List.
Migrate dashboards
Dashboards preserve:
Layout and widget positions
Query logic and filters
Time ranges and visualization settings
Colors and formatting
Check out the dashboard preview to confirm the migration worked and that all your assets came through successfully.
Migrate dashboards
Click Migrate to install. Dashboards appear under Dashboards immediately.
Tip: Migrate critical dashboards first. Verify queries return expected data before bulk migrating.
OAuth (recommended)
OAuth is the default if your agent supports it.
To get started, add the config below to your MCP client.
Once you run it, your browser will open and prompt you to log in with your groundcover credentials.
🧘 Pro tip: You can copy a ready-to-go command from the UI.
Go to the sidebar → Click your profile picture → "Connect to our MCP"
The tenant's UUID and your time zone will be auto-filled.
Please make sure to have node installed (to use npx)
Configuration Example
Cursor
Claude Code
API Key
If your agent doesn’t support OAuth, or if you want to connect a service account, this is the way to go.
Prerequisites
Service‑account API key – create one or use an existing API Key. Learn more at groundcover API keys.
Your local time zone (IANA format, for example America/New_York or Asia/Jerusalem). See how to find it below.
Configuration Example
Parameters
AUTH_HEADER – your groundcover API key.
TIMEZONE – your time zone in IANA format.
Have a Multi‑backend setup?
If you're using a multi-backend setup (OAuth or API Key), just add the following header to the args list:
First, grab your backend ID (it’s basically the name):
Open Data Explorer in groundcover.
Click the Backend picker (top‑right) and copy the backend’s name.
How to find your time zone
OS
command
macOS
sudo systemsetup -gettimezone
Linux
timedatectl grep "Time zone"
Windows PowerShell
Get-TimeZone
Client‑specific Guides
Depending on your client, you can usually set up the MCP server through the UI - or just ask the client to add it for you. Here are quick links for common tools:
groundcover may work on many other linux kernels, but we might just didn't get a chance to test it yet. Can't find yours in the list? let us know over Slack.
Kernel Version
Version v5.3 or higher (anything since 2020).
Linux Distributions
Name
Supported Versions
Debian
11+
RedHat Enterprise Linux
8.2+
Ubuntu
20.10+
CentOS
7.3+
Fedora
31+
BottlerocketOS
1.10+
Permissions
Loading eBPF code requires running privileged containers. While this might seem unusual, there's nothing to worry about - eBPF is safe by design!
CO:RE support
Our sensor uses eBPF’s CO:RE feature in order to support the vast variety of linux kernels and distributions detailed above. This feature requires the kernel to be compiled with BTF information (enabled using the CONFIG_BTF_ENABLE=Y kernel compilation flag). This is the case for most common distributions nowadays.
You can check if your kernel has CO:RE support by manually looking for the BTF file:
If the file exists, congratulations! Your kernel supported CO:RE.
What happens if my kernel is not supported?
If your system does not fit into any of the above - unfortunately, our eBPF sensor will not be able to run on your environment. However, this does not mean groundcover won’t collect any data. You will still be able to inspect your k8s environment, see all collected logs and use integrations with outer data sources.
Prerequisites
A groundcover account with permissions to create/edit Dashboards
A Terraform environment (groundcover provider >v1.1.1)
The groundcover Terraform provider configured with your API credentials
You can export a Dashboard into as a Terraform resource:
Open the Dashboard.
Click Actions → Export.
Download or copy the Terraform tab’s content and paste it into your .tf file (see placeholder above).
1.3) Add the dashboard resource to your Terraform configuration
The example below is a placeholder, paste your generated snippet or hand‑write your own.
After saving this file as main.tf along with the provider details, type:
2) Managing existing provisioned Dashboard
2.1) "Provisioned" badge for IaC‑managed Dashboards
Dashboards added via Terraform are marked as Provisioned in the UI so you can quickly distinguish IaC‑managed Dashboards from manually created ones, both from the Dashboard List and inside the Dashboard itself.
2.2) Edit behavior for Provisioned Dashboards
Provisioned Dashboards are read‑only by default to protect the source of truth in your Terraform code.
To make a quick change, click Unlock dashboard. This allows editing directly in the UI, all changes are automatically saved as always.\
Important: Any changes can be overwritten the next time your provisioner runs terraform apply.
Safer alternative: Duplicate the Dashboard and edit the copy, then migrate those changes back into code.
2.3) Editing dashboards via Terraform
Changing the resource and reapplying Terraform willupdate the Dashboard in groundcover.
Deleting the resource from your code (and applying) will delete it from groundcover.
Send Real‑User‑Monitoring events using JS snippet embedded in web pages
Third Party
Integrate 3rd-party data sources that push data (e.g. OpenTelemtry, AWS Firehose, FluentBit, etc.)
*Only the Sensor has limited read capability in order to support pulling remote configuration such as OTTL parsing rules applied from the UI. RUM and Third Party have write-only configurations.
Creating an Ingestion Key
It is recommended to create a dedicated Ingestion Key for every data source, so that they can be managed and rotated appropriately, minimize exposure or risk, and allow groundcover to identify the datasource of all the ingested data.
Open Settings → Access → Ingestion Keys and click Create key.
Give the key a clear, descriptive Name (for example k8s-prod‑eu‑central‑1).
Select the Type that matches your integration.
Click Click & Copy Key.
Unlike API Keys, Ingestion Keys stay visible on the page. Treat every reveal as sensitive and follow the same secret‑handling practices.
Store they Key securely, and continue to integrate your data source.
Using an Ingestion Key
Kubernetes sensor example
OpenTelemetry integration (OTel/HTTP) example
Viewing keys
The Ingestion Keys table lets you:
Reveal the key at any time.
See who created the key and when.
Sort by Type or Creator to locate specific credentials quickly.
Revoking a key
Click ⋮ → Revoke next to the key. Revocation permanently deletes the key, unlike API Keys which only disables it:
The key will disappear from the list.
Any service using it will receive 403 / PERMISSION_DENIED and will not be able to continue to send data or pull latest configurations.
This operation cannot be undone — create a new key and update your deployments if you need access again.
Ingestion Keys vs. API Keys
Ingestion Key
API Key
Primary purpose
Write data (ingest)
Read data / manage resources via REST
Permissions capabilities
Write‑only + optional remote‑config read
Mirrors service‑account RBAC
Visibility after creation
Always revealable
Shown once only
Typical lifetime
Tied to integration lifecycle
Rotated for CI/CD automations
Best Practices
One key per integration – simplifies rotation and blast radius.
Natively from docker containers running on the machine
JournalD ()
Static log files on the machine ()
Traces
Natively from docker containers running on the machine
APM metrics and insights from the traces
How to install?
Installation currently requires running a script on the machine.
The script will pull the latest sensor version and install it as a service: groundcover-sensor (requires privileges)
Install/Upgrade existing sensor:
Where:
{ingestion_Key} - A dedicated ingestion key, you can generate or find existing ones from Settings -> Access -> Ingestion Keys
Ingestion Key needs to be of Type Sensor
{BYOC_Endpoint} - Your BYOC public ingestion endpoint
{selected_Env} - The Environment that will group those machines on the cluster drop down in the top right corner (We recommend setting a separate one for non k8s deployments)
Check installed sensor status:
Check service status: systemctl status groundcover-sensor
Filter by data sources (empty array for all sources)
start
String
Yes
Start timestamp in ISO 8601 format (UTC)
end
String
Yes
Time Range Parameters
Format: ISO 8601 format with milliseconds: YYYY-MM-DDTHH:mm:ss.sssZ
Timezone: All timestamps must be in UTC (denoted by 'Z' suffix)
Response
The response contains an array of namespaces for the specified time period.
Response Fields
Field
Type
Description
namespaces
Array
Array of namespace names or namespace objects
Examples
Basic Request
Response Example
Time Range Usage
Last 24 Hours
Infrastructure Monitoring
Get complete visibility into your cloud infrastructure performance at any scale, easily access all your metrics in one place and optimize infrastructure efficiency.
Overview
The groundcover platform offers infrastructure monitoring capabilities that were built for cloud-native environments. It enables you to track the health and efficiency of your infrastructure instantly, with an effortless deployment process.
Troubleshoot efficiently - acting as a centralized hub for all your infrastructure, application and customer metrics allows you to query, correlate and troubleshoot your cloud environments using real time data and insight on your entire stack.
Store it all, without a sweat - store any metrics volume without worrying about cardinality or retention limits. remain unaffected by the granularity of metrics you store or query.
Collection
groundcover's proprietary eBPF sensor leverages all its innovative powers to collect comprehensive data across your cloud environments without the burden of performance overhead. This data is sourced from various Kubernetes components, including kube-system workloads, cluster information via the Kubernetes API, and the applications' interactions with the Kubernetes infrastructure. This level of detailed collection at the kernel level enables the ability to provide actionable insights into the health of your Kubernetes clusters, which are indispensable for troubleshooting existing issues and taking proactive steps to future-proof your cloud environments.
Configuration
You also have the option to for your metrics in the VictoriaMetrics database. By default, logs are retained for 7 days, but you can adjust this period to your preferences.
Enrichment
Beyond collecting data, groundcover's methodology involves a strategic layer of data enrichment that seeks to correlate Kubernetes metrics with application performance indicators. This correlation is crucial for creating a transparent image of the Kubernetes ecosystem. It enables a deep understanding of how Kubernetes interacts with applications, identifying across the interconnected environment. By monitoring Kubernetes not as an isolated platform but as an integral part of the application infrastructure, groundcover ensures that the monitoring strategy aligns with your dynamic and complex cloud operations.
Infrastructure Metrics
Monitoring a cluster involves tracking resources that are critical to the performance and stability of the entire system. Monitoring these essential metrics is crucial for maintaining a healthy Kubernetes cluster:
CPU consumption: It's essential to track the CPU resources being utilized against the total capacity to prevent workloads from failing due to insufficient CPU availability.
Memory utilization: Keeping an eye on the remaining memory resources ensures that your cluster doesn't encounter disruptions due to memory shortages.
Disk space allocation: For Kubernetes clusters running stateful applications or requiring persistent storage for data, such as etcd databases, tracking the available disk space is crucial to avert potential storage deficiencies.
is_loopback and remote_is_external are special labels that indicate the remote service is either the same service as the recording side (loopback) or resides in an external network, e.g managed service outside of the cluster (external).
In both cases the remote_service_name and the remote_namespace labels will be empty
Available Metrics
Name
Description
Type
List Monitors
Get a list of all configured monitors in the system with their identifiers, titles, and types.
Endpoint
POST/api/monitors/list
Authentication
This endpoint requires API Key authentication via the Authorization header.
Headers
Header
Required
Description
Request Body
The request body supports filtering by sources:
Parameters
Parameter
Type
Required
Description
Response
Response Schema
Field Descriptions
Field
Type
Description
Monitor Types
Type
Description
Examples
Basic Request
Get all monitors:
Response Example
Create Ingestion Key
Create a new ingestion key.
Endpoint
POST/api/rbac/ingestion-keys/create
Authentication
This endpoint requires API Key authentication via the Authorization header.
Headers
Request Body
Required and optional fields for creating an ingestion key:
Parameter
Type
Required
Description
Examples
Create Basic Sensor Key
Create Third-Party Integration Key
Create RUM Key with Configuration
Response
Response Schema
Response Example
Key Types
Type
Description
Default remoteConfig
Verification
To verify the key was created successfully, use the List Ingestion Keys endpoint:
Naming Requirements
Names must be lowercase with hyphens as separators
No capital letters, spaces, or special characters (except hyphens)
Examples of valid names: production-k8s-sensor, otel-staging-api, rum-frontend
Related Documentation
For comprehensive information about ingestion keys, including usage and management, see:
Get Monitor
Retrieve detailed configuration for a specific monitor by its UUID, including queries, thresholds, display settings, and evaluation parameters.
Endpoint
GET/api/monitors/{uuid}
Authentication
This endpoint requires API Key authentication via the Authorization header.
Headers
Header
Required
Description
Path Parameters
Parameter
Type
Required
Description
Field Descriptions
Field
Type
Description
Examples
Basic Request
Get monitor configuration by UUID:
Response Example - Metrics Monitor
Email via Zapier
This guide shows how to route groundcover alerts to email using Zapier. Since groundcover supports webhook-based alerting, and Zapier can receive webhooks and send emails, you can easily set up this workflow without writing code.
Prerequisites
A groundcover account with access to the Workflows tab.
A Zapier account (free plan is sufficient).
An email address where you want to receive alerts.
Step 1: Create a Webhook Integration in groundcover
Go to Settings → Integrations.
Click Create Integration.
Choose Webhook as the type.
Step 2: Create a Zapier Webhook-to-Email Workflow
Create a Webhook Trigger
Go to .
Click "Create Zap".
Set Trigger:
Configure the Email Step
Set Action:
App: Email by Zapier
Event: Send Outbound Email
Step 3: Create a Workflow in groundcover
Go to the Workflows section in your groundcover.
Create a Notification Workflow with the integration we created in step 1.
Edit the worflow YAML and use the following structure:
Step 4: Test the Flow
Trigger a test alert in groundcover.
Check Zapier to ensure the webhook was received.
Confirm the email arrives with the right content.
Synthetics
Synthetics allow you to proactively monitor the health, availability, and performance of your endpoints. By simulating requests from your infrastructure, you can verify that your critical services are reachable and returning correct data, even when no real user traffic is active.
Overview
groundcover Synthetics execute checks from your installed groundcover backend, working on BYOC deployments only.
Creating dashboards
Note: Only users with Write or Admin permissions can create and edit dashboards.
How to create a new Dashboard in groundcover?
Workflow Examples
This page provides practical examples of workflows for different use cases and integrations.
Triggers Examples
Filter by Monitor Name
List Ingestion Keys
Get a list of ingestion keys with optional filtering by name, type, and remote configuration status.
Endpoint
POST/api/rbac/ingestion-keys/list
SQL Based Monitors
Sometimes there are use cases that involve complex queries and conditions for triggering a monitor. This might go beyond the built-in query logic that is provided within the groundcover logs page query language.
An example for such a use case could be the need to compare some logs to the same ones in a past period. This is not something that is regularly available for log search but can definitely be something to alert on. If the number of errors for a group of logs dramatically changes from a previous week, this could be an event to alert and investigate.
For such use cases you can harness the powerful ClickHouse SQL language to create an SQL based monitor within groundcover.
Create a Grafana alert
Alerts in groundcover leverage a fully integrated Grafana interface. To learn how you can create alerts using Grafana Terraform, follow .
Setup an alert based on metrics
groundcover Terraform Provider
Overview
Terraform is an infrastructure-as-code (IaC) tool for managing cloud and SaaS resources using declarative configuration. The groundcover Terraform provider enables you to manage observability resources such as policies, service accounts, API keys, and monitors as code—making them consistent, versioned, and automated.
We've partnered up with Hashicorp as an official Terraform provider-
Also available is our provider Github repository:
Saved Views
Save the view of any groundcover page exactly the way you like it, then jump back in a click.
A Saved View captures your current page layout: filters, columns, toggles, etc.. so you and your team can reopen the page with the same context every time. Each groundcover page maintains its own catalogue of views, and every user can pick their personal Favorites.
Where to find them
On the pages: Traces, Logs, API Catalog, Events.
Look for the Views selector next to the time‑picker. Click it to open the list, create a new view, or switch between existing ones.
Role-Based Access Control (RBAC)
This capability is only available to organizations subscribed to our .
Role-Based Access Control (RBAC) in groundcover gives you a flexible way to manage who can access certain features and data in the platform. By defining both default roles and policies, you ensure each team member only sees and does what their level of access permits. This approach strengthens security and simplifies onboarding, allowing administrators to confidently grant or limit access.
Notification Routes
Notification Routes let you automatically send notifications to Connected Apps when monitor issues change state.
Creating and editing Notification Routes requires editor privileges in groundcover
How It Works
Backup & Restore Metrics
Learn how to backup and restore metrics into groundcover metrics storage
groundcover uses as its underlying metrics storage solution. As such, groundcover integrates seamlessly with VictoriaMetrics and tools.
Doing incremental backups
Connect Kubernetes clusters
Get up and running in minutes in Kubernetes
Before installing groundcover in Kubernetes, please make sure your cluster meets the .
After ensuring your cluster meets the requirements, complete the , then choose your preferred installation method:
Network usage: Visualize traffic rates and connections being established and closed on a service-to-service level of granularity, and easily pinpoint cross availability zone communication to investigate misconfigurations and surging costs.
groundcover_container_memory_rss_bytes
current memory RSS (B)
Gauge
groundcover_container_memory_request_bytes
K8s container memory request (B)
Gauge
groundcover_container_memory_limit_bytes
K8s container memory limit (B)
Gauge
groundcover_container_cpu_delay_seconds
K8s container CPU delay accounting in seconds
Counter
groundcover_container_disk_delay_seconds
K8s container disk delay accounting in seconds
Counter
groundcover_container_cpu_throttled_seconds_total
K8s container total CPU throttling in seconds
Counter
groundcover_node_free_disk_space
amount of free disk space in current node (B)
Gauge
groundcover_node_total_disk_space
amount of total disk space in current node (B)
Gauge
groundcover_node_used_percent_disk_space
percent of used disk space in current node (0-100)
Gauge
protocol
role
server_port
encryption
transport_protocol
is_loopback
is_cross_az means the traffic was sent and/or received between two different availability zones. This is a helpful flag to quickly identify this special kind of communication.
The actual zones are detailed in the availability_zone and remote_availability_zone labels
This example shows how to create a workflow that only triggers for a specific monitor (by its name):
Filter by Environment
Execute only on the Prod environment. The "env" attribute needs to be part of the monitor context attributes (either by using it in the group by section or by explicitly adding it as a context label):
Filter by Multiple Conditions
This example shows how to combine multiple filters. In this case it will match events from the prod environment and also monitors that explicitly routed the workflow with the name "actual-name-of-workflow":
Filter by Regex
In this case we will use a regular expression to filter on events coming from the groundcover OR monitoring namespaces. Note that any regular expression can be used:
Consts Examples
The consts section is the best location to create pre-defined attributes and apply different transformations on the monitor's metadata for formatting the notification messaging.
Map Severities
Severities in your notified destination may not match the groundcover predefined severities. By using a dictionary, you can map any groundcover severity value to another, and extract it by using the actual monitor severity. Use the "keep.dictget" function to extract from a dictionary and apply a default in case the value is missing.
Best Practice for Accessing Monitor Labels
When accessing a context label via alert.labels, if this label is not transferred within the monitor - the workflow might crash. Best practice to pre-define labels is to declare them in the consts section with a default value, using "keep.dictget" so the value is gracefully pulled from the labels object.
Note: Label names that are dotted, like "cloud.region" in this example, cannot be referenced in the monitor itself and can only be retrieved using this technique of pulling the value with "keep.dictget" from the alert.labels object.
Additional Useful Functions
keep.dict_pop({{alert.labels}}, "_gc_monitor_id", "_gc_monitor_name", "_gc_severity", "backend_id", "grafana_folder", "_gc_issue_header") - "Clean" a key-value dictionary from some irrelevant values (keys). In this case, the groundcover labels dictionary has some internal keys that you might not want to include in your notification content.
keep.join(["a", "b", "c"], ",") - Joins a list of elements into a string using a given delimiter. In this case the output is "a,b,c".
Action Examples
Conditional Statements
Use "if" condition to apply logic on different actions.
Create a separate block for a firing monitor (a resolved monitor can use different logic to change formatting of the notification):
"If" statements can include and/or logic for multiple conditions:
Notification by Specific Hours
Use the function keep.is_business_hours combined with an "if" statement to trigger an action within specific hours only.
In this example the action block will execute on Sundays (6) between 20-23 (8pm to 11pm) or on Mondays (0) between 0-1am:
Scope: A gcQL query that filters which monitors' issues will trigger this route
Rules: Define what happens when an issue is in a specific state (Firing or Resolved)
Connected Apps: Choose where to send the notification (e.g., Slack Webhook, Pagerduty, etc)
Prerequisites
Before creating notification routes, set up your Connected Apps in Settings → Connected-Apps.
Create a Notification Route
Go to Monitors → Notification Routes
Click Create Notification Route
Complete the wizard:
Step 1: Route Name
Give your route a descriptive name (e.g., prod-critical-alerts, infra-team-notifications).
Step 2: Scope Monitors
Define which monitors this route applies to using gcQL on:
The grouping labels defined in the query
The custom labels
The Monitor's metadata such as the name or severity
Examples:
env:prod — All monitors with grouping key 'env' and possible value of 'prod'
env:prod AND severity:S1 — Only critical production alerts
team:platform — Monitors with a custom label for the platform team
*:* — To match all monitors
Step 3: Rules
Rules define what happens when a scoped monitor's issue changes state.
Each rule has:
Status: When to trigger — Firing (issue is active) or Resolved (issue cleared)
Connected Apps: Where to send the notification
Example setup:
When Firing or Resolved → Send to #prod-alerts Slack channel
When Firing (only) → Send to Pagerduty service directory
Click Add Rule to create multiple rules with different status/destination combinations.
Re-notification Interval
Configure how long to wait before sending another notification while an alert is still firing.
The CLI will automatically use existing ingestion keys or provision a new one if none exist
Installing groundcover CLI
Deploying groundcover using the CLI
To upgrade groundcover to the latest version, simply re-run the groundcover deploy command with your desired overrides (such as -f values.yaml). The CLI will automatically detect and apply the latest available version during the deployment process.
global:
backend:
enabled: false
ingress:
site: {BYOC_ENDPOINT}
clusterId: "your-cluster-name" # CLI will automatically detect cluster name
env: "your-environment-name" # Add it to differnciate between different environments
sh -c "$(curl -fsSL https://groundcover.com/install.sh)"
groundcover deploy -f values.yaml
sh -c "$(curl -fsSL https://groundcover.com/install.sh)"
# delete the namespace in order to remove the PVCs as well
kubectl delete ns groundcover
Source: Checks run from within your backend, when having multiple groundcover backends you can select the specific backend to use. We will support region selection for running tests from specific locations.
Supported Protocols: Currently, Synthetics supports HTTP/HTTPS tests. Support for additional protocols, including gRPC, ICMP (Ping), DNS, and dedicated SSL monitors are coming soon.
Alerting: Creating a Synthetic test automatically creates a corresponding Monitor (See: Monitors). Using monitors you can get alerted on failed synthetic tests, see: Notification Channels . The monitor is uneditable.
Trace Integration: We generate traces for all synthetic tests, which you can see as first-class citizens in groundcover platform. You can query these traces by using source:synthetics in traces page.
Creating a Synthetic Test
Navigate to Monitors > Synthetics and click + Create Synthetic Test .
Only Editors can create/edit/delete synthetic tests, see Default Policies
Request Configuration
Define the endpoint and parameters for the test.
Synthetic Test Name: A descriptive name for the test.
Target: Select the method (GET, POST, etc.) and URL. Include HTTP scheme as well, for example: https://api.groundcover.com/api/backends/list
Tip: Use Import from cURL to paste a command and auto-fill these fields.
HTTP Settings
Follow redirects: Should the test follow 3xx responses, when disabled the test will return the 3xx response as the result set for assertions.
Allow insecure: Disables SSL/TLS certificate verification. Use this only for internal testing or self-signed certificates. Not recommended for production endpoints as it exposes you to Man-in-the-Middle attacks.
Timing
Interval: Frequency of the check (e.g., every 60s).
Timeout: Max duration to wait before marking the test as failed. Timeout must be less than interval.
Payload: Select the body type if your request requires data (e.g., POST/PUT).
Options: None, JSON, Text, Raw.
Headers & Auth:
Authentication (Bearer tokens, API keys) will be released soon.
Headers: You can add custom headers passing key and values.
Assertions (Validation Logic)
Assertions are the rules that determine if a test passed or failed. You can add multiple assertions to a single test. If any assertion fails, the entire check is marked as failed.
Available Assertion Fields
The "Field" determines which part of the response groundcover inspects.
Field
Description
statusCode
Checks the HTTP response code (e.g., 200, 404, 500).
responseHeader
Checks for the presence or value of a specific response header (e.g., Content-Type).
jsonBody
Inspects specific keys or values within a JSON response payload.
body
Checks the raw text body of the response.
responseTime
Checks the response time of the response
jsonBody
Inspects specific keys or values within a JSON response payload.
Assertion Operators
The "Operator" defines the logic applied to the Field.
Operator
Function
Example Use Case
is equal to
Exact match. Case-sensitive.
statusCode is equal to 200
is not equal to
Ensures a value does not appear.
statusCode is not equal to 500
contains
Checks if a substring exists within the target.
body contains "error"
starts with
Verifies the beginning of a string.
"status"
Custom Labels
Add custom labels, these labels will exist on traces generated by checks. You can use these labels to filter traces.
Auto-Generated Monitors
When you create a Synthetic Test, groundcover eliminates the need to manually configure separate alert rules. A Monitor is automatically generated and permanently bound to your test. See: Monitors .
Managed Logic: The monitor's threshold and conditions are derived directly from your Synthetic Test's assertions. If the test fails (e.g., status code != 200), the Monitor automatically enters a "Failing" state.
Lifecycle: This monitor handles the lifecycle of the alert, transitioning between Pending, Firing (when the test fails), and Resolved (when the test passes).
Zero Maintenance: You do not need to edit this monitor's query. Any changes you make to the Synthetic Test (such as changing the target URL or assertions) are automatically synced to the Monitor.
Note: To prevent configuration drift, these auto-generated monitors are read-only. You cannot edit their query logic directly; you simply edit the Synthetic Test itself.
Navigate to the Dashboard List and click on the Create New Dashboard button.
Provide an indicative name for your dashboard and, optionally, a description.
Steps to creating a Dashboard
Create a new widget
Choose a Widget Type
Select a Widget Mode
Build your query
Choose a Display Type
Save the widget
Optional:
Add variables
Apply variable(s) to the widget
Create a new Widget
Widgets can be added by clicking on the Create New Widget button.
Choose a Widget Type
Widgets are the main building blocks of dashboards. groundcover supports the following widget types:
Chart Widget: Visualize your data through various display types.
Textual Widget: Add context to your dashboard, such as headers or instructions for issue investigations.
Since selecting a Textual Widget is the last step for this type of widget, the rest of this guide is relevant only to Chart Widgets.
Select a Widget Mode
Metrics: Work with all your available metrics for advanced use cases and custom metrics.
Infra Metrics: Use expert-built, predefined queries for common infrastructure scenarios. Ideal for quick starts.
Logs: Query and visualize log data.
Traces: Query and visualize trace data similar to logs.
Build your query
Once the Widget Mode selected, build your query for the visualization.
If you're unfamiliar with query building in groundcover, refer to the Query Builder section for full details on the different components.
Choose a Display Type
Type
Configuration options
Supported modes
Time Series
Choose a Y-axis unit from the predefined list.
Select a visualization type: Stacked Bar or Line Chart.
Metrics
Infra Metrics
Logs
Traces
Table
Define columns based on data fields or metrics.
Choose a Y-axis unit from the predefined list.
Metrics
Infra Metrics
Logs
Traces
Stat
Select a Y-axis unit from the predefined list.
Metrics
Infra Metrics
Logs
Traces
Top List
Choose a ranking metric and sort order.
Variables
Variables dynamically filter your entire dashboard or specific widgets with just one click. They consist of a key-value pair that you define once and reuse across multiple widgets.
Our predefined variables cover most use cases, but if you’re missing an important one, let us know. Advanced variables are also on our roadmap.
Adding a Variable
Click on Add Variable and configure the variable using the following fields.
The data source to be used:
Suggested Variables - A predefined list of popular variables which use 'All Datasources' behind the scenes.
All Datsources - Will show values for the chosen key from logs, traces, all metrics, and events.
Traces, Logs, Events - Only show values for the chosen key for data coming from these sources.
Metrics - Show values for a chosen label key from a chosen metric name.
Choose the label key to be used to fetch values from the data source selected.
Choose the name of the variable to be used in the widgets with $ as explained below.
Using a Variable
Variables can be referenced in the Filter Bar of the Widget Creation Modal using their name.
In this following example we selected Clusters from the predefined list, and named it 'clusters'.
While creating or editing a Chart Widget, add a reference to the variable using a dollar sign in the filter bar, (for example, $clusters).
The data will automatically filter by the variable's key with the selected values. If all values are selected, the filter will be followed by an asterisk (for example, cluster:.*)
After configuring the Variable in the widget queries, you may select the values you which to filter and choose the default to be used when the dashboard loads on first time.
After selecting values in at least one Variable, all other relevant Variables will render an 'Associated Values' section in the dropdown list. This list renders the values of the selected variable's key which are associated with the values of the currently selected variables' keys.
For example- selecting the value production in a variable called cluster which uses the key cluster from All Datasources, then when going to workloads variable I can see in the 'Associated Values' section a list of workload values that are in the production cluster.
Below in the 'Additional Values' will be shown all other values.
The association is done by relevant data types only, if you are getting unexpected associated results you may be advised to narrow down the data sources that the variable uses from 'All Datasources' to a specific type or a specific metric.
Limitations and tips-
It's possible that there are associated values which don't appear in the list, this list is not hermetic, but anything associated is necessarily associated.
Start to type the value you are searching to narrow down the list.
Authentication
This endpoint requires API Key authentication via the Authorization header.
Headers
Request Body
Optional filters for ingestion keys:
Parameter
Type
Required
Description
name
string
No
Filter by exact key name
type
string
No
Filter by key type ("sensor", "thirdParty", "rum")
remoteConfig
boolean
No
Examples
Get All Ingestion Keys
Filter by Type
Get only sensor keys:
Filter by Name and Remote Config
Response Example
Response Schema
Field
Type
Description
id
string
Unique identifier for the ingestion key (UUID)
name
string
Human-readable name for the key
createdBy
string
Email of the user who created the key
creationDate
string
ISO 8601 timestamp of key creation
Related Documentation
For comprehensive information about ingestion keys, including creation, usage, and best practices, see:
Log and Trace telemetry data is stored within a ClickHouse database.
You can directly query this data using SQL statements and create powerful monitors.
To create and test your SQL queries use the Grafana Explore page within the groundcover app.
Select the ClickHouse@groundcover datasource with the SQL Editor option to start crafting your SQL queries
Start with show tables; to see of all the available tables to use for your queries: logs and traces would be popular choices (table names are case sensitive).
Query best practices
While testing your queries always use LIMIT to limit your results to a small set of data.
To apply the Grafana timeframe on your queries make sure to add the following conditions:
Logs: WHERE $__timeFilter(timestamp)
Traces: WHERE $__timeFilter(start_timestamp)
Note: When querying logs with SQL, it's crucial to use efficient filters to prevent timeouts and enhance performance. Implementing primary filters like cluster, workload, namespace, and env will significantly speed up queries. Always integrate these filters when writing your queries to avoid inefficient queries.
Filtering on attributes and tags
Traces and Logs have rich context that is normally stored in dedicated columns in json format. Accessing the context for filtering and retrieving values is a popular need when querying the data.
To get to the relevant context item, either in the attributes or tags you can use the following syntax:
WHERE string_attributes['host_name'] = 'my.host'
WHERE string_tags['cloud.name'] = 'aws'
WHERE float_attributes['hradline_count'] = 4
WHERE float_tags['container.size'] = 22.4
To use the float context ensure that the relevant attributes or tags are indeed numeric. To do that, check the relevant log in json format to see if the referenced field is not wrapped with quotes (for example, headline_count in the screenshot below)
SQL Query structure for a monitor
In order to be able to use an SQL query to create a monitor you must make sure the query returns no more than a single numeric field - this is the monitored field on which the threshold is placed.
The query can also contain any number of "group by" fields that are passed to the monitor as context labels.
Here is an exmaple of an SQL query that can be used for a monitor
In this query the threshold field is a ratio between some value measured on the last week and in the last 10 minutes.
tenantID and env are the group by labels that are passed to the monitor as context labels.
Here is another query example (check the percentage of errors in a set of logs):
A single numeric value is calculated and grouped by cluster, namespace and workload
Applying the SQL query as a monitor
Applying an SQL query can only happens in YAML mode. You can use the following YAML template to add your query
Give your monitor a name and a description
Paste your SQL query in the expression field
Set the threshold value and the relevant operator - in this example this is "lower than" 0.5 (< 0.5)
Set your workflow name in the annotations section
Set the check interval and the pending time
Save the monitor
Trigger an alert when no logs are coming from a Linux host
Use YAML mode to add the following template to your monitor.
In this example we are creating a list of Linux hosts that were sending logs in the last 24 hours and then checking if there were any logs collected from those hosts in the last 5 minutes.
This monitor can be used e.g. to catch when the host is down.
It would be helpful if you add an indication on the monitor name that this is SQL based. For example, add an [SQL] prefix or suffix to the monitor name as shown in the example
Setting up an alert in groundcover involves defining conditions based on data collected in the platform, such as metrics, traces, logs, or Kubernetes events. This guide will walk you through the process of creating an alert based on metrics. More guides will follow to include all different types of data.
Step 1: Access the Alerts section
Log in to groundcover and navigate to the Alerts section by clicking on it on the left navigation menu.
Once in the Alerts section, click on Alerting in the inner menu on the left.
If you can't see the inner menu, click on the 3 bars next to "Home" in the upper left corner.
Click on Alert Rules
Then click on the blue "+ New alert rule" button in the upper right.
Step 2: Give the alert a name and define query and conditions
Type a name for your alert. It's recommended to use a name that will make it easy for you to understand its function later.
Select the data source:
ClickHouse: For alerts based on your traces, logs, and Kubernetes events.
Prometheus: For alerts based on metrics (includes APM metrics, infrastructure metrics, and custom metrics from your environment)
Click on "Select metric"
Note: Make sure you are in "Builder" view (see screenshot) to see this option.
Click on "Metrics explorer"
Start typing the name of the metric you want this alert to be based on. Note that the Metrics explorer will start displaying matches as you type, so you can find your metric even if you don't remember it exact name. You can also check out our list of .
Once you see your metric in the list, click on "Select" in that row.
Note: You can click on "Run queries" to see the results of this query.
Step 3: Define expressions - Reduce & Threshold
In the Reduce section, open on the "Function" dropdown menu and choose the type of value you want to use.
Min - the lowest value
Max - the highest value
Mean - the average of the values
Sum - the sum of all values
Count - the number of values in the result
Last - the last value
In the Threshold section, type a value and choose whether you want the alert to fire when the query result is above or below that value. You can also select a range of values.
Step 4: Set evaluation behavior
Click on "+ New folder" and type a name for the folder in which this rule will be stored. You can choose any name, but it's recommended to use a name that will make it easy for you to find the relevant evaluation groups, should you want to use them again in future alerts.
Click on "+ New evaluation group" and type a name for this evaluation group. The same recommendation applies here too.
In the Evaluation interval textbox, type how often the rule should be evaluated to see if it matches the conditions set in Step 3.
Then, click "Create".
Note: For the Evaluation interval, use the format (number)(unit), where units are:
s = seconds
m = minutes
h = hours
d = days
w = weeks\
In the Pending period box, type how often you want the alert to match the conditions before it fires.
Evaluation interval = how often do you want to check if the alert should fire
Pending period = how long do you want this to be true before it fires
As an example, you can define the alert to fire only if the Mean percentage of memory used by a node is above 90% in the past 2 minutes (Pending period = 2m) and you want to check if that's true every 30 seconds (Evaluation interval = 30s).
Step 5: Choose contact point
If you already have a contact point set up, simply select it from the dropdown menu at the bottom of the "Configure lables and notifications" section. If not, click on the blue "View or create contact points" link, which will open a new tab.
Click on the blue "Add contact point" button
This will get you to the Contact points screen. Then:
Type a name for the contact point
From the dropdown menu, choose which system you want to use to push the alert to.
The information required to push the alert will change based on the system you select. Follow on-screen instructions (for example, if email is selected, you'll need to enter the email address(es) for that contact.
Click "Save contact point"
You can now close this tab to go back to the alert rule screen.
Next to the link you clicked to create this new contact point, you'll find a dropdown menu, where you can select the contact point you just created.
Step 6: Add annotations
Under "Add annotations", you have two free text boxes that give you the option to add any information that can be useful to you and/or the recipient(s) of this alert, such as a summary that reminds you of the alert's functionality or purpose, or next step instructions when this alert fires.
Step 7: Save and exit
Once all of it is ready, you can click the blue "Save rule and exit" button on the upper right of the screen, which will bring you back to the Alert rules screen.
You will now be able to see your alert, as well as its status - normal (green), pending (yellow), or firing (red), as well as the Evaluation interval (blue).
Configuring Alert from existing dashboard:
Log in to your groundcover account and navigate to the dashboard that you want to create an alert from.
Locate the Grafana panel that you want to create an alert from and click on the panel's header and select edit .
Click on the alert tab as seen in the image below. Select the Manage alerts option from the dropdown menu.
Click on the New Alert Rule button.
Note: only time series panels support alert creation.
An alert is derived from three parts that will be configured in the screen that you are navigated to:
Expression - the query that defines the alert input itself,
Reduction - the value that should be leveraged from the aforementioned expression
Threshold - value to measure against said reduciton output to see if an alert should be triggered
Verify expression value and enter reduction and threshold values in line with your alerting expectation
Select folder - if needed you can navigate to dashboard tab in left nav and create new folder
Select evaluation ground or type text in order to create a new group as shown below
Click "Save and Exit" on top right hand side of screen to create alert
Ensure your notification is configured to have alerts sent to end users. See "Configuring Slack Contact Point" section below if needed.
Note: Make sure to test the alert to ensure that it is working as expected. You can do this by triggering the conditions that you defined and verifying that the alert is sent to the specified notification channels.
api_key (String, Required, Sensitive): Your groundcover API key. It is strongly recommended to configure this using the TF_VAR_groundcover_api_key environment variable rather than hardcoding it.
base_url (String, Optional): The base URL for the groundcover API. Defaults to https://api.groundcover.com if not specified.
backend_id (String, Required): Your Backend ID can be found in the API Keys screen in the groundcover UI (Under Settings -> Access):
Configure the page until it looks exactly right—filters, columns, panels, etc.
Click ➕Save View.
Give the view a clear Name.
Hit Save. The view is now listed and available to everyone in the project.
Scope – Saved Views are per‑page. A Logs view appears only in Logs; a Traces view only in Traces.
What a view stores
Common to all pages
Category
Details
Filters & facets
All query filters plus facet open/closed state
Columns
Chosen columns, and their order, sort, width
FilterPanel & Added Facets
Filter panel open/closed
Facets added/removed
Page‑specific additions
Page
Extra properties saved
Logs
logs / patterns
textWrap
Insight
Traces
traces / span
table / drilldown
API Catalog
protocol
Kafka role: Fetcher / Producer
Events
textWrap
Updating a view
The Update View button appears only when you are the creator of the view. Click it to overwrite the view with your latest changes.
Underneath every View you can see which user created it.
Managing views (row operations)
Action
Who can do it?
Edit / Rename
Creator
Delete
Creator
Star / Unstar
Any user for themselves
Searching, Sorting, and Filtering the list
Searching the Views will look up based on View names and the creators.
The default sorting pins the favorites views at the top, and the rest of the views below. Each group of views is sorted from A→Z.
In addition, 3 filtering options are available:
All Views - The entire workspace's views for a specific page
My Favorites – The favorite views of the user for a specific page
Created By Me - The views created by the user
Policies
Policies are the foundational elements of groundcover’s RBAC. Each policy defines:
A permission level – which actions the user can perform (Admin, Editor, or Viewer-like capabilities).
A data scope – which clusters, environments, or namespaces the user can see.
By assigning one or more policies to a user, you can precisely control both what they can do and where they can do it.
Default Policies
groundcover provides three default policies to simplify common use cases:
Default Admin Policy
Permission: Admin
Data Scope: Full (no restrictions)
Behavior: Unlimited access to groundcover features and configurations.
Default Editor Policy
Permission: Editor
Data Scope: Full (no restrictions)
Default Viewer Policy
Permission: Viewer
Data Scope: Full (no restrictions)
These default policies allow you to quickly onboard new users with typical Admin/Editor/Viewer capabilities. However, you can also create custom policies with narrower data scopes, if needed.
Policy Structure
A policy’s data scope can be defined in two modes: Simple or Advanced.
Simple Mode
Uses AND logic across the specified conditions.
Applies the same scope to all entity types (e.g., logs, traces, events, workloads).
Example: “Cluster = DevAND Environment = QA,” restricting all logs, traces, events, etc. to the Dev cluster and QA environment.
Advanced Mode
Lets you define different scopes for each data entity (logs, traces, events, workloads, etc.).
Each scope can use OR logic among conditions, allowing more fine-grained control.
When creating or editing a policy, you select permission (Admin, Editor, or Viewer) and a data scope mode (Simple or Advanced).
Multiple Policies
A user can be associated with multiple policies. When that occurs:
Permission Merging
The user’s final permission level is the highest among all assigned policies.
Example: If one policy grants Editor and another grants Viewer, the user is effectively an Editor overall.
Data Scope Merging
Data scopes merge via OR logic, broadening the user's overall data access.
Example: Policy A => "Cluster = A," Policy B => "Environment = B," so final scope is "Cluster A OR Environment B."
A user may be assigned a policy granting the Editor role with a data scope relevant to specific clusters, and simultaneously be assigned another policy granting the Viewer role with a different data scope. The user's effective access is determined by the highest role across all assigned policies and by the union (OR) of scopes.
In summary:
Policies define both permission (Admin, Editor, or Viewer) and data scope (clusters, environments, namespaces).
Default Policies (Admin, Editor, Viewer) provide no data restrictions, suitable for quick onboarding.
Custom Policies allow more granular restrictions, specifying exactly which entities a user can see or modify.
Multiple Policies can co-exist, merging permission levels and data scopes via OR logic across all data types.
This flexible system gives you robust control over observability data in groundcover, ensuring each user has precisely the access they need.
This document explains how to paginate through large log result sets using the groundcover logs query API.
Overview
When querying logs, you may receive more results than can be efficiently returned in a single response. Pagination allows you to retrieve results in smaller, manageable chunks by using the limit and skip parameters.
Parameters
Parameter
Type
Description
Example
Note: The limit parameter can get a maximum value of 5000
Basic Pagination
To paginate through results:
First page: Set skip: 0 and limit to your desired page size
Subsequent pages: Increment skip by the limit value for each page
Example: First Page
Example: Second Page
Example: Third Page
Pagination Formula
To calculate the skip value for any page:
Where:
page_number is the page you want to retrieve (1, 2, 3, ...)
limit is your page size
Examples:
Page 1 with limit 200: skip = (1 - 1) × 200 = 0
Page 2 with limit 200: skip = (2 - 1) × 200 = 200
Page 3 with limit 200: skip = (3 - 1) × 200 = 400
Determining When to Stop
The response includes a limitReached field that indicates whether the result limit was reached:
If limitReached: true, there may be more results available. Continue to the next page.
If limitReached: false and you received fewer results than your limit, you've reached the end of the results.
Best Practices
Consistent Page Size: Use the same limit value for all pages in a pagination sequence to ensure consistent results.
Reasonable Limits: Choose a limit value that balances performance and usability. Common values are 50, 100, or 200.
Sort Order: Always use the same sortBy
Example: Pagination Loop
Here's a pseudo-code example of how to implement pagination:
Pagination with Filtering
When using pagination with filtered queries (using the group parameter), ensure you use the same group conditions across all pages to maintain consistent filtering:
Important: Keep the same group, start, end, sortBy, and sortOrder values across all pages to ensure consistent pagination through the same filtered dataset.
List Workflows
Get a list of all configured alert workflows with their complete definitions, provider integrations, execution status, and YAML configurations.
Endpoint
POST/api/workflows/list
Authentication
This endpoint requires API Key authentication via the Authorization header.
Headers
Header
Required
Description
Request Body
This endpoint does not require a request body for the POST method.
Field Descriptions
Field
Type
Description
Examples
Basic Request
Get all workflows:
Response Example
Integration Examples with Workflows
This page provides examples of how to integrate workflows with different notification systems and external services.
Slack Notification
This workflow sends a simple Slack message when triggered:
Slack with Rich Formatting
This workflow sends a formatted Slack message using Block Kit:
PagerDuty Integration
This workflow creates a PagerDuty incident:
Opsgenie Integration
This workflow creates an Opsgenie alert:
Alias is used to group identical events together in Opsgenie (alias key in the payload)
Severities must be mapped to Opsgenie valid severities (priority key in the payload)
Tags are a list of string values (tags key in the payload)
Jira Ticket Creation
This workflow creates a Jira ticket using webhook integration:
Multiple Actions
This workflow performs multiple actions for the same alert:
Application Metrics
Our metrics philosophy
The groundcover platform generates 100% of its metrics from the actual data. There are no sample rates or complex interpolations to make up for partial coverage. Our measurements represent the real, complete flow of data in your environment.
allows us to construct the majority of the metrics on the very node where the raw transactions are recorded. This means the raw data is turned into numbers the moment it becomes possible - removing the need for storing or sending it elsewhere.
with engineStatusLastWeek as (
select string_attributes['tenantID'] tenantID, , string_attributes['env'] env, max(float_attributes['engineStatus.numCylinders']) cylinders
from logs
where timestamp >= now() - interval 7 days
and workload = 'engine-processing'
and string_attributes['tenantID'] != ''
group by tenantID, env
),
engineStatusNow as (
select string_attributes['tenantID'] tenantID, string_attributes['env'] env, min(float_attributes['engineStatus.numCylinders']) cylinders
from logs
where timestamp >= now() - interval 10 minutes
and workload = 'engine-processing'
and string_attributes['tenantID'] != ''
group by tenantID, env
)
select n.tenantID, n.env, n.cylinders/lw.cylinders AS threshold
from engineStatusNow n
left join engineStatusLastWeek lw using (tenantID)
where n.cylinders/lw.cylinders <= 0.5
SELECT cluster, namespace, workload,
round( 100.0 * countIf(level = 'error') /
nullIf(count(), 0), 2 ) AS error_ratio_pct
FROM "groundcover"."logs"
WHERE timestamp >= now() - interval '10 minute' AND
namespace IN ('refurbished', 'interface') GROUP BY cluster, namespace, workload
title: Host not sending logs more than 5 minutes
display:
header: Host "{{host}}" is not sending logs for more than 5 minutes
severity: S2
measurementType: event
model:
queries:
- name: threshold_input_query
expression: "
WITH
(
SELECT groupArray(DISTINCT host)
FROM logs
WHERE timestamp >= now() - INTERVAL 24 HOUR
AND env_type = 'host'
) AS all_hosts
SELECT
host,
coalesce(log_count, 0) AS log_count
FROM
(
SELECT arrayJoin(all_hosts) AS host
) AS h
LEFT JOIN
(
SELECT host, count(*) AS log_count
FROM logs
WHERE timestamp >= now() - INTERVAL 5 MINUTE
AND env_type = 'host'
GROUP BY host
) AS l
USING (host)
ORDER BY host
"
datasourceType: clickhouse
queryType: instant
thresholds:
- name: threshold_1
inputName: threshold_input_query
operator: lt
values:
- 10
annotations:
{Put Your Workflow Name Here}: enabled
executionErrorState: Error
noDataState: NoData
evaluationInterval:
interval: 5m
pendingFor: 0s
isPaused: false
terraform {
required_providers {
groundcover = {
source = "registry.terraform.io/groundcover-com/groundcover"
version = ">= 0.0.0" # Replace with actual version constraint
}
}
}
provider "groundcover" {
api_key = "YOUR_API_KEY" # Required
base_url = "https://api.groundcover.com" # Optional, change if using onprem/airgap deployment
backend_id = "groundcover" # Your Backend ID can be found in the groundcover UI under Settings->Access->API Keys
}
resource "groundcover_policy" "read_only" {
name = "Read-Only Policy"
description = "Grants read-only access"
claim_role = "ci-readonly-role"
roles = {
read = "read"
}
}
resource "groundcover_serviceaccount" "ci_account" {
name = "ci-pipeline-account"
description = "Service account for CI"
policy_uuids = [groundcover_policy.read_only.id]
}
resource "groundcover_apikey" "ci_key" {
name = "CI Key"
description = "Key for CI pipeline"
service_account_id = groundcover_serviceaccount.ci_account.id
}
page_size = 200
skip = 0
all_logs = []
while True:
response = query_logs(limit=page_size, skip=skip)
logs = response['logs']
all_logs.extend(logs)
# Check if we've reached the end
if not response['limitReached'] or len(logs) < page_size:
break
# Move to next page
skip += page_size
Behavior: Full creation/editing capabilities on observability data, but no user or system management.
Behavior: Read-only access to all data in groundcover.
Example:
Logs: “Cluster = DevORProd,”
Traces: “Namespace = abc123,”
Events: “Environment = StagingORProd.”
This applies to all data types including logs, traces, events, workloads, and metrics.
victoria-metrics
deployment, ensuring top-notch performance on every scale.
Golden signals
In the world of excessive data, it's important to have a rule of thumb for knowing where to start looking. For application metrics, we rely on our golden signals.
The following metrics are generated for each resource being aggregated:
Requests per second (RPS)
Errors rate
Latencies (p50 and p95)
The golden signals are then displayed in two important ways: Workload and Resource aggregations.
See below for the full list of generated workload and resource golden metrics.
Resource aggregations are highly granularity metrics, providing insights into individual APIs.
Workload aggregations are designed to show an overview of each service, enabling a higher level inspection. These are constructed using all of the resources recorded for each service.
Controlling retention
groundcover allows full control over the retention of your metrics. Learn more here.
List of available metrics
Below you will find the full list of our APM metrics, as well as the labels we export for each. These labels are designed with high granularity in mind for maximal insight depth. All of the metrics listed are available out of the box after installing groundcover, without any further setup.
We fully support the ingestion of custom metrics to further expand the visibility into your environment.
We also allow for building custom dashboards, enabling full freedom in deciding how to display your metrics - building on groundcover's metrics below plus every custom metric ingested.
Our labels
Label name
Description
Relevant types
clusterId
Name identifier of the K8s cluster
region
Cloud provider region name
namespace
K8s namespace
workload_name
K8s workload (or service) name
Summary based metrics have an additional quantile label, representing the percentile. Available values: [”0.5”, “0.95”, 0.99”].
This document provides advanced examples of querying logs using the groundcover API with complex filtering conditions.
Overview
The group parameter supports various filter operations and combinations. This page demonstrates advanced patterns including:
OR conditions within a single field
NOT conditions (excluding values)
Wildcard matching
Free text search
OR Conditions on a Single Field
To match multiple values for the same field (OR condition), include multiple filters in the same condition:
Example: Match logs where env is either "prod" OR "dev" AND level is "error"
Structure:
Multiple filters in the same conditions entry create an OR relationship
In this example, the env field matches if it equals "prod" OR "dev"
The overall
NOT Conditions (Excluding Values)
To exclude specific values, use the "op": "not_match" operation:
Example: Match logs where level is NOT "error" AND env is "prod"
Structure:
Use "op": "not_match" to exclude values
This example returns logs where level is anything except "error" AND env is "prod"
Wildcard Matching
To match values that start with, end with, or contain a pattern, use wildcard characters:
Example: Match logs where workload starts with "groundcover" AND level is NOT "error"
Structure:
Use * as a wildcard character
"groundcover*" matches any value starting with "groundcover" (e.g., "groundcover-backend", "groundcover-frontend")
Wildcards can be used with both match and not_match
Free Text Search
To search for specific text or phrases within log content (not limited to specific fields), use free text search with phrase_search:
Example: Search for logs containing the phrase "POST /ingest" with additional fields
Structure:
Use "op": "phrase_search" for free text search
Set "type": "freetext" (instead of "type": "string")
Do not include a "key"
Filter Operations
The following filter operations are supported:
Operation
Description
Example
Combining Conditions
AND Operator
When using "operator": "and", all conditions must be satisfied:
This means: level condition ANDenv condition must both be true.
OR Within a Field
Multiple filters in the same condition create an OR relationship:
This means: env equals "prod" ORenv equals "dev".
Complex Combinations
You can combine AND and OR operations:
This means: level equals "error" AND (env equals "prod" ORenv equals "dev").
Best Practices
Wildcard Performance: Use wildcards judiciously as they can impact query performance on large datasets.
NOT Operations: not_match operations can be slower than match operations. Consider if you can achieve the same result with positive matches.
Multiple Filters: When using multiple filters in a single condition (OR), ensure they all apply to the same field.
List Clusters
Retrieve a list of Kubernetes clusters with their resource usage metrics, metadata, and health information.
Endpoint
Authentication
This endpoint requires API Key authentication via the Authorization header.
Headers
Header
Required
Description
Request Body
Parameter
Type
Required
Description
Response
The response contains an array of clusters with detailed resource usage and metadata.
Response Fields
Field
Type
Description
Cluster Object Fields
Field
Type
Description
CPU Metrics
Field
Type
Description
Memory Metrics
Field
Type
Description
Pod Information
Field
Type
Description
Examples
Basic Request
Response Example
LLM Observability
Overview
LLM Observability is the practice of monitoring, analyzing, and troubleshooting interactions with Large Language Models (LLMs) across distributed systems. It focuses on capturing data regarding prompt content, response quality, performance latency, and token costs.
groundcover provides a unified view of your GenAI traffic by combining two powerful data collection methods: zero-instrumentation eBPF tracing and native OpenTelemetry ingestion.
List Workloads
Retrieve a list of Kubernetes workloads with their performance metrics, resource usage, and metadata.
Endpoint
Authentication
This endpoint requires API Key authentication via the Authorization header.
List Nodes with Resource Information
Endpoint
POST /api/k8s/v2/nodes/info-with-resources
POST /api/k8s/v3/clusters/list
pod_name
K8s pod name
container_name
K8s container name
container_image
K8s container image name
remote_namespace
Remote K8s namespace (other side of the communication)
remote_service_name
Remote K8s service name (other side of the communication)
remote_container_name
Remote K8s container name (other side of the communication)
groundcover automatically detects and traces LLM API calls without requiring SDKs, wrappers, or code modification.
The sensor captures traffic at the kernel level, extracting key data points and transforming requests into structured spans and metrics. This allows for instant visibility into third-party providers without altering application code. This method captures:
Payloads: Full prompt and response bodies (supports redaction).
Usage: Token counts (input, output, total).
Metadata: Model versions, temperature, and parameters.
Performance: Latency and completion time.
Status: Error messages and finish reasons.
OpenTelemetry Instrumentation Support
In addition to auto-detection, groundcover supports the ingestion of traces generated by manual OpenTelemetry instrumentation.
If your applications are already instrumented using OpenTelemetry SDKs (e.g., using the OpenTelemetry Python or JavaScript instrumentation for OpenAI/LangChain), groundcover will seamlessly ingest, process, and visualize these spans alongside your other telemetry data.
Generative AI Span Structure
When groundcover captures traffic via eBPF, it automatically transforms the data into structured spans that adhere to the OpenTelemetry GenAI Semantic Conventions.
This standardization allows LLM traces to correlate with existing application telemetry. Below are the attributes captured for each eBPF-generated LLM span:
Attribute
Description
Example
gen_ai.system
The Generative AI provider
openai
gen_ai.request.model
The model name requested by the client
gpt-4
gen_ai.response.model
The name of the model that generated the response
gpt-4-0613
gen_ai.response.usage.input_tokens
Tokens consumed by the input (prompt)
Generative AI Metrics
groundcover automatically generates rate, errors, duration and usage metrics from the LLM traces. These metrics adhere to OpenTelemetry GenAI conventions and are enriched with Kubernetes context (cluster, namespace, workload, etc).
Metrics can be filtered by: workload, namespace, cluster, gen_ai_request_model, gen_ai_system, client, server, and status_code.
Configuration
Obfuscation Configuration
LLM payloads often contain sensitive data (PII, secrets). By default, groundcover collects full payloads to aid in debugging. You can configure the agent to obfuscate specific fields within the prompts or responses using the httphandler configuration in your values.yaml.
By default groundcover does not obfuscate LLM payloads.
Obfuscating Request Prompts
This configuration will obfuscate request prompts, while keeping metadata like model, tokens, etc
Obfuscating Response Prompts
This configuration will obfuscate response data, while keeping metadata like model, tokens, etc
Supported Providers
groundcover currently supports the following providers via auto-detection:
OpenAI (Chat Completion API)
Anthropic (Chat Completion API)
AWS Bedrock APIs
For providers not listed above, manual OpenTelemetry instrumentation can be used to send data to groundcover.
Headers
Header
Required
Description
Authorization
Yes
Bearer token with your API key
X-Backend-Id
Yes
Your backend identifier
Content-Type
Yes
Must be application/json
Accept
Yes
Must be application/json
Request Body
Parameter
Type
Required
Default
Description
conditions
Array
No
[]
Filter conditions for workloads
limit
Integer
No
100
Maximum number of workloads to return (1-1000)
Response
The response contains a paginated list of workloads with their metrics and metadata.
Response Fields
Field
Type
Description
total
Integer
Total number of workloads available
workloads
Array
Array of workload objects
Workload Object Fields
Field
Type
Description
uid
String
Unique identifier for the workload
envType
String
Environment type (e.g., "k8s")
env
String
Environment name (e.g., "prod", "ga", "alpha")
cluster
String
Kubernetes cluster name
Examples
Basic Request
Response Example
Pagination
To retrieve all workloads, use pagination by incrementing the skip parameter:
Fetching All Results
Pagination Logic
To fetch all results programmatically:
Start with skip=0 and limit=100 (or your preferred page size)
Check the total field in the response
Continue making requests, incrementing skip by your limit value
Stop when skip >= total
Example calculation:
If total is 6314 and limit is 100
You need ⌈6314/100⌉ = 64 requests
Last request: skip=6300, limit=100 (returns 14 items)
Authentication
This endpoint requires API key authentication.
Headers
Header
Value
Description
Authorization
Bearer <YOUR_API_KEY>
Your groundcover API key
Content-Type
application/json
Request body format
X-Backend-Id
<YOUR_BACKEND_ID>
Your backend identifier
Request Body
Field
Type
Required
Description
start
string
Yes
Start time in ISO 8601 UTC format
end
string
Yes
End time in ISO 8601 UTC format
sources
array
No
Sources Structure (Cluster Filter)
Example Request
Response
Response Fields
Field
Type
Description
nodes
array
Array of node objects
nodes[].uid
string
Unique identifier for the node
nodes[].name
string
Node name
nodes[].cluster
string
Cluster name
Filter Operations
Operator
Description
eq
Equals
ne
Not equals
gt
Greater than
lt
Less than
contains
Contains substring
Common Use Cases
Get All Nodes
Filter by Specific Cluster
List Deployments
Get a list of Kubernetes deployments with status information, replica counts, and operational conditions for a specified time range.
Endpoint
POST/api/k8s/v2/deployments/list
Authentication
This endpoint requires API Key authentication via the Authorization header.
Headers
Header
Required
Description
Request Body
The request body requires a time range and supports filtering by fields:
Parameters
Parameter
Type
Required
Description
Response
Response Schema
Field Descriptions
Field
Type
Description
Common Condition Types
Type
Description
Common Condition Reasons
Reason
Description
Examples
Basic Request
Get deployments for a specific time range:
Filter by Namespace
Get deployments from specific namespaces:
Response Example
Time Range Guidelines
Use ISO 8601 UTC format for timestamps
Typical time ranges: 1-24 hours for operational monitoring
Maximum recommended range: 7 days
The basics of querying logs
This document provides guidance on querying logs using the groundcover API. It outlines the necessary authentication steps, request structure, and examples to help you retrieve log data effectively.
Endpoint
POSThttps://app.groundcover.com/api/logs/v2/query
This endpoint allows you to execute queries to retrieve log data based on specified criteria.
Authentication
To authenticate your requests, include your API key in the Authorization header:
Authorization: Bearer <YOUR_API_KEY>
Example:
Request Body
The request body should be a JSON object containing the following parameters:
Parameter
Type
Description
Example
Note: The limit parameter can get a maximum value of 5000
Example Request Body:
Example Queries
Query Error Logs from Last 6 Hours
To retrieve logs with the level "error" from the last 6 hours, use the group parameter with filtering conditions:
Note: Ensure that the start and end times are adjusted to reflect the desired time range. The group parameter structure filters logs where the level field matches "error".
Group Parameter Structure
The group parameter uses the following structure for filtering:
conditions: Array of filter conditions
filters: Array of filter operations
op
To query all logs without filtering, set group to null.
Response
The response is a JSON object containing log entries and metadata. Each log entry includes details such as timestamp, log level, line content, workload information, and associated metadata.
Example Response:
Using Selectors
The selectors parameter allows you to specify additional fields and attributes to return with the logs. By default, logs include standard fields like timestamp, level, line, workload, namespace, cluster, etc. Use selectors to request additional fields.
Example: Query with selectors to get additional fields
Filtering with Group Parameter
The group parameter allows you to filter logs based on field values. Here are some examples:
Single condition (level = error):
Multiple conditions (using AND operator):
No filtering (query all logs):
Best Practices
Time Range: Always specify appropriate start and end times to limit the scope of your query and improve performance.
Result Limits: Use the limit parameter to control the number of results returned. For large datasets, consider using pagination (see the ).
Create a new Monitor
Learn how to create and configure monitors using the Wizard, Monitor Catalog, or Import options. The following guide will help you set up queries, thresholds, and alert routing for effective monitoring.
You can either create monitors using our web application following this guide, or use our API, see: or use our Terraform provider, see: .
In the Monitors section (left navigation bar), navigate to the Issues page or the Monitor List page to create a new Monitor. Click on the “Create Monitor” button at the top right and select one of the following options from the dropdown:
The Monitor Wizard is a guided, user-friendly approach to creating and configuring monitors tailored to your observability needs. By breaking down the process into simple steps, it ensures consistency and accuracy.
Section 1: Query
Select the data source, build the query and define thresholds for the monitor.
If you're unfamiliar with query building in groundcover, refer to the Query Builder section for full details on the different components.
Data Source (Required):
Select the type of data (Metrics, Logs, Traces, or Events).
Query Functions:
Choose how to process the data (e.g., average, count).
Add aggregation (group by) clauses if applicable, you MUST use aggregations if you want to add labels to your issues.
Important: The labels used for aggregation (group by) maybe also be used for notification routes and the issue summary & description.
Examples:cluster, node, container_name
Time Window (Required):
Specify the period over which data is aggregated (the look-behind window).
Example: “Over the last 5 minutes.”
Window Aggregation (Required):
Specify the aggregation function to be used on the selected time window.
Example: "avg over the last 5m"
Threshold Conditions (Required):
Define when a monitor should trigger an Issue. You can use:
Greater Than - Trigger when the value exceeds X.
Preview Settings (Optional):
Preview data using Stacked Bar or Line Chart for better clarity while building the monitor.
Choose the Y axis units.
Advanced (Optional):
Evaluation Interval:
Specify how often the monitor evaluates the query
Section 2: Monitor Details
Set up the basic information for the monitor.
Monitor Name (Required):
Add a title for the monitor. The title will appear in notifications and in the Monitor List page.
Give the Monitor a clear, short name, that describes its function at a high level.
Examples:
“Workload High API Error Rate”
“Node High Memory”
The title will appear in the monitors page table and be accessible in notification routes.
Severity (required):
Use severity to categorize alerts by importance.
Select a severity level (S1-S4).
Important: For connected apps (OpsGenie, PagerDuty) that require using specific Severities like P1-P4 or Critical-Info, we translate automatically to the relevant respective Severity.
Custom Labels (formally called 'metadata labels'):
Add custom labels (key:value) that will be added to all Issues generated by this monitor
Example: To create a notification route for my team's issues, add "Team:Infra" and use it in the notification route's scope
Section 3: Issue Details
Customize how the Monitor’s Issues will appear and what content will be sent in it's notifications. This section also includes a live preview of the way it will appear in the notification.
Ensure that the labels you wish to use dynamically (e.g., cluster, workload) or statically (e.g. team:infra) are defined in the query and monitor details.
Issue Summary (required):
Define a short title for issues that this Monitor will raise. It's useful to include variables that can be informative at first glance.
Example: Adding {{ labels.statusCode }} to the header will inject the status code to the name of the issue - this becomes especially useful when one Monitor raises multiple issues and you want to quickly understand their content without having to open each one.
“HTTP API Error {{ labels.status_code }}” -> HTTP API Error 500
“Workload {{ labels.workload }} Pod Restart” -> Workload frontend Pod Restart
Note: Autocomplete is supported to view what is usable in the Issue and will help ensure you put in the variables correctly.
The new format for templating variables is {{ variable }} or {{ labels.<label> }} , but the previous format {{ alert.labels.statusCode }} used for keep is still supported.
Description:
Used as the body of the message for the Issue.
The templating format is Jinja2, and can be used with variables similarly to the Issue Summary and various more advanced functions.
Example: Adding all the labels to be shown in the Slack message's body should be inserted into here using {{labels.<label>}} , you can add the severity {{severity}}, the monitor's name {{monitor_name}} and many more.
Any pongo2 functionality like { %if } evaluations are supported, which can be used to render different descriptions for different conditions.
Advanced (Optional)
Display Labels (formally called 'context labels'):
These Labels will be displayed and filterable in Monitors>Issues page.
This list gets automatically populated based on the labels used in the aggregation function in the Query.
Note: You can remove labels from this list if you do not wish to see them in the Issues page.
Section 4: Notifications
Set up notifications behavior for issues from this monitor
Workflows (used for Keep) and Notification Routes may work in parallel and do not affect each other.
Based on matching notification routes (Required)
The issues generated by this monitor will be evaluated by the Notification Route's scopes and rules and notifications will be sent accordingly
Note: The 'Preview' can be used in order to align expectations as for which notification routes may match this monitor's future issues
Routing (Workflows) (Optional)
Select Workflow:
Route alerts to existing workflows only, this means that other workflows will not process them. Use this to send alerts for a critical application such as Slack or PagerDuty.
Configure Routing for Keep Workflows only
Advanced (Optional)
Override Renotify Interval
Used to override the interval configured on the Notification Route for when a certain monitior's issue should send another notification at a different inerval.
Example: If it's set to 1m while the evaluation interval is 1m a notification will be sent with every firing evaluation. If it's set to 2d, even if the monitor evaluates every 1m, a notification will be sent once every 2 days.
Note: If the Issue stops firing and starts firing again, a new notification will be sent, this is not considered 'renotification'
Important: Minimum interval is the evaluation interval
Using the Import option
This is an advanced feature, please use it with caution.
In the "Import Bulk Monitors" you can add multiple monitors using an array of Monitors that follows the Monitor YAML structure.
Execute PromQL queries against groundcover metrics data. Two endpoints are available: instant queries for point-in-time values and range queries for time-series data over specific periods.
Endpoints
Instant Query
monitors:
- title: K8s Cluster High Memory Requests Monitor
display:
header: K8s Cluster High Memory Requests
description: Alerts when a K8s Cluster's total Container Memory Requests exceeds 90% of the Allocatable Memory of all the Nodes for 5 minutes
contextHeaderLabels:
- env
- cluster
severity: S1
measurementType: state
model:
queries:
- name: threshold_input_query
expression: avg_over_time( (((sum(groundcover_node_rt_mem_requests_bytes{}) by (cluster, env)) / (sum(groundcover_node_rt_allocatable_mem_bytes{}) by (cluster, env))) * 100)[5m] )
queryType: instant
datasourceType: prometheus
thresholds:
- name: threshold_1
inputName: threshold_input_query
operator: gt
values:
- 90
noDataState: OK
evaluationInterval:
interval: 1m
pendingFor: 0s
- title: K8s PVC Pending For 5 Minutes Monitor
display:
header: K8s PVC Pending Over 5 Minutes
description: This monitor triggers an alert when a PVC remains in a Pending state for more than 5 minutes.
contextHeaderLabels:
- cluster
- namespace
- persistentvolumeclaim
severity: S2
measurementType: state
model:
queries:
- name: threshold_input_query
expression: last_over_time(max(groundcover_kube_persistentvolumeclaim_status_phase{phase="Pending"}) by (cluster, namespace, persistentvolumeclaim)[1m])
queryType: instant
datasourceType: prometheus
thresholds:
- name: threshold_1
inputName: threshold_input_query
operator: gt
values:
- 0
executionErrorState: OK
noDataState: OK
evaluationInterval:
interval: 1m
pendingFor: 5m
Lower Than - Trigger when the value falls below X.
Within Range - Trigger when the value is between X and Y.
Outside Range - Trigger when the value is not between X and Y.
Important: The units in which the threshold is being measured in must be the same as the units the query uses.
For metrics queries the threshold should match the unit the metric is measured in.
For logs, traces, and events, it's just a number.
Example: “Trigger if disk space usage is greater than 10%.”
Choose the rollup to present.
Important: These configurations only affect the preview graph, not the monitor's evaluation.
Example: “Evaluate every 1 minute.”
Pending Period:
Specify how many times the evaluation needs to pass the threshold in order to trigger an Issue. This refers to a consecutive evaluations passing the threshold.
Monitors that have entered the pending period (the first evaluation passed the threshold) will be in 'Pending' state, only after all consecutive evaluations passed the threshold, the monitor will be 'Firing' and an Issue will be created. If even 1 of the evaluations did not pass the threshold, the Monitor will be set right back to 'Normal'.
Example: “When Evaluation Interval of 5m, setting this to 2 (10m) ensures the condition must be evaluated 3 times before a monitor will fire.
Evaluation #1 at 0m
Evaluation #2 at 5m
Evaluation #3 at 10m -> If all 3 passed the threshold, an the monitor will 'Fire'
Note: This ensures that transient conditions do not trigger alerts, reducing false positives or smoothing sudden unwanted spikes.
Important: The default configuration is 0, which means the monitor will trigger an Issue immediately when an evaluation was run and the threshold was passed.
Treat No Data As:
Whether the monitor should treat no data as an Issue or Normal behavior
Example: "I want to be notified if the metric has a gap for the entire look-behind window of the query, so I will set it to 'Firing'"
“{{ labels.team }} APIs High Latency” -> Infra APIs High Latency
No Routing:
This means that any workflow (without filters), will process the issue.
# Get current value (no time parameter)
# Returns: {"value":[1761040224,"18.45"]} - timestamp is "right now"
curl '...query=avg(groundcover_node_rt_disk_space_used_percent{})'
# Get historical value (with time parameter)
# Returns: {"value":[1761038504,"18.44"]} - timestamp is exactly what you specified
curl '...query=avg(groundcover_node_rt_disk_space_used_percent{})&time=2025-10-21T09:21:44.398Z'
While we strongly suggest building monitors using our Wizard or Catalog, groundcover supports building and editing your Monitors using YAML. If you choose to do so, the following will provide you the necessary definitions.
Monitor fields explained
In this section, you'll find a breakdown of the key fields used to define and configure Monitors within the groundcover platform. Each field plays a critical role in how a Monitor behaves, what data it tracks, and how it responds to specific conditions. Understanding these fields will help you set up effective Monitors to track performance, detect issues, and provide timely alerts.
Below is a detailed explanation of each field, along with examples to illustrate their usage, ensuring your team can manage and respond to incidents efficiently.
Field
Explanation
Example
Monitor YAML Examples
Traces Based Monitors
MySQL Query Errors Monitor
gRPC API Errors Monitor
Log Based Monitors
High Error Log Rate Monitor
Query Monitors Summary
Get a comprehensive list of monitor configurations with detailed execution status, alert states, performance metrics, and complete query definitions. This endpoint provides real-time monitoring data f
Endpoint
POST/api/monitors/summary/query
ResourceHeaderLabels
A list of labels that help you identify the resources that are related to the Monitor. This appear as a secondary header in all Issues tables across the platform.
["span_name", "kind"] for monitors on protocol issues.
ContextHeaderLabels
A list of contextual labels that help you identify the location of the issue. This appears as a subset of the Issue’s labels, and is displayed on all Issues tables across the platform.
["cluster", "namespace", "pod_name"]
Labels
A set of pre-defined labels that were set to Issues related to the selected Monitor. Labels can be static, or dynamic using a Monitor's query results.
team: sre_team
ExecutionErrorState
Defines the actions that take place when a Monitor encounters query execution errors.
Valid options are Alerting, OK and Error.
When Alerting is set, query execution errors will result in a firing issue.
NoDataState
This defines what happens when queries in the Monitor return empty datasets.
Valid options are: NoData , Alerting, OK
When NoData is set, monitor instance's state will be No Data.
Interval
Defines how frequently the Monitor evaluates the conditions. Common intervals could be 1m, 5m, etc.
PendingFor
Defines the period of consecutive intervals where threshold condition must be met to trigger the alert.
Trigger
Defines the condition under which the Monitor fires. This is the definition of threshold for the Monitor, with op - operator and value .
op: gt, value: 5
Model
Describes the queries, thresholds and data processing of the Monitor. It can have the following fields:
Queries: List of one or more queries to run, this can be either SQL over ClickHouse, PromQL over VictoriaMetrics, SqlPipeline. Each query will have a name for reference in the monitor.
Thresholds: This is the threshold of your Monitor, a threshold has a name, inputName for data input, operator one of gt , lt ,
measurementType
Describe how will we present issues of this Monitor. Some Monitors count events, and some a state. And we will display them differently in our dashboards.
state - Will present issues in line chart.
event - Will present issues in bar chart, counting events.
Title
A string that defines the human-readable name of the Monitor. The title is what you will see in the list of all existing Monitors in the Monitors section.
Description
Additional information about the Monitor.
Severity
When triggered, this will show the severity level of the Monitor's issue. You can set any severity you want here.
s1 for Critical
s2 for High
s3 for Medium
s4 for Low
Header
This is the header of the generated issues from the Monitor.
A short string describing the condition that is being monitored. You can also use this as a pattern using labels from you query.
“HTTP API Error {{ alert.labels.return_code}}”
Authentication
This endpoint requires API Key authentication via the Authorization header.
Headers
Request Body
The request body supports filtering, pagination, sorting, and time range parameters:
Request Parameters
Parameter
Type
Required
Description
conditions
array
No
Array of filter conditions for monitors (empty array returns all)
limit
integer
No
Maximum number of monitors to return (default: 200)
skip
integer
No
Sorting Options
Sort Field
Description
"lastFiringStart"
Last time monitor started firing alerts
"title"
Monitor title alphabetically
"severity"
Monitor severity level
"createdAt"
Monitor creation date
"updatedAt"
Last modification date
"state"
Current monitor state
Filter Conditions
The conditions array accepts filter objects for targeted monitor queries:
Response
The endpoint returns a JSON object containing an array of detailed monitor configurations:
is_loopback and remote_is_external are special labels that indicate the remote service is either the same service as the recording side (loopback) or resides in an external network, e.g managed service outside of the cluster (external).
In both cases the remote_service_name and the remote_namespace labels will be empty
In the lists below, we describe error and issue counters. Every issue flagged by the platform is an error; but not every error is flagged as an issue.
Resource metrics
Name
Description
Unit
Type
Workload metrics
Name
Description
Unit
Type
Kafka specific metrics
Name
Description
Unit
Type
groundcover_node_used_disk_space
Current used disk space in the current node
Bytes
Gauge
groundcover_node_free_disk_space
Free disk space in the current node
Bytes
Gauge
groundcover_node_total_disk_space
Total disk space in the current node
Bytes
Gauge
groundcover_node_used_percent_disk_space
Percentage of used disk space in the current node
Percentage
Gauge
groundcover_pvc_usage_percent
Percentage of used Persistent Volume Claim (PVC) storage
Percentage
Gauge
groundcover_pvc_read_bytes_total
Total bytes read by the workload from the Persistent Volume Claim (PVC)
Bytes
Counter
groundcover_pvc_write_bytes_total
Total bytes written by the workload to the Persistent Volume Claim (PVC)
Bytes
Counter
groundcover_pvc_reads_total
Total read operations performed by the workload from the Persistent Volume Claim (PVC)
Number
Counter
groundcover_pvc_writes_total
Total write operations performed by the workload to the Persistent Volume Claim (PVC)
Number
Counter
groundcover_pvc_read_latency
Latency of read operations from the Persistent Volume Claim (PVC) by the workload
Seconds
Summary
groundcover_pvc_write_latency
Latency of write operations to the Persistent Volume Claim (PVC) by the workload
Seconds
Summary
groundcover_pvc_read_latency_count
Count of read operations latency for the Persistent Volume Claim (PVC)
Number
Counter
groundcover_pvc_read_latency_sum
Sum of read operation latencies for the Persistent Volume Claim (PVC)
Seconds
Counter
groundcover_pvc_read_latency_summary
Summary of read operations latency for the Persistent Volume Claim (PVC)
Milliseconds
Counter
groundcover_pvc_write_latency_count
Count of write operations sampled for latency on the Persistent Volume Claim (PVC)
Number
Counter
groundcover_pvc_write_latency_sum
Sum of write operation latencies for the Persistent Volume Claim (PVC)
Seconds
Counter
groundcover_pvc_write_latency_summary
Summary of write operations latency for the Persistent Volume Claim (PVC)
Milliseconds
Counter
is_cross_az
protocol
role
server_port
encryption
transport_protocol
is_loopback
is_cross_az means the traffic was sent and/or received between two different availability zones. This is a helpful flag to quickly identify this special kind of communication.
The actual zones are detailed in the availability_zone and remote_availability_zone labels
The number of requested limit resource by a container. It is recommended to use the `kube_pod_resource_limits` metric exposed by kube-scheduler instead, as it is more precise.
Number
Gauge
groundcover_kube_pod_container_resource_requests
The number of requested request resource by a container. It is recommended to use the `kube_pod_resource_requests` metric exposed by kube-scheduler instead, as it is more precise.
