> For the complete documentation index, see [llms.txt](https://docs.groundcover.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.groundcover.com/collect-data/data-sources/clickhouse-database-monitoring.md). # ClickHouse Database Monitoring Collect ClickHouse health metrics and query-performance traces without running a separate exporter groundcover connects directly to a ClickHouse server to collect built-in health metrics and query statistics without requiring a separate Prometheus exporter. You can use the health metrics in dashboards and monitors and inspect query and materialized-view statistics as traces. ## Prerequisites The integration must run from a groundcover backend or monitored cluster that can reach the ClickHouse host and port. A cluster-run integration collects through the integrations agent in the selected monitored cluster, which requires groundcover sensor chart version `1.12.290` or later. As a ClickHouse administrator, create a dedicated monitoring user and grant read access to ClickHouse system tables: ```sql CREATE USER groundcover IDENTIFIED BY ''; GRANT SELECT ON system.* TO groundcover; GRANT SHOW TABLES ON *.* TO groundcover; GRANT SHOW DICTIONARIES ON *.* TO groundcover; ``` The `SHOW TABLES` and `SHOW DICTIONARIES` grants allow table and dictionary discovery across databases. Replace `groundcover` with your integration username throughout the example. For a ClickHouse cluster, also complete the [Cluster mode](#cluster-mode) preparation before saving the integration. The user-creation and grant commands above affect only the node where they are run. {% hint style="info" %} The integration checks available server capabilities while collecting. A missing table, column, or grant can make only the affected health group unavailable while other groups continue collecting. {% endhint %} ## What it collects | Signal | What it contains | Where to use it | | ---------------------------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Built-in health metrics | ClickHouse engine health and capacity under the `groundcover_db_ch_*` prefix | [Data Explorer](https://app.groundcover.com/explore/data-explorer), monitors, and the ClickHouse dashboard from the [Dashboard Catalog](/investigate/dashboards-and-alerts/dashboard-catalog.md) | | Query statistics | Rows from `system.query_log`, represented as traces with database semantic-convention attributes | [Traces](https://app.groundcover.com/traces) | | Materialized-view statistics | Rows from `system.query_views_log`, represented as traces | [Traces](https://app.groundcover.com/traces) | Materialized-view statistics describe the work ClickHouse performs when executing a materialized view. groundcover collects these records from `system.query_views_log` as traces, showing execution time, rows processed, memory usage, and errors. Query-statistics collection is independent of health-metric collection: * **`query_log`** produces one span per query. If the querying application sets a JSON `log_comment` containing hex-encoded `trace.id` and `span.parent.id` values, groundcover joins the database span to the application trace that issued it. Otherwise, the span starts a new trace. * **`query_views_log`** produces one span per materialized-view execution. These spans start new traces and are not stitched through `log_comment`. See [Verify collection](#verify-collection) for copyable metric queries and trace filters. ### Health metric groups All health groups are enabled by default when built-in health metrics are enabled. You can disable individual groups in the app or configuration. | Group | Description | Availability and empty results | | -------------- | ---------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | | Parts | Part counts, sizes, row counts and merge depth for the largest tables, so a merge backlog shows up before queries slow down. | Applies to tables backed by parts; limited to the configured top tables. | | Merges | Merges currently running, how long the longest has been going and how far it has progressed. | Empty when no merge is active. | | Mutations | Pending and failed mutations from ALTER ... UPDATE and ALTER ... DELETE operations. | Empty when no mutation is queued or failed. | | Replication | Replica count and availability, replication delay, and queued inserts and merges per replicated table. | Empty for tables that do not use replication. | | Connections | Current client connections to the server. | Depends on the connection counters exposed by the server. | | Capacity | Free and total disk space, plus selected server metrics from system.asynchronous\_metrics and system.metrics. | Disk metrics are always queried; optional metric names use curated defaults unless overridden. | | Events | Selected cumulative counters from system.events, such as queries executed, failed queries and cache activity. | Uses a curated event set unless an explicit include list is provided. | | Detached parts | Parts the server has set aside and will neither merge nor query, usually after corruption or a failed operation. | Empty when no detached parts exist. | | Table sizes | On-disk size and row count for the largest tables on the instance. | Limited to the configured top tables. | | View refreshes | Status, time since last success and rows written for refreshable materialized views. | Skipped when the required system-table schema is unavailable; empty when no refreshable view exists. | | Dictionaries | Load status, item count and memory used by each dictionary. | Empty when no dictionaries are configured. | ## Set it up {% tabs %} {% tab title="In the app (recommended)" %} Open [Data Sources](https://app.groundcover.com/data-sources), select **ClickHouse**, and complete the wizard:
ClickHouse setup wizard showing connection settings

Configure the ClickHouse connection in the Data Sources wizard.

1. **Prepare ClickHouse** — create the monitoring user and apply the grants in [Prerequisites](#prerequisites). 2. **Connection** — enter a name, host, optional port and database, choose Native or HTTP, and configure TLS. 3. **Cluster scoping** — run the integration from the backend or from a monitored cluster that can reach the database. 4. **Authentication** — provide the monitoring user's basic-auth credentials and enable TLS under **Connection**. Select no authentication only when the database is deliberately configured for passwordless access and network controls restrict access to trusted clients, including the integration's execution location. Never expose an unauthenticated database endpoint to the public internet. 5. **Cluster settings** — enable ClickHouse cluster mode and provide the ClickHouse cluster name when collection must cover multiple nodes. 6. **Collection** — choose query-log and materialized-view collection, INSERT filtering, the shared scrape interval, and health-metric groups. 7. **Ingestion settings** — add labels such as `environment` or `team` to every collected metric. Save the integration, then use [Verify collection](#verify-collection) to confirm both metrics and query traces are arriving. {% endtab %} {% tab title="Terraform" %} Use the [canonical Terraform examples](https://github.com/groundcover-com/terraform-provider-groundcover/blob/main/examples/resources/groundcover_dataintegration/resource.tf), selecting the `clickhouse_dbm` resource with `type = "clickhousedbm"`. Follow the [provider configuration guide](https://registry.terraform.io/providers/groundcover-com/groundcover/latest/docs) and [`groundcover_dataintegration` reference](https://registry.terraform.io/providers/groundcover-com/groundcover/latest/docs/resources/dataintegration). Replace the example endpoint, database, username, and integration name with your values. Configure the execution cluster and password reference as described in [Credentials and execution location](#credentials-and-execution-location), then review `terraform plan` before applying. {% endtab %} {% tab title="Crossplane" %} Use the [canonical ClickHouse `DataIntegration` manifest](https://github.com/groundcover-com/crossplane-provider-groundcover/blob/main/examples/dataintegration-clickhouse.yaml). Follow the [provider setup instructions](https://github.com/groundcover-com/crossplane-provider-groundcover#readme) to install the provider and configure its credentials. Replace the example endpoint, database, username, integration name, and `providerConfigRef` with your values. Configure the execution cluster and password reference as described in [Credentials and execution location](#credentials-and-execution-location), then apply the manifest to the Kubernetes cluster running Crossplane. {% endtab %} {% endtabs %} ## Configuration notes ### Credentials and execution location Choose an execution location with network access to the ClickHouse endpoint: * A cluster-run integration uses the integrations agent in the selected monitored cluster. Use `secretRef::k8s::::::` to read an existing Kubernetes Secret from that cluster. * A backend-run integration uses the integrations agent on the groundcover backend. Store the password through the app or use `secretRef::store::` in API-managed configuration. #### Create a Kubernetes Secret For a cluster-run integration, store the monitoring user's database password in a local file using your secret-management workflow. Restrict access to the file and ensure it contains only the password, with no trailing newline. Create the Secret in the cluster selected in **Cluster scoping**: ```bash kubectl --context "" --namespace groundcover \ create secret generic groundcover-clickhouse \ --from-file=admin-password=/path/to/password-file ``` The namespace must already exist. This creates a Secret named `groundcover-clickhouse` with a data key named `admin-password`. Reference it in the integration's `authentication.basicAuth.password` field: ``` secretRef::k8s::groundcover::groundcover-clickhouse::admin-password ``` Replace the context, namespace, Secret name, and key to match your environment. Use the dedicated monitoring username created in [Prerequisites](#prerequisites). #### Backend-run credentials For a backend-run integration, select **Run the integration from the backend** and enter the monitoring username and password under **Authentication** in the app wizard. Saving the integration stores the password as a secret. A `secretRef::store::` reference identifies a secret in that store; use an existing secret's ID for IaC configuration. Omit `cluster` for backend execution. Never put a plaintext production password in Terraform or Crossplane configuration. ### Protocol and TLS Choose the protocol exposed by the ClickHouse endpoint: | Protocol | Conventional non-TLS port | Conventional TLS port | | -------- | ------------------------: | --------------------: | | Native | `9000` | `9440` | | HTTP | `8123` | `8443` | Enable TLS for encrypted connections. Keep certificate verification enabled whenever the endpoint presents a certificate trusted by the integration's runtime. Use the skip-verification option only when you understand and accept the risk of an internal or self-signed certificate chain. ### Cluster mode Enable `clusterMode` when the integration should query all nodes through ClickHouse's `cluster()` table function. Set `clusterName` to the server-side cluster definition; when omitted in cluster mode, it resolves to `default`. The monitoring user needs the `REMOTE` grant in addition to read access on system tables. Before enabling cluster mode: 1. Run the user-creation and grant commands from [Prerequisites](#prerequisites) on every cluster member, using the same monitoring credentials. If a user already exists, verify its credentials and grants instead of recreating it. You can also use your existing centralized user-management workflow to provision the equivalent access across the cluster. 2. On each member where the monitoring user will initiate cluster queries, apply the cluster-only grant below. ```sql -- Cluster mode only GRANT REMOTE ON *.* TO groundcover; ``` 3. Verify that the initiating server can reach the other members and that the credentials used by its cluster definition are valid there. If that definition specifies a different remote user, give that user the same read access to system tables on the destination nodes. Single-node collection does not need the `REMOTE` grant. ### Health-metric limits `healthMetrics.topN` controls the default number of tables returned by the Parts and Table sizes groups. Capacity and Events use curated metric and event lists when their `include` settings are omitted. Use explicit include lists only when you need to narrow or extend those defaults. ### Database exclusions `ignoreDatabases` is a top-level field inside the integration's `config`. It is available through Terraform and Crossplane configuration, but has no control in the ClickHouse app wizard. Add or update it in the Terraform resource's `config = jsonencode({ ... })` object, or in the JSON string at `spec.forProvider.config` in a Crossplane manifest. For example, merge this field into the existing configuration to exclude the listed database names from health queries that operate on individual databases or tables: ```json { "ignoreDatabases": ["system", "information_schema", "scratch"] } ``` Apply the updated configuration through your IaC workflow. This field does not filter `system.query_log` or `system.query_views_log` traces, or server-wide health metrics such as Connections and Capacity. ## Verify collection 1. Open the integration from [Data Sources](https://app.groundcover.com/data-sources). 2. Use **Stats** to confirm that ClickHouse metrics are arriving and **Activity Traces** to inspect collection attempts and errors. 3. In [Data Explorer](https://app.groundcover.com/explore/data-explorer), query the built-in metrics for this integration: ```promql {__name__=~"groundcover_db_ch_.*", gc_integration_name=""} ``` 4. Install the ClickHouse dashboard from the [Dashboard Catalog](/investigate/dashboards-and-alerts/dashboard-catalog.md) and select the integration or server endpoint. 5. In [Traces](https://app.groundcover.com/traces), choose a time range covering recent database activity. Filter by `gc_integration_name` using the **Name** you entered in the wizard (or `config.name` in IaC), and by `source.table` to find the collected database statements: ``` gc_integration_name:"" source.table:"system.query_log" ``` For materialized-view statistics, use: ``` gc_integration_name:"" source.table:"system.query_views_log" ``` Open a matching trace to inspect its SQL statement and database statistics. These are the collected query-statistics traces; **Activity Traces** in the data-source drawer show the integration's collection attempts. ## Troubleshooting ### The integration cannot connect Confirm that the selected execution location can resolve and reach the host and port. Then verify that the configured protocol matches the endpoint, and check the username, secret reference, TLS mode, and certificate trust. Use the most recent **Activity Traces** entry to inspect reported network, authentication, and query failures. ### Health metrics are missing * For a cluster-run integration, confirm that the cluster's groundcover sensor is on chart version `1.12.290` or later. Earlier versions collect query traces but not health metrics, so the integration looks healthy while the dashboard stays empty. * Edit the integration and confirm that **Collect health metrics** and the expected groups are enabled. * Confirm that the user can read the required system tables using the grant in [Prerequisites](#prerequisites). In cluster mode, also verify the [`REMOTE` grant and remote-node credentials](#cluster-mode). * Check whether the group applies to the monitored data. Merges, mutations, replication, detached parts, view refreshes, and dictionaries can legitimately return no rows. * Check the configured `topN`, [database exclusions (`config.ignoreDatabases`)](#database-exclusions), and Capacity or Events include lists. Database exclusions are configured through IaC, not the ClickHouse wizard. * Check **Stats**, **Activity Traces**, and the MetricsQL selector above. The product does not currently expose a persisted per-group status view. ### Query traces are missing Confirm that query-log collection is enabled and that `system.query_log` contains recent finished queries. For materialized views, also confirm that collection is enabled and `system.query_views_log` contains rows. Check whether INSERT filtering excludes the workload you expect. ### Cluster-mode queries fail Confirm that `clusterName` matches a cluster defined on the ClickHouse server, that every member is reachable from the initiating server, and that the monitoring user has `REMOTE` plus the required system-table permissions across the cluster. For general integration diagnostics, see [Monitor Data Sources Integrations](/collect-data/monitor-data-sources-integrations.md). ## Related documentation * [PostgreSQL Database Monitoring](/collect-data/data-sources/postgresql-database-monitoring.md) * [Monitor Data Sources Integrations](/collect-data/monitor-data-sources-integrations.md) * [Dashboard Catalog](/investigate/dashboards-and-alerts/dashboard-catalog.md) * [groundcover Terraform Provider](/reference/groundcover-terraform-provider.md) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.groundcover.com/collect-data/data-sources/clickhouse-database-monitoring.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `build a script that syncs our docs to a CMS` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.