ClickHouse Database Monitoring

Collect ClickHouse health metrics and query-performance traces without running a separate exporter

groundcover connects directly to a ClickHouse server to collect built-in health metrics and query statistics without requiring a separate Prometheus exporter. You can use the health metrics in dashboards and monitors and inspect query and materialized-view statistics as traces.

Prerequisites

The integration must run from a groundcover backend or monitored cluster that can reach the ClickHouse host and port.

A cluster-run integration collects through the integrations agent in the selected monitored cluster, which requires groundcover sensor chart version 1.12.290 or later.

As a ClickHouse administrator, create a dedicated monitoring user and grant read access to ClickHouse system tables:

CREATE USER groundcover IDENTIFIED BY '<password>';
GRANT SELECT ON system.* TO groundcover;

For a ClickHouse cluster, also complete the Cluster mode preparation before saving the integration. The user-creation and grant commands above affect only the node where they are run.

The integration checks available server capabilities while collecting. A missing table, column, or grant can make only the affected health group unavailable while other groups continue collecting.

What it collects

Signal	What it contains	Where to use it
Built-in health metrics	ClickHouse engine health and capacity under the groundcover_db_ch_* prefix	Data Explorer, monitors, and the ClickHouse dashboard from the Dashboard Catalog
Query statistics	Rows from system.query_log, represented as traces with database semantic-convention attributes	Traces
Materialized-view statistics	Rows from system.query_views_log, represented as traces	Traces

Materialized-view statistics describe the work ClickHouse performs when executing a materialized view. groundcover collects these records from system.query_views_log as traces, showing execution time, rows processed, memory usage, and errors.

Query-statistics collection is independent of health-metric collection:

• query_log produces one span per query. If the querying application sets a JSON log_comment containing hex-encoded trace.id and span.parent.id values, groundcover joins the database span to the application trace that issued it. Otherwise, the span starts a new trace.

• query_views_log produces one span per materialized-view execution. These spans start new traces and are not stitched through log_comment.

See Verify collection for copyable metric queries and trace filters.

Health metric groups

All health groups are enabled by default when built-in health metrics are enabled. You can disable individual groups in the app or configuration.

Group	Description	Availability and empty results
Parts	Part counts, sizes, row counts and merge depth for the largest tables, so a merge backlog shows up before queries slow down.	Applies to tables backed by parts; limited to the configured top tables.
Merges	Merges currently running, how long the longest has been going and how far it has progressed.	Empty when no merge is active.
Mutations	Pending and failed mutations from ALTER ... UPDATE and ALTER ... DELETE operations.	Empty when no mutation is queued or failed.
Replication	Replica count and availability, replication delay, and queued inserts and merges per replicated table.	Empty for tables that do not use replication.
Connections	Current client connections to the server.	Depends on the connection counters exposed by the server.
Capacity	Free and total disk space, plus selected server metrics from system.asynchronous_metrics and system.metrics.	Disk metrics are always queried; optional metric names use curated defaults unless overridden.
Events	Selected cumulative counters from system.events, such as queries executed, failed queries and cache activity.	Uses a curated event set unless an explicit include list is provided.
Detached parts	Parts the server has set aside and will neither merge nor query, usually after corruption or a failed operation.	Empty when no detached parts exist.
Table sizes	On-disk size and row count for the largest tables on the instance.	Limited to the configured top tables.
View refreshes	Status, time since last success and rows written for refreshable materialized views.	Skipped when the required system-table schema is unavailable; empty when no refreshable view exists.
Dictionaries	Load status, item count and memory used by each dictionary.	Empty when no dictionaries are configured.

Set it up

In the app (recommended)

Open Data Sources, select ClickHouse, and complete the wizard:

Configure the ClickHouse connection in the Data Sources wizard.

1. Prepare ClickHouse — create the monitoring user and apply the grants in Prerequisites.

2. Connection — enter a name, host, optional port and database, choose Native or HTTP, and configure TLS.

3. Cluster scoping — run the integration from the backend or from a monitored cluster that can reach the database.

4. Authentication — provide the monitoring user's basic-auth credentials and enable TLS under Connection. Select no authentication only when the database is deliberately configured for passwordless access and network controls restrict access to trusted clients, including the integration's execution location. Never expose an unauthenticated database endpoint to the public internet.

5. Cluster settings — enable ClickHouse cluster mode and provide the ClickHouse cluster name when collection must cover multiple nodes.

6. Collection — choose query-log and materialized-view collection, INSERT filtering, the shared scrape interval, and health-metric groups.

7. Ingestion settings — add labels such as environment or team to every collected metric.

Save the integration, then use Verify collection to confirm both metrics and query traces are arriving.

Terraform

Use the canonical Terraform examples, selecting the clickhouse_dbm resource with type = "clickhousedbm". Follow the provider configuration guide and groundcover_dataintegration reference.

Replace the example endpoint, database, username, and integration name with your values. Configure the execution cluster and password reference as described in Credentials and execution location, then review terraform plan before applying.

Crossplane

Use the canonical ClickHouse DataIntegration manifest. Follow the provider setup instructions to install the provider and configure its credentials.

Replace the example endpoint, database, username, integration name, and providerConfigRef with your values. Configure the execution cluster and password reference as described in Credentials and execution location, then apply the manifest to the Kubernetes cluster running Crossplane.

Configuration notes

Credentials and execution location

Choose an execution location with network access to the ClickHouse endpoint:

• A cluster-run integration uses the integrations agent in the selected monitored cluster. Use secretRef::k8s::<namespace>::<secret-name>::<key> to read an existing Kubernetes Secret from that cluster.

• A backend-run integration uses the integrations agent on the groundcover backend. Store the password through the app or use secretRef::store::<id> in API-managed configuration.

Create a Kubernetes Secret

For a cluster-run integration, store the monitoring user's database password in a local file using your secret-management workflow. Restrict access to the file and ensure it contains only the password, with no trailing newline. Create the Secret in the cluster selected in Cluster scoping:

kubectl --context "<execution-cluster-context>" --namespace groundcover \
  create secret generic groundcover-clickhouse \
  --from-file=admin-password=/path/to/password-file

The namespace must already exist. This creates a Secret named groundcover-clickhouse with a data key named admin-password. Reference it in the integration's authentication.basicAuth.password field:

secretRef::k8s::groundcover::groundcover-clickhouse::admin-password

Replace the context, namespace, Secret name, and key to match your environment. Use the dedicated monitoring username created in Prerequisites.

Backend-run credentials

For a backend-run integration, select Run the integration from the backend and enter the monitoring username and password under Authentication in the app wizard. Saving the integration stores the password as a secret. A secretRef::store::<id> reference identifies a secret in that store; use an existing secret's ID for IaC configuration. Omit cluster for backend execution.

Never put a plaintext production password in Terraform or Crossplane configuration.

Protocol and TLS

Choose the protocol exposed by the ClickHouse endpoint:

Protocol	Conventional non-TLS port	Conventional TLS port
Native	9000	9440
HTTP	8123	8443

Enable TLS for encrypted connections. Keep certificate verification enabled whenever the endpoint presents a certificate trusted by the integration's runtime. Use the skip-verification option only when you understand and accept the risk of an internal or self-signed certificate chain.

Cluster mode

Enable clusterMode when the integration should query all nodes through ClickHouse's cluster() table function. Set clusterName to the server-side cluster definition; when omitted in cluster mode, it resolves to default. The monitoring user needs the REMOTE grant in addition to read access on system tables.

Before enabling cluster mode:

1. Run the user-creation and grant commands from Prerequisites on every cluster member, using the same monitoring credentials. If a user already exists, verify its credentials and grants instead of recreating it. You can also use your existing centralized user-management workflow to provision the equivalent access across the cluster.

2. On each member where the monitoring user will initiate cluster queries, apply the cluster-only grant below.

   -- Cluster mode only
   GRANT REMOTE ON *.* TO groundcover;

3. Verify that the initiating server can reach the other members and that the credentials used by its cluster definition are valid there. If that definition specifies a different remote user, give that user the same read access to system tables on the destination nodes.

Single-node collection does not need the REMOTE grant.

Health-metric limits

healthMetrics.topN controls the default number of tables returned by the Parts and Table sizes groups. Capacity and Events use curated metric and event lists when their include settings are omitted. Use explicit include lists only when you need to narrow or extend those defaults.

Database exclusions

ignoreDatabases is a top-level field inside the integration's config. It is available through Terraform and Crossplane configuration, but has no control in the ClickHouse app wizard. Add or update it in the Terraform resource's config = jsonencode({ ... }) object, or in the JSON string at spec.forProvider.config in a Crossplane manifest.

For example, merge this field into the existing configuration to exclude the listed database names from health queries that operate on individual databases or tables:

{
  "ignoreDatabases": ["system", "information_schema", "scratch"]
}

Apply the updated configuration through your IaC workflow. This field does not filter system.query_log or system.query_views_log traces, or server-wide health metrics such as Connections and Capacity.

Verify collection

1. Open the integration from Data Sources.

2. Use Stats to confirm that ClickHouse metrics are arriving and Activity Traces to inspect collection attempts and errors.

3. In Data Explorer, query the built-in metrics for this integration:

   {__name__=~"groundcover_db_ch_.*", gc_integration_name="<integration-name>"}

4. Install the ClickHouse dashboard from the Dashboard Catalog and select the integration or server endpoint.

5. In Traces, choose a time range covering recent database activity. Filter by gc_integration_name using the Name you entered in the wizard (or config.name in IaC), and by source.table to find the collected database statements:

   gc_integration_name:"<integration-name>" source.table:"system.query_log"

   For materialized-view statistics, use:

   gc_integration_name:"<integration-name>" source.table:"system.query_views_log"

   Open a matching trace to inspect its SQL statement and database statistics. These are the collected query-statistics traces; Activity Traces in the data-source drawer show the integration's collection attempts.

Troubleshooting

The integration cannot connect

Confirm that the selected execution location can resolve and reach the host and port. Then verify that the configured protocol matches the endpoint, and check the username, secret reference, TLS mode, and certificate trust. Use the most recent Activity Traces entry to inspect reported network, authentication, and query failures.

Health metrics are missing

• For a cluster-run integration, confirm that the cluster's groundcover sensor is on chart version 1.12.290 or later. Earlier versions collect query traces but not health metrics, so the integration looks healthy while the dashboard stays empty.

• Edit the integration and confirm that Collect health metrics and the expected groups are enabled.

• Confirm that the user can read the required system tables using the grant in Prerequisites. In cluster mode, also verify the REMOTE grant and remote-node credentials.

• Check whether the group applies to the monitored data. Merges, mutations, replication, detached parts, view refreshes, and dictionaries can legitimately return no rows.

• Check the configured topN, database exclusions (config.ignoreDatabases), and Capacity or Events include lists. Database exclusions are configured through IaC, not the ClickHouse wizard.

• Check Stats, Activity Traces, and the MetricsQL selector above. The product does not currently expose a persisted per-group status view.

Query traces are missing

Confirm that query-log collection is enabled and that system.query_log contains recent finished queries. For materialized views, also confirm that collection is enabled and system.query_views_log contains rows. Check whether INSERT filtering excludes the workload you expect.

Cluster-mode queries fail

Confirm that clusterName matches a cluster defined on the ClickHouse server, that every member is reachable from the initiating server, and that the monitoring user has REMOTE plus the required system-table permissions across the cluster.

For general integration diagnostics, see Monitor Data Sources Integrations.

Related documentation

• PostgreSQL Database Monitoring

• Monitor Data Sources Integrations

• Dashboard Catalog

• groundcover Terraform Provider