> For the complete documentation index, see [llms.txt](https://docs.groundcover.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.groundcover.com/~/revisions/ETrLpNk6KtHjyaVUTLoE/use-groundcover/metrics-and-labels.md).

# Metrics & Labels

## Infrastructure Metrics & Labels

### <mark style="color:blue;">Container CPU and Memory</mark>

#### **Labels**

**`type`**

**`clusterId`** **`region`** **`namespace`** **`node_name`** **`workload_name`**

**`pod_name`** **`container_name`** **`container_image`**

#### **Metrics**

<table><thead><tr><th width="389.3333333333333">Name</th><th width="160">Description</th><th width="107">Unit</th><th>Type<select><option value="c476a927c24d4e67b8115feb2e039751" label="Counter" color="blue"></option><option value="655b3932f5bd49b0aacc43978ad78425" label="Gauge" color="blue"></option><option value="c0106f6abc0244adbf70f29447aa3875" label="Summary" color="blue"></option></select></th></tr></thead><tbody><tr><td>groundcover_container_cpu_usage_rate_millis</td><td>CPU usage in mCPU</td><td>mCPU</td><td><span data-option="655b3932f5bd49b0aacc43978ad78425">Gauge</span></td></tr><tr><td>groundcover_container_cpu_request_m_cpu</td><td>K8s container CPU request </td><td>mCPU</td><td><span data-option="655b3932f5bd49b0aacc43978ad78425">Gauge</span></td></tr><tr><td>groundcover_container_cpu_limit_m_cpu</td><td>K8s container CPU limit</td><td>mCPU</td><td><span data-option="655b3932f5bd49b0aacc43978ad78425">Gauge</span></td></tr><tr><td>groundcover_container_memory_working_set_bytes</td><td>current memory working set</td><td>Bytes</td><td><span data-option="655b3932f5bd49b0aacc43978ad78425">Gauge</span></td></tr><tr><td>groundcover_container_memory_rss_bytes</td><td>current memory RSS</td><td>Bytes</td><td><span data-option="655b3932f5bd49b0aacc43978ad78425">Gauge</span></td></tr><tr><td>groundcover_container_memory_request_bytes</td><td>K8s container memory request</td><td>Bytes</td><td><span data-option="655b3932f5bd49b0aacc43978ad78425">Gauge</span></td></tr><tr><td>groundcover_container_memory_limit_bytes</td><td>K8s container memory limit</td><td>Bytes</td><td><span data-option="655b3932f5bd49b0aacc43978ad78425">Gauge</span></td></tr><tr><td>groundcover_container_cpu_delay_seconds</td><td>K8s container CPU delay</td><td>Seconds</td><td><span data-option="c476a927c24d4e67b8115feb2e039751">Counter</span></td></tr><tr><td>groundcover_container_disk_delay_seconds</td><td>K8s container disk delay</td><td>Seconds</td><td><span data-option="c476a927c24d4e67b8115feb2e039751">Counter</span></td></tr><tr><td>groundcover_container_cpu_throttled_seconds_total</td><td>K8s container total CPU throttling</td><td>Seconds</td><td><span data-option="c476a927c24d4e67b8115feb2e039751">Counter</span></td></tr></tbody></table>

### <mark style="color:blue;">Node CPU, Memory and Disk</mark>

#### Labels

**`type`**  **`clusterId`** **`region`** **`node_name`**&#x20;

#### **Metrics**

<table><thead><tr><th width="367.3333333333333">Name</th><th width="193">Description</th><th width="95">Unit</th><th>Type<select><option value="c476a927c24d4e67b8115feb2e039751" label="Counter" color="blue"></option><option value="655b3932f5bd49b0aacc43978ad78425" label="Gauge" color="blue"></option><option value="c0106f6abc0244adbf70f29447aa3875" label="Summary" color="blue"></option></select></th></tr></thead><tbody><tr><td>groundcover_node_allocatable_cpum_cpu</td><td>amount of allocatable CPU in the current node</td><td>mCPU</td><td><span data-option="655b3932f5bd49b0aacc43978ad78425">Gauge</span></td></tr><tr><td>groundcover_node_allocatable_mem_bytes</td><td>amount of allocatable memory in the current node</td><td>Bytes</td><td><span data-option="655b3932f5bd49b0aacc43978ad78425">Gauge</span></td></tr><tr><td>groundcover_node_mem_used_percent</td><td>percent of used memory in current node</td><td>0-100</td><td><span data-option="655b3932f5bd49b0aacc43978ad78425">Gauge</span></td></tr><tr><td>groundcover_node_used_disk_space</td><td>current used disk space in current node</td><td>Bytes</td><td><span data-option="655b3932f5bd49b0aacc43978ad78425">Gauge</span></td></tr><tr><td>groundcover_node_free_disk_space</td><td>amount of free disk space in current node</td><td>Bytes</td><td><span data-option="655b3932f5bd49b0aacc43978ad78425">Gauge</span></td></tr><tr><td>groundcover_node_total_disk_space</td><td>amount of total disk space in current node</td><td>Bytes</td><td><span data-option="655b3932f5bd49b0aacc43978ad78425">Gauge</span></td></tr><tr><td>groundcover_node_used_percent_disk_space</td><td>percent of used disk space in current node</td><td>0-100</td><td><span data-option="655b3932f5bd49b0aacc43978ad78425">Gauge</span></td></tr></tbody></table>

### <mark style="color:blue;">Storage Usage</mark>

#### Labels

**`type`**  **`clusterId`** **`region`** **`name`**  **`namespace`**

#### Metrics

<table><thead><tr><th width="314.3333333333333">Name</th><th width="169">Description</th><th width="109">Unit</th><th>Type<select><option value="c476a927c24d4e67b8115feb2e039751" label="Counter" color="blue"></option><option value="655b3932f5bd49b0aacc43978ad78425" label="Gauge" color="blue"></option><option value="c0106f6abc0244adbf70f29447aa3875" label="Summary" color="blue"></option></select></th></tr></thead><tbody><tr><td>groundcover_pvc_usage_bytes</td><td>PVC usage</td><td>Bytes</td><td><span data-option="655b3932f5bd49b0aacc43978ad78425">Gauge</span></td></tr><tr><td>groundcover_pvc_capacity_bytes</td><td>PVC capacity</td><td>Bytes</td><td><span data-option="655b3932f5bd49b0aacc43978ad78425">Gauge</span></td></tr><tr><td>groundcover_pvc_available_bytes</td><td>PVC available</td><td>Bytes</td><td><span data-option="655b3932f5bd49b0aacc43978ad78425">Gauge</span></td></tr><tr><td>groundcover_pvc_usage_percent</td><td>percent of used PVC storage</td><td>0-100</td><td><span data-option="655b3932f5bd49b0aacc43978ad78425">Gauge</span></td></tr><tr><td>groundcover_pvc_read_bytes_total</td><td>total amount of bytes read by the workload from the PVC</td><td>Bytes</td><td><span data-option="c476a927c24d4e67b8115feb2e039751">Counter</span></td></tr><tr><td>groundcover_pvc_write_bytes_total</td><td>total amount of bytes written by the workload to the PVC</td><td>Bytes</td><td><span data-option="c476a927c24d4e67b8115feb2e039751">Counter</span></td></tr><tr><td>groundcover_pvc_reads_total</td><td>total amount of read operations done by the workload from the PVC</td><td>Number</td><td><span data-option="c476a927c24d4e67b8115feb2e039751">Counter</span></td></tr><tr><td>groundcover_pvc_writes_total</td><td>total amount of write operations done by the workload to the PVC</td><td>Number</td><td><span data-option="c476a927c24d4e67b8115feb2e039751">Counter</span></td></tr><tr><td>groundcover_pvc_read_latency</td><td>latency of read operation by the workload from the PVC</td><td>Seconds</td><td><span data-option="c0106f6abc0244adbf70f29447aa3875">Summary</span></td></tr><tr><td>groundcover_pvc_write_latency</td><td>latency of write operation by the workload to the PVC</td><td>Seconds</td><td><span data-option="c0106f6abc0244adbf70f29447aa3875">Summary</span></td></tr></tbody></table>

### <mark style="color:blue;">Network Usage</mark>

#### Labels

**`clusterId workload_name`** **`namespace`** **`container_name`** **`remote_service_name`** **`remote_namespace`** **`remote_is_external`** **`availability_zone`** **`region`** **`remote_availability_zone`** **`remote_region`** **`is_cross_az`** **`protocol`** **`role`** **`server_port`** **`encryption`** **`transport_protocol`** **`is_loopback`**

Note&#x73;**:**

* `is_loopback` and `remote_is_external` are special labels that indicate the remote service is either the same service as the recording side (loopback) or resides in an external network, e.g managed service outside of the cluster (external).
  * In both cases the `remote_service_name` and the `remote_namespace` labels will be empty
* `is_cross_az` means the traffic was sent and/or received between two different availability zones. This is a helpful flag to quickly identify this special kind of communication.
  * The actual zones are detailed in the `availability_zone` and `remote_availability_zone` labels &#x20;

#### Metrics

<table><thead><tr><th width="339.3333333333333">Name</th><th>Description</th><th>Unit</th><th>Type<select><option value="c476a927c24d4e67b8115feb2e039751" label="Counter" color="blue"></option><option value="655b3932f5bd49b0aacc43978ad78425" label="Gauge" color="blue"></option><option value="c0106f6abc0244adbf70f29447aa3875" label="Summary" color="blue"></option></select></th></tr></thead><tbody><tr><td>groundcover_network_rx_bytes_total</td><td>Bytes received by the workload</td><td>Bytes</td><td><span data-option="c476a927c24d4e67b8115feb2e039751">Counter</span></td></tr><tr><td>groundcover_network_tx_bytes_total</td><td>Bytes sent by the workload</td><td>Bytes</td><td><span data-option="c476a927c24d4e67b8115feb2e039751">Counter</span></td></tr><tr><td>groundcover_network_connections_opened_total</td><td>Connections opened by the workload</td><td>Number</td><td><span data-option="c476a927c24d4e67b8115feb2e039751">Counter</span></td></tr><tr><td>groundcover_network_connections_closed_total</td><td>Connections closed by the workload</td><td>Number</td><td><span data-option="c476a927c24d4e67b8115feb2e039751">Counter</span></td></tr><tr><td>groundcover_network_connections_opened_failed_total</td><td>Connections attempts failed per workload (including refused connections)</td><td>Number</td><td><span data-option="c476a927c24d4e67b8115feb2e039751">Counter</span></td></tr><tr><td>groundcover_network_connections_refused_failed_total</td><td>Connections attempts refused per workload</td><td>Number</td><td><span data-option="c476a927c24d4e67b8115feb2e039751">Counter</span></td></tr></tbody></table>

## Application Metrics & Labels

<table><thead><tr><th width="265">Label name</th><th width="266">Description</th><th>Relevant types</th></tr></thead><tbody><tr><td>clusterId</td><td>Name identifier of the K8s cluster</td><td>All</td></tr><tr><td>region</td><td>Cloud provider region name</td><td>All</td></tr><tr><td>namespace</td><td>K8s namespace</td><td>All</td></tr><tr><td>workload_name</td><td>K8s workload (or service) name</td><td>All</td></tr><tr><td>pod_name</td><td>K8s pod name</td><td>All</td></tr><tr><td>container_name</td><td>K8s container name</td><td>All</td></tr><tr><td>container_image</td><td>K8s container image name</td><td>All</td></tr><tr><td>remote_namespace</td><td>Remote K8s namespace (other side of the communication)</td><td>All</td></tr><tr><td>remote_service_name</td><td>Remote K8s service name (other side of the communication)</td><td>All</td></tr><tr><td>remote_container_name</td><td>Remote K8s container name (other side of the communication)</td><td>All</td></tr><tr><td>type</td><td>The protocol in use (HTTP, gRPC, Kafka, DNS etc.)</td><td>All</td></tr><tr><td>sub_type</td><td>The sub type of the protocol (GET, POST, etc)</td><td>All</td></tr><tr><td>role</td><td>Role in the communication (client or server)</td><td>All</td></tr><tr><td>clustered_resource_name</td><td>The clustered name of the resource, depends on the protocol</td><td>All</td></tr><tr><td>status_code</td><td>"ok", "error" or "unset"</td><td>All</td></tr><tr><td>server</td><td>The server workload/name</td><td>All</td></tr><tr><td>client</td><td>The client workload/name</td><td>All</td></tr><tr><td>server_namesapce</td><td>The server namespace</td><td>All</td></tr><tr><td>client_namespace</td><td>The client namespace</td><td>All</td></tr><tr><td>server_is_external</td><td>Indicate whether the server is external</td><td>All</td></tr><tr><td>client_is_external</td><td>Indicate wheter the client is external</td><td>All</td></tr><tr><td>is_encrypted</td><td>Indicate whether the communication is encrypted</td><td>All</td></tr><tr><td>is_cross_az</td><td>Indicate wether the communication is cross availability zone</td><td>All</td></tr><tr><td>clustered_path</td><td>HTTP / gRPC aggregated resource path (e.g. /metrics/*)</td><td>http, grpc</td></tr><tr><td>method</td><td>HTTP / gRPC method (e.g GET)</td><td>http, grpc</td></tr><tr><td>response_status_code</td><td>Return status code of a HTTP / gPRC request (e.g. 200 in HTTP)</td><td>http, grpc</td></tr><tr><td>dialect</td><td>SQL dialect (MySQL or PostgreSQL)</td><td>mysql, postgresql</td></tr><tr><td>response_status</td><td>Return status code of a SQL query (e.g 42P01 for undefined table)</td><td>mysql, postgresql</td></tr><tr><td>client_type</td><td>Kafka client type (Fetcher / Producer)</td><td>kafka</td></tr><tr><td>topic</td><td>Kafka topic name</td><td>kafka</td></tr><tr><td>partition</td><td>Kafka partition identifier</td><td>kafka</td></tr><tr><td>error_code</td><td>Kafka return status code</td><td>kafka</td></tr><tr><td>query_type</td><td>type of DNS query (e.g. AAAA)</td><td>dns</td></tr><tr><td>response_return_code</td><td>Return status code of a DNS resolution request (e.g. Name Error)</td><td>dns</td></tr><tr><td>exit_code</td><td>K8s container termination exit code</td><td>container_state, container_crash</td></tr><tr><td>state</td><td>K8s container current state (Running, Waiting or Terminated)</td><td>container_state</td></tr><tr><td>state_reason</td><td>K8s container state transition reason (e.g CrashLoopBackOff or OOMKilled)</td><td>container_state</td></tr><tr><td>crash_reason</td><td>K8s container crash reason (e.g Error, OOMKilled)</td><td>container_crash</td></tr><tr><td>pvc_name</td><td>K8s PVC name</td><td>storage</td></tr></tbody></table>

{% hint style="info" %}
Summary based metrics have an additional ***quantile*** label, representing the percentile. Available values: \[`”0.5”, “0.95”, 0.99”`].
{% endhint %}

{% hint style="warning" %}
We also use a set of internal labels which are not relevant in most use-cases. Find them interesting? [Let us know over Slack!](https://www.groundcover.com/join-slack)

**`issue_id`** **`entity_id`** **`resource_id`** **`query_id`** **`aggregation_id`** **`parent_entity_id`** **`perspective_entity_id`** **`perspective_entity_is_external`** **`perspective_entity_issue_id`** **`perspective_entity_name`** **`perspective_entity_namespace`** **`perspective_entity_resource_id`**
{% endhint %}

### <mark style="color:blue;">Golden Signals (Errors & Issues)</mark>

In the lists below, we describe **error** and **issue** counters. Every issue flagged by the platform is an error; but not every error is flagged as an issue.

#### Resource metrics

<table><thead><tr><th width="342.3333333333333">Name</th><th width="214">Description</th><th>Unit</th><th>Type<select><option value="c476a927c24d4e67b8115feb2e039751" label="Counter" color="blue"></option><option value="655b3932f5bd49b0aacc43978ad78425" label="Gauge" color="blue"></option><option value="c0106f6abc0244adbf70f29447aa3875" label="Summary" color="blue"></option></select></th></tr></thead><tbody><tr><td>groundcover_resource_total_counter</td><td>total amount of resource requests</td><td>Number</td><td><span data-option="c476a927c24d4e67b8115feb2e039751">Counter</span></td></tr><tr><td>groundcover_resource_error_counter</td><td>total amount of requests with error status codes</td><td>Number</td><td><span data-option="c476a927c24d4e67b8115feb2e039751">Counter</span></td></tr><tr><td>groundcover_resource_issue_counter</td><td>total amount of requests which were flagged as issues</td><td>Number</td><td><span data-option="c476a927c24d4e67b8115feb2e039751">Counter</span></td></tr><tr><td>groundcover_resource_success_counter</td><td>total amount of resource requests with OK status codes</td><td>Number</td><td><span data-option="c476a927c24d4e67b8115feb2e039751">Counter</span></td></tr><tr><td>groundcover_resource_latency_seconds</td><td>resource latency</td><td>Seconds</td><td><span data-option="c0106f6abc0244adbf70f29447aa3875">Summary</span></td></tr></tbody></table>

#### Workload metrics

<table><thead><tr><th width="346.3333333333333">Name</th><th width="210">Description</th><th>Unit</th><th>Type<select><option value="c476a927c24d4e67b8115feb2e039751" label="Counter" color="blue"></option><option value="655b3932f5bd49b0aacc43978ad78425" label="Gauge" color="blue"></option><option value="c0106f6abc0244adbf70f29447aa3875" label="Summary" color="blue"></option></select></th></tr></thead><tbody><tr><td>groundcover_workload_total_counter</td><td>total amount of requests handled by the workload</td><td>Number</td><td><span data-option="c476a927c24d4e67b8115feb2e039751">Counter</span></td></tr><tr><td>groundcover_workload_error_counter</td><td>total amount of requests handled by the workload with error status codes</td><td>Number</td><td><span data-option="c476a927c24d4e67b8115feb2e039751">Counter</span></td></tr><tr><td>groundcover_workload_issue_counter</td><td>total amount of requests handled by the workload which were flagged as issues</td><td>Number</td><td><span data-option="c476a927c24d4e67b8115feb2e039751">Counter</span></td></tr><tr><td>groundcover_workload_success_counter</td><td>total amount of requests handled by the workload with OK status codes</td><td>Number</td><td><span data-option="c476a927c24d4e67b8115feb2e039751">Counter</span></td></tr><tr><td>groundcover_workload_latency_seconds</td><td>resource latency across all of the workload APIs</td><td>Seconds</td><td><span data-option="c0106f6abc0244adbf70f29447aa3875">Summary</span></td></tr></tbody></table>

### <mark style="color:blue;">Kafka specific metrics</mark>

<table><thead><tr><th width="323.3333333333333">Name</th><th width="234">Description</th><th>Unit</th><th>Type<select><option value="c476a927c24d4e67b8115feb2e039751" label="Counter" color="blue"></option><option value="655b3932f5bd49b0aacc43978ad78425" label="Gauge" color="blue"></option><option value="c0106f6abc0244adbf70f29447aa3875" label="Summary" color="blue"></option></select></th></tr></thead><tbody><tr><td>groundcover_client_offset</td><td>client last message offset (for producer the last offset produced, for consumer the last requested offset)</td><td></td><td><span data-option="655b3932f5bd49b0aacc43978ad78425">Gauge</span></td></tr><tr><td>groundcover_workload_client_offset</td><td>client last message offset (for producer the last offset produced, for consumer the last requested offset), aggregated by workload</td><td></td><td><span data-option="655b3932f5bd49b0aacc43978ad78425">Gauge</span></td></tr><tr><td>groundcover_calc_lagged_messages</td><td>current lag in messages</td><td>Number</td><td><span data-option="655b3932f5bd49b0aacc43978ad78425">Gauge</span></td></tr><tr><td>groundcover_workload_calc_lagged_messages</td><td>current lag in messages, aggregated by workload</td><td>Number</td><td><span data-option="655b3932f5bd49b0aacc43978ad78425">Gauge</span></td></tr><tr><td>groundcover_calc_lag_seconds</td><td>current lag in time</td><td>Seconds</td><td><span data-option="655b3932f5bd49b0aacc43978ad78425">Gauge</span></td></tr><tr><td>groundcover_workload_calc_lag_seconds</td><td>current lag in time, aggregated by workload</td><td>Seconds</td><td><span data-option="655b3932f5bd49b0aacc43978ad78425">Gauge</span></td></tr></tbody></table>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.groundcover.com/~/revisions/ETrLpNk6KtHjyaVUTLoE/use-groundcover/metrics-and-labels.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.