Metrics & Labels

Infrastructure Metrics & Labels

Container CPU and Memory

Labels

type

clusterId region namespace node_name workload_name

pod_name container_name container_image

Metrics

Name	Description	Unit	Type
groundcover_container_cpu_usage_rate_millis	CPU usage in mCPU	mCPU	Gauge
groundcover_container_cpu_request_m_cpu	K8s container CPU request	mCPU	Gauge
groundcover_container_cpu_limit_m_cpu	K8s container CPU limit	mCPU	Gauge
groundcover_container_memory_working_set_bytes	current memory working set	Bytes	Gauge
groundcover_container_memory_rss_bytes	current memory RSS	Bytes	Gauge
groundcover_container_memory_request_bytes	K8s container memory request	Bytes	Gauge
groundcover_container_memory_limit_bytes	K8s container memory limit	Bytes	Gauge
groundcover_container_cpu_delay_seconds	K8s container CPU delay	Seconds	Counter
groundcover_container_disk_delay_seconds	K8s container disk delay	Seconds	Counter
groundcover_container_cpu_throttled_seconds_total	K8s container total CPU throttling	Seconds	Counter

Node CPU, Memory and Disk

Labels

type clusterId region node_name

Metrics

Name	Description	Unit	Type
groundcover_node_allocatable_cpum_cpu	amount of allocatable CPU in the current node	mCPU	Gauge
groundcover_node_allocatable_mem_bytes	amount of allocatable memory in the current node	Bytes	Gauge
groundcover_node_mem_used_percent	percent of used memory in current node	0-100	Gauge
groundcover_node_used_disk_space	current used disk space in current node	Bytes	Gauge
groundcover_node_free_disk_space	amount of free disk space in current node	Bytes	Gauge
groundcover_node_total_disk_space	amount of total disk space in current node	Bytes	Gauge
groundcover_node_used_percent_disk_space	percent of used disk space in current node	0-100	Gauge

Storage Usage

Labels

type clusterId region name namespace

Metrics

Name	Description	Unit	Type
groundcover_pvc_usage_bytes	PVC usage	Bytes	Gauge
groundcover_pvc_capacity_bytes	PVC capacity	Bytes	Gauge
groundcover_pvc_available_bytes	PVC available	Bytes	Gauge
groundcover_pvc_usage_percent	percent of used PVC storage	0-100	Gauge
groundcover_pvc_read_bytes_total	total amount of bytes read by the workload from the PVC	Bytes	Counter
groundcover_pvc_write_bytes_total	total amount of bytes written by the workload to the PVC	Bytes	Counter
groundcover_pvc_reads_total	total amount of read operations done by the workload from the PVC	Number	Counter
groundcover_pvc_writes_total	total amount of write operations done by the workload to the PVC	Number	Counter
groundcover_pvc_read_latency	latency of read operation by the workload from the PVC	Seconds	Summary
groundcover_pvc_write_latency	latency of write operation by the workload to the PVC	Seconds	Summary

Network Usage

Labels

clusterId workload_name namespace container_name remote_service_name remote_namespace remote_is_external availability_zone region remote_availability_zone remote_region is_cross_az protocol role server_port encryption transport_protocol is_loopback

Notes:

• is_loopback and remote_is_external are special labels that indicate the remote service is either the same service as the recording side (loopback) or resides in an external network, e.g managed service outside of the cluster (external).

  • In both cases the remote_service_name and the remote_namespace labels will be empty

• is_cross_az means the traffic was sent and/or received between two different availability zones. This is a helpful flag to quickly identify this special kind of communication.

  • The actual zones are detailed in the availability_zone and remote_availability_zone labels

Metrics

Name	Description	Unit	Type
groundcover_network_rx_bytes_total	Bytes received by the workload	Bytes	Counter
groundcover_network_tx_bytes_total	Bytes sent by the workload	Bytes	Counter
groundcover_network_connections_opened_total	Connections opened by the workload	Number	Counter
groundcover_network_connections_closed_total	Connections closed by the workload	Number	Counter
groundcover_network_connections_opened_failed_total	Connections attempts failed per workload (including refused connections)	Number	Counter
groundcover_network_connections_refused_failed_total	Connections attempts refused per workload	Number	Counter

Application Metrics & Labels

Label name	Description	Relevant types
clusterId	Name identifier of the K8s cluster	All
region	Cloud provider region name	All
namespace	K8s namespace	All
workload_name	K8s workload (or service) name	All
pod_name	K8s pod name	All
container_name	K8s container name	All
container_image	K8s container image name	All
remote_namespace	Remote K8s namespace (other side of the communication)	All
remote_service_name	Remote K8s service name (other side of the communication)	All
remote_container_name	Remote K8s container name (other side of the communication)	All
type	The protocol in use (HTTP, gRPC, Kafka, DNS etc.)	All
sub_type	The sub type of the protocol (GET, POST, etc)	All
role	Role in the communication (client or server)	All
clustered_resource_name	The clustered name of the resource, depends on the protocol	All
status_code	"ok", "error" or "unset"	All
server	The server workload/name	All
client	The client workload/name	All
server_namesapce	The server namespace	All
client_namespace	The client namespace	All
server_is_external	Indicate whether the server is external	All
client_is_external	Indicate wheter the client is external	All
is_encrypted	Indicate whether the communication is encrypted	All
is_cross_az	Indicate wether the communication is cross availability zone	All
clustered_path	HTTP / gRPC aggregated resource path (e.g. /metrics/*)	http, grpc
method	HTTP / gRPC method (e.g GET)	http, grpc
response_status_code	Return status code of a HTTP / gPRC request (e.g. 200 in HTTP)	http, grpc
dialect	SQL dialect (MySQL or PostgreSQL)	mysql, postgresql
response_status	Return status code of a SQL query (e.g 42P01 for undefined table)	mysql, postgresql
client_type	Kafka client type (Fetcher / Producer)	kafka
topic	Kafka topic name	kafka
partition	Kafka partition identifier	kafka
error_code	Kafka return status code	kafka
query_type	type of DNS query (e.g. AAAA)	dns
response_return_code	Return status code of a DNS resolution request (e.g. Name Error)	dns
exit_code	K8s container termination exit code	container_state, container_crash
state	K8s container current state (Running, Waiting or Terminated)	container_state
state_reason	K8s container state transition reason (e.g CrashLoopBackOff or OOMKilled)	container_state
crash_reason	K8s container crash reason (e.g Error, OOMKilled)	container_crash
pvc_name	K8s PVC name	storage

Summary based metrics have an additional quantile label, representing the percentile. Available values: [”0.5”, “0.95”, 0.99”].

We also use a set of internal labels which are not relevant in most use-cases. Find them interesting? Let us know over Slack!

issue_id entity_id resource_id query_id aggregation_id parent_entity_id perspective_entity_id perspective_entity_is_external perspective_entity_issue_id perspective_entity_name perspective_entity_namespace perspective_entity_resource_id

Golden Signals (Errors & Issues)

In the lists below, we describe error and issue counters. Every issue flagged by the platform is an error; but not every error is flagged as an issue.

Resource metrics

Name	Description	Unit	Type
groundcover_resource_total_counter	total amount of resource requests	Number	Counter
groundcover_resource_error_counter	total amount of requests with error status codes	Number	Counter
groundcover_resource_issue_counter	total amount of requests which were flagged as issues	Number	Counter
groundcover_resource_success_counter	total amount of resource requests with OK status codes	Number	Counter
groundcover_resource_latency_seconds	resource latency	Seconds	Summary

Workload metrics

Name	Description	Unit	Type
groundcover_workload_total_counter	total amount of requests handled by the workload	Number	Counter
groundcover_workload_error_counter	total amount of requests handled by the workload with error status codes	Number	Counter
groundcover_workload_issue_counter	total amount of requests handled by the workload which were flagged as issues	Number	Counter
groundcover_workload_success_counter	total amount of requests handled by the workload with OK status codes	Number	Counter
groundcover_workload_latency_seconds	resource latency across all of the workload APIs	Seconds	Summary

Kafka specific metrics

Name	Description	Unit	Type
groundcover_client_offset	client last message offset (for producer the last offset produced, for consumer the last requested offset)		Gauge
groundcover_workload_client_offset	client last message offset (for producer the last offset produced, for consumer the last requested offset), aggregated by workload		Gauge
groundcover_calc_lagged_messages	current lag in messages	Number	Gauge
groundcover_workload_calc_lagged_messages	current lag in messages, aggregated by workload	Number	Gauge
groundcover_calc_lag_seconds	current lag in time	Seconds	Gauge
groundcover_workload_calc_lag_seconds	current lag in time, aggregated by workload	Seconds	Gauge