Application Metrics
Our metrics philosophy
The groundcover platform generates 100% of its metrics from the actual data. There are no sample rates or complex interpolations to make up for partial coverage. Our measurements represent the real, complete flow of data in your environment.
Stream processing allows us to construct the majority of the metrics on the very node where the raw transactions are recorded. This means the raw data is turned into numbers the moment it becomes possible - removing the need for storing or sending it elsewhere.
Metrics are stored in groundcover's victoria-metrics deployment, ensuring top-notch performance on every scale.
Golden signals
In the world of excessive data, it's important to have a rule of thumb for knowing where to start looking. For application metrics, we rely on our golden signals.
The following metrics are generated for each resource being aggregated:
Requests per second (RPS)
Errors rate
Latencies (p50 and p95)
The golden signals are then displayed in two important ways: Workload and Resource aggregations.
Resource aggregations are highly granularity metrics, providing insights into individual APIs.
Workload aggregations are designed to show an overview of each service, enabling a higher level inspection. These are constructed using all of the resources recorded for each service.
Controlling retention
groundcover allows full control over the retention of your metrics. Learn more here.
List of available metrics
Below you will find the full list of our APM metrics, as well as the labels we export for each. These labels are designed with high granularity in mind for maximal insight depth. All of the metrics listed are available out of the box after installing groundcover, without any further setup.
Our labels
clusterId
Name identifier of the K8s cluster
region
Cloud provider region name
namespace
K8s namespace
workload_name
K8s workload (or service) name
pod_name
K8s pod name
container_name
K8s container name
container_image
K8s container image name
remote_namespace
Remote K8s namespace (other side of the communication)
remote_service_name
Remote K8s service name (other side of the communication)
remote_container_name
Remote K8s container name (other side of the communication)
type
The protocol in use (HTTP, gRPC, Kafka, DNS etc.)
role
Role in the communication (client or server)
clustered_path
HTTP / gRPC aggregated resource path (e.g. /metrics/*)
http, grpc
method
HTTP / gRPC method (e.g GET)
http, grpc
response_status_code
Return status code of a HTTP / gPRC request (e.g. 200 in HTTP)
http, grpc
dialect
SQL dialect (MySQL or PostgreSQL)
mysql, postgresql
response_status
Return status code of a SQL query (e.g 42P01 for undefined table)
mysql, postgresql
client_type
Kafka client type (Fetcher / Producer)
kafka
topic
Kafka topic name
kafka
partition
Kafka partition identifier
kafka
error_code
Kafka return status code
kafka
query_type
type of DNS query (e.g. AAAA)
dns
response_return_code
Return status code of a DNS resolution request (e.g. Name Error)
dns
method_name, method_class_name
Method code for the operation
amqp
response_method_name, response_method_class_name
Method code for the operation's response
amqp
exit_code
K8s container termination exit code
container_state, container_crash
state
K8s container current state (Running, Waiting or Terminated)
container_state
state_reason
K8s container state transition reason (e.g CrashLoopBackOff or OOMKilled)
container_state
crash_reason
K8s container crash reason (e.g Error, OOMKilled)
container_crash
pvc_name
K8s PVC name
storage
groundcover uses a set of internal labels which are not relevant in most use-cases. Find them interesting? Let us know over Slack!
issue_id entity_id resource_id query_id aggregation_id parent_entity_id perspective_entity_id perspective_entity_is_external perspective_entity_issue_id perspective_entity_name perspective_entity_namespace perspective_entity_resource_id
Golden Signals Metrics
The golden signals are queried through the APM data source, available in Explore, dashboards and monitors. It runs against groundcover's pre-aggregated APM measurements and exposes the golden signals as ready-made metrics that you scope with structured pickers instead of writing a query against raw metric series:
Requests
Number of requests observed in the selection
Number
Errors
Number of requests that completed with an error status
Number
Error Rate
Share of requests that completed with an error status
Percent
Request Rate
Requests per second over the queried time range
Req/s
AVG Latency
Average request latency
Seconds
P50 Latency
Median request latency (50th percentile)
Seconds
P95 Latency
95th percentile of request latency
Seconds
P99 Latency
99th percentile of request latency
Seconds
Total Time Spent
Total time spent serving requests, highlighting the most expensive paths
Seconds
Scope the result with the Server / Client, Resource Type and Endpoint pickers, switch between an inbound and an outbound view of your traffic, and break the result down with Group By on any label listed above.
The groundcover_resource_* and groundcover_workload_* golden signal series are deprecated - that is, the _total_counter, _error_counter, _issue_counter, _success_counter and _latency_seconds metrics for both prefixes. Use the APM data source for new dashboards, monitors and queries.
Storage usage metrics
groundcover_pvc_read_bytes_total
total amount of bytes read by the workload from the PVC
groundcover_pvc_write_bytes_total
total amount of bytes written by the workload to the PVC
groundcover_pvc_reads_total
total amount of read operations done by the workload from the PVC
groundcover_pvc_writes_total
total amount of write operations done by the workload to the PVC
groundcover_pvc_read_latency
latency of read operation by the workload from the PVC, in microseconds
groundcover_pvc_write_latency
latency of write operation by the workload to the PVC, in microseconds
Kafka specific metrics
groundcover_client_offset
client last message offset (for producer the last offset produced, for consumer the last requested offset)
groundcover_workload_client_offset
client last message offset (for producer the last offset produced, for consumer the last requested offset), aggregated by workload
groundcover_calc_lagged_messages
current lag in messages
groundcover_workload_calc_lagged_messages
current lag in messages, aggregated by workload
groundcover_calc_lag_seconds
current lag in time [sec]
groundcover_workload_calc_lag_seconds
current lag in time, aggregated by workload [sec]
Last updated