High Availability

Knowledge of the groundcover architecture is basis for understanding how we implement high availability. We strongly advise starting with the architecture section.

groundcover's inCloud Managed architecture is deployed with High Availability in mind, ensuring continuity in data ingestion.

There are a few main techniques groundcover utilizes to ensure high availability in the ingestion pipeline:

1. Redundancy in ingestion components

2. Data transfer via object storage

3. Database redundancy - coming soon!

Redundancy in ingestion components

groundcover's pipeline is made of multiple components to receive, process and ship data being picked up by sensors. These are the main ones:

1. Victoria Metrics Agent (vmagent)- for metrics ingestion and shipping

2. Vector - for traces, logs and events ingestion and shipping

3. Kong - as an API gateway in the managed backend

As seen in the architecture diagram, these components are deployed in various ways, both inside each environment being monitored and in the managed backend. Together, they make up a full pipeline from sensors to databases.

In order to ensure availability, these services are deployed with horizontal redundancy, ensuring no single node failure can cause a downtime in the pipeline. The components are also scaled vertically in a way that allows them to make up for some replicas being down.

Data transfer via object storage

This currently only applies to logs, traces, and events. Metrics are shipped via network.

groundcover applies an innovative approach to transfer data from the monitored environments into the managed backend - using object storage, provisioned entirely in your environment.

In addition to reducing network costs by over 95%, this approach greatly compliments high availability needs. Unlike sending data over the network, the aggregation components in each monitored environment rely only on object storage services being available for writing, which is guaranteed in the cloud providers' SLAs.

This creates an asynchronous ingestion flow, where data is written from the monitored environments to the object storage, which then makes it immediately available to be read by the managed backend. It is an approach that is unique to groundcover, and completely eliminates the bottlenecks commonly seen in synchronous architectures, allowing the ingestion to always continue even during traffic peaks or any kind of I/O throttling on the receiving end.

Database redundancy

Coming soon!