Writing Remap Transforms

Below are the essentials relevant to writing remap transforms in groundcover. Extended information can be found in Vector's documentation.

We support using all types of Vector transforms as pipeline steps.

For testing VRL before deployment we recommend the VRL playground.

Working with fields and attributes

When processing Vector events, fields names need to be prefixed by . , a single period. For example, the content field in a log, representing the body of the log, is accessible using .content. Specifically in groundcover, attributes parsed from logs or associated with traces will be stored under the string_attributes for string values, and under float_attributes for numerical values. Accessing attributes is possible by adding additional . as needed. For example, a JSON log that looks like this:

{"name":"my-log","count":5} 

Will be translated into an event with the following attributes:

.string_attributes.name --> "my-log"
.float_attributes.count --> 5

Fallible functions

Each of Vector's built-in function can be either fallible or infallible. Fallible functions can throw an error when called, and require error handling, whereas infallible functions will never throw an error.

When writing Vector transforms in VRL it's important to use error handling where needed. Below are the two ways error handling in Vector is possible - see more on these docs.

VRL code without proper error handling will throw an error during compilation, resulting in error logs in the Vector deployment.

Option #1 - Handling the error

Let's take a look at the following code.

.parsed, .err = parse_json("{\"Hello\": \"World!\"}")
if err == null {
  // do something with .parsed
}

The code above can either succeed in parsing the json, or fail in parsing it. The err variable will contain indication of the result status, and we can proceed accordingly.

Option #2 - Aborting on error

Let's take a look at this slightly different version of the code above:

parsed = parse_json!("{\"Hello\": \"World!\"}")

This time there's no error handling around, but ! was added after the function call.

This method of error handling is called abort on error - it will fail the transform entirely if the function returns an error, and proceed normally otherwise.

Choosing which method to use

Both methods above are valid VRL for handling errors, and you must choose one or the other when handling fallible functions. However, they carry one big difference in terms of pipelines in groundcover:

• Transforms which use option #1 (error handling) will not stop the pipeline in case of error - the following steps will continue to execute normally. This is useful when writing optional enrichment steps that could potentially fail with no issue.

• Transforms which use option #2 (abort) will stop the pipeline in case of error - the event will not proceed to the other steps. This is mostly useful for mandatory steps which can't fail no matter what.

The default behavior above can be changed using the drop_on_error flag. When this flag is set to false, errors encountered will never stop the pipeline - both for method #1 and for method #2.

This is useful for writing simpler code with less explicit error handling, as can be seen in this log pipeline example.