Logs To Metrics Parsing

What is Logs-to-Metrics?

At its simplest, logs-to-metrics is the process of extracting numerical, structured data from your unstructured, text-based logs and converting it into time-series metrics.

Think of it like this:

A Log is a Sentence: INFO: HTTP GET /api/users request completed in 55ms with status 200.
A Metric is a Spreadsheet Row:

Timestamp

Metric Name

Value

Labels

[now]

http_requests_total

1

method:GET, endpoint:/api/users, status:200

[now]

http_request_duration_ms

55

method:GET, endpoint:/api/users, status:200

You are essentially turning descriptive sentences into countable, measurable data points.

Where Should You Use It?

Logs-to-metrics is not for replacing logs entirely. Logs are essential for debugging specific events. Instead, you should use it in these key scenarios:

1. Counting Key Business or Application Events

You want to count how often something happens.

Use Case: Count the number of user logins, failed payments, or items added to a shopping cart.
Example: Create a metric named user_logins_total by counting every log line that contains "User successfully authenticated".

2. Monitoring Error Rates

You want to track the frequency of errors to understand system health.

Use Case: Alerting when the rate of 500 status codes or ERROR level logs spikes.
Example: Create a metric named http_requests_total with a label for the status code. Then you can easily calculate the error rate: rate(http_requests_total{status="500"}) / rate(http_requests_total).

3. Monitoring Legacy or Third-Party Applications

You need to monitor an application that you cannot modify to expose its own metrics.

Use Case: An old Java application or a third-party tool only writes its status to a log file.
Example: You can parse that log file to extract key performance indicators (like active connections or tasks processed) and turn them into metrics, giving you visibility you wouldn't otherwise have.

How to Apply Log to Metric Parsing Rule

To create achieve log to metric we will need to create parsing rule.

The rule will be structured from defending the l2m values or attributes

and from the log_to_metric_X logic that can be one of the following:

log_to_metric_count - count all logs with the desired values or attributes
log_to_metric_max - max value of logs with the desired values or attributes
log_to_metric_min - mix value of logs with the desired values or attributes
log_to_metric_sum - summarize value of logs with the desired values or attributes

Please note that groundcover adds gc_op suffix with the numeric operation

Example Structure

  - ruleName: l2m-kong
    statements:
      - set(l2m["request_path"],attributes["request.path"])
      - set(l2m["cluster"],cluster)
      - log_to_metric_sum("kong_request_volume", l2m,
        Double(attributes["bytes"]))
      - log_to_metric_min("kong_request_volume", l2m,
        Double(attributes["bytes"]))
      - log_to_metric_max("kong_request_volume", l2m,
        Double(attributes["bytes"]))
      - set(l2m["status_code"],attributes["status_code"])
      - log_to_metric_count("kong_access_log_metrics", l2m)
    conditions:
      - container_name == "proxy"
      - workload == "groundcover-incloud-kong"
    statementsErrorMode: propagate
    conditionLogicOperator: and

log 1

10.1.139.127 - - [07/Aug/2025:12:07:00 +0000] "GET /fleet-manager/api/client/config HTTP/2.0" 403 19 "-" "Go-http-client/2.0"

log 2

10.1.139.127 - - [07/Aug/2025:12:08:00 +0000] "GET /fleet-manager/api/client/config HTTP/2.0" 403 189 "-" "Go-http-client/2.0"

log 3

10.1.139.127 - - [07/Aug/2025:12:10:00 +0000] "GET /fleet-manager/api/client/config HTTP/2.0" 403 6 "-" "Go-http-client/2.0"

Results:

Metric

Value

kong_access_log_metrics_sum

214

kong_access_log_metrics_min

kong_access_log_metrics_max

189

kong_access_log_metrics_count

Those result are also create a metric which will be available at Metric explorer :

Last updated 2 days ago