Monitor Lambdas using OTEL Layers

This guide explains how to instrument your AWS Lambda functions using OpenTelemetry and ship traces, metrics, and logs to groundcover.

Overview

AWS Lambda's serverless architecture presents unique observability challenges:

• Ephemeral execution: Functions may exist only for a single request, requiring telemetry to be exported quickly

• Cold starts: Each new execution environment introduces latency that should be tracked

• Distributed nature: Serverless architectures span multiple services, making trace correlation essential

The OpenTelemetry Lambda project provides Lambda layers that solve these challenges with automatic instrumentation and an embedded collector.

How It Works

The setup uses two Lambda layers working together:

1. Instrumentation Layer (language-specific) - Automatically instruments your code, captures context from upstream callers, creates spans, and instruments the AWS SDK

2. Collector Layer - Embeds a stripped-down OpenTelemetry Collector as a Lambda extension that processes and exports telemetry data

The collector runs as an AWS Lambda Extension, receiving telemetry from your instrumented function and forwarding it to groundcover. It also captures your function's stdout and stderr as logs.

OpenTelemetry Lambda supports automatic instrumentation for Python, Node.js, Java, and Ruby. For .NET and Go, manual instrumentation is required.

Prerequisites

Before you begin, ensure you have:

• An AWS account with permissions to create/modify Lambda functions and layers

• A groundcover account with a BYOC endpoint

• A groundcover Ingestion Key

• Knowledge of your Lambda function's runtime and architecture (x86_64 or arm64)

Step 1: Create the Collector Configuration

The OpenTelemetry Collector requires a configuration file that defines how telemetry is received and exported.

Configuration File

Create a file named otel-collector-config.yaml:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317
      http:
        endpoint: localhost:4318
  telemetryapi:

exporters:
  otlphttp/groundcover:
    endpoint: https://{BYOC_ENDPOINT}
    compression: gzip
    headers:
      apikey: "{INGESTION_KEY}"
      # Optional: Add enrichment headers for better context in groundcover
      x-groundcover-service-name: "{SERVICE_NAME}"
      x-groundcover-env-name: "{ENVIRONMENT}"
      x-groundcover-source: "otel-lambda"

service:
  pipelines:
    traces:
      receivers:
        - otlp
        - telemetryapi
      exporters:
        - otlphttp/groundcover
    metrics:
      receivers:
        - otlp
      exporters:
        - otlphttp/groundcover
    logs:
      receivers:
        - otlp
        - telemetryapi
      exporters:
        - otlphttp/groundcover

The telemetryapi receiver captures Lambda platform telemetry including:

• Cold start spans - Records function initialization time (platform.initRuntimeDone)

• Function logs - Captures stdout and stderr from your Lambda function

Replace the following placeholders:

• {BYOC_ENDPOINT} - Your groundcover BYOC endpoint

• {INGESTION_KEY} - Your ingestion key

• {SERVICE_NAME} - Your Lambda function or service name

• {ENVIRONMENT} - Your environment (e.g., production, staging)

Find your endpoint and ingestion key in Settings > Ingestion Keys.

Creating a Configuration Layer

Package the collector configuration as a Lambda layer for easy reuse:

1. Create a zip file with otel-collector-config.yaml at the root (not inside a directory)

2. Open the Lambda Layers console

3. Click Create layer

4. Enter a name (e.g., otel-collector-config)

5. Upload your zip file

6. Select the compatible architectures and runtimes

7. Click Create

Alternatively, you can include the config file directly in your function bundle at the root level, or store it in S3 and reference via URI (ensure your Lambda's IAM role has s3:GetObject permission).

Step 2: Add the Lambda Layers

Your Lambda function needs three layers:

1. OpenTelemetry Instrumentation Layer - Auto-instruments your code (language-specific)

2. OpenTelemetry Collector Layer - Processes and exports telemetry

3. Configuration Layer - Contains your collector config (from Step 1)

Layer ARNs

Replace <region> with your AWS region and <version> with the latest version from the OpenTelemetry Lambda releases.

Layer versions are updated frequently. Always check the releases page for the latest versions before deploying.

Instrumentation Layers

Runtime	Layer ARN	Wrapper Path
Node.js	arn:aws:lambda:<region>:184161586896:layer:opentelemetry-nodejs-<version>:1	/opt/otel-handler
Python	arn:aws:lambda:<region>:184161586896:layer:opentelemetry-python-<version>:1	/opt/otel-instrument
Java Agent	arn:aws:lambda:<region>:184161586896:layer:opentelemetry-javaagent-<version>:1	/opt/otel-handler
Java Wrapper	arn:aws:lambda:<region>:184161586896:layer:opentelemetry-javawrapper-<version>:1	/opt/otel-handler
Ruby	arn:aws:lambda:<region>:184161586896:layer:opentelemetry-ruby-<version>:1	/opt/otel-handler

Collector Layer

Architecture	Layer ARN
x86_64 (amd64)	arn:aws:lambda:<region>:184161586896:layer:opentelemetry-collector-amd64-<version>:1
arm64	arn:aws:lambda:<region>:184161586896:layer:opentelemetry-collector-arm64-<version>:1

Make sure the collector layer architecture matches your Lambda function's architecture.

Adding Layers via AWS Console

1. Open the Lambda Console

2. Select your function

3. In the Layers section, click Add a layer

4. Select Specify an ARN

5. Paste the appropriate ARN

6. Click Verify, then Add

7. Repeat for all three layers (instrumentation, collector, configuration)

Adding Layers via Terraform

resource "aws_lambda_function" "my_function" {
  function_name = "my-function"
  runtime       = "python3.12"
  handler       = "index.handler"
  architectures = ["x86_64"]
  
  layers = [
    # Instrumentation layer - check releases for latest version
    "arn:aws:lambda:us-east-1:184161586896:layer:opentelemetry-python-<version>:1",
    # Collector layer (match your architecture) - check releases for latest version
    "arn:aws:lambda:us-east-1:184161586896:layer:opentelemetry-collector-amd64-<version>:1",
    # Your configuration layer
    aws_lambda_layer_version.otel_config.arn
  ]
}

resource "aws_lambda_layer_version" "otel_config" {
  layer_name          = "otel-collector-config"
  filename            = "otel-config-layer.zip"
  compatible_runtimes = ["python3.12"]
}

Adding Layers via AWS CDK

import { Function, Code, Runtime, LayerVersion, Architecture } from 'aws-cdk-lib/aws-lambda';

const fn = new Function(this, 'MyFunction', {
  code: Code.fromAsset('lambda'),
  runtime: Runtime.PYTHON_3_12,
  handler: 'index.handler',
  architecture: Architecture.X86_64,
});

// Add instrumentation layer - check releases for latest version
fn.addLayers(LayerVersion.fromLayerVersionArn(
  this, 'OtelInstrumentationLayer',
  'arn:aws:lambda:us-east-1:184161586896:layer:opentelemetry-python-<version>:1'
));

// Add collector layer - check releases for latest version
fn.addLayers(LayerVersion.fromLayerVersionArn(
  this, 'OtelCollectorLayer',
  'arn:aws:lambda:us-east-1:184161586896:layer:opentelemetry-collector-amd64-<version>:1'
));

// Add your configuration layer
fn.addLayers(LayerVersion.fromLayerVersionArn(
  this, 'OtelConfigLayer',
  'arn:aws:lambda:us-east-1:123456789012:layer:otel-collector-config:1'
));

Step 3: Configure Environment Variables

Set the following environment variables on your Lambda function:

Variable	Value	Description
AWS_LAMBDA_EXEC_WRAPPER	See wrapper path in the table above	Enables automatic instrumentation
OPENTELEMETRY_COLLECTOR_CONFIG_URI	/opt/otel-collector-config.yaml	Path to collector config

The collector config can also be loaded from your function bundle (/var/task/otel-collector-config.yaml) or from S3 (s3://<bucket>.s3.<region>.amazonaws.com/config.yaml).

Step 4: Deploy and Test

1. Deploy your Lambda function with the layers and configuration

2. Invoke the function (via AWS Console, CLI, or trigger)

3. Check groundcover for incoming telemetry data

Allow 1-2 minutes for data to appear in groundcover after the first invocation.

Complete Example

Collector Config (otel-collector-config.yaml)

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317
      http:
        endpoint: localhost:4318
  telemetryapi:

exporters:
  otlphttp/groundcover:
    endpoint: https://m234r1.platform.grcv.io
    compression: gzip
    headers:
      apikey: "your-ingestion-key"
      x-groundcover-service-name: "my-python-lambda"
      x-groundcover-env-name: "production"
      x-groundcover-source: "otel-lambda"

service:
  pipelines:
    traces:
      receivers:
        - otlp
        - telemetryapi
      exporters:
        - otlphttp/groundcover
    metrics:
      receivers:
        - otlp
      exporters:
        - otlphttp/groundcover
    logs:
      receivers:
        - otlp
        - telemetryapi
      exporters:
        - otlphttp/groundcover

Terraform Configuration

# Configuration layer
resource "aws_lambda_layer_version" "otel_config" {
  layer_name          = "otel-collector-config"
  filename            = "otel-config-layer.zip"
  compatible_runtimes = ["python3.12"]
}

# Lambda function
resource "aws_lambda_function" "my_python_lambda" {
  function_name = "my-python-lambda"
  runtime       = "python3.12"
  handler       = "index.handler"
  filename      = "lambda.zip"
  role          = aws_iam_role.lambda_role.arn
  timeout       = 30
  memory_size   = 256
  architectures = ["x86_64"]

  layers = [
    # Check https://github.com/open-telemetry/opentelemetry-lambda/releases for latest versions
    "arn:aws:lambda:us-east-1:184161586896:layer:opentelemetry-python-<version>:1",
    "arn:aws:lambda:us-east-1:184161586896:layer:opentelemetry-collector-amd64-<version>:1",
    aws_lambda_layer_version.otel_config.arn
  ]

  environment {
    variables = {
      AWS_LAMBDA_EXEC_WRAPPER            = "/opt/otel-instrument"
      OPENTELEMETRY_COLLECTOR_CONFIG_URI = "/opt/otel-collector-config.yaml"
    }
  }
}

Enriching Telemetry Data

groundcover supports enriching ingested data with additional context using HTTP headers in the collector configuration.

Add enrichment headers in the exporter configuration:

exporters:
  otlphttp/groundcover:
    endpoint: https://{BYOC_ENDPOINT}
    compression: gzip
    headers:
      apikey: "{INGESTION_KEY}"
      x-groundcover-service-name: "my-lambda-function"
      x-groundcover-env-name: "production"
      x-groundcover-source: "otel-lambda"

Available Enrichment Headers

Header	Description	Example
x-groundcover-service-name	Service/workload name	payment-processor
x-groundcover-env-name	Environment name	production, staging
x-groundcover-env-type	Environment type	lambda, vm
x-groundcover-source	Data source identifier	otel-lambda

For more details, see Enriching 3rd Party Data.

Sampling Configuration

Always Sample (Development)

OTEL_TRACES_SAMPLER=always_on

Ratio-Based Sampling (Production)

Sample 10% of traces to reduce costs:

OTEL_TRACES_SAMPLER=traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1

For high-traffic Lambda functions, ratio-based sampling reduces costs while maintaining visibility.

Collecting Logs

The telemetryapi receiver automatically captures your Lambda function's stdout and stderr output via the Lambda Telemetry API and forwards them as logs to groundcover.

To correlate logs with traces, use a logger that outputs trace context. For Python:

import logging
from opentelemetry import trace

logger = logging.getLogger(__name__)

def handler(event, context):
    span = trace.get_current_span()
    trace_id = span.get_span_context().trace_id
    logger.info(f"Processing event", extra={"trace_id": format(trace_id, '032x')})

Lambda Platform Metrics

The OpenTelemetry layers capture application-level metrics. For Lambda platform metrics (invocation count, duration, throttles, etc.), use the CloudWatch Metrics integration to ingest the AWS/Lambda namespace.

Troubleshooting

Data Not Appearing in groundcover

1. Check config path: Verify OPENTELEMETRY_COLLECTOR_CONFIG_URI points to the correct location

2. Verify credentials: Confirm your ingestion key is valid

3. Check Lambda logs: Look for collector errors in CloudWatch Logs

4. Enable debug logging: Set OPENTELEMETRY_EXTENSION_LOG_LEVEL=debug

5. Network connectivity: Ensure Lambda can reach the groundcover endpoint (port 443)

403 Forbidden Errors

• Verify your ingestion key is valid and not revoked

• Ensure you're using a Third Party type ingestion key

• Check the apikey header format in your collector config

Architecture Mismatch

If you see errors about missing libraries:

• Ensure the collector layer architecture (amd64 vs arm64) matches your function's architecture

Additional Resources

• OpenTelemetry Lambda GitHub

• OpenTelemetry Lambda Releases

• groundcover OpenTelemetry Integration

• groundcover Ingestion Keys

• Ingest CloudWatch Metrics - For Lambda platform metrics