# Monitor Lambdas using OTEL Layers This guide explains how to instrument your AWS Lambda functions using OpenTelemetry and ship traces, metrics, and logs to groundcover. ## Overview AWS Lambda's serverless architecture presents unique observability challenges: * **Ephemeral execution**: Functions may exist only for a single request, requiring telemetry to be exported quickly * **Cold starts**: Each new execution environment introduces latency that should be tracked * **Distributed nature**: Serverless architectures span multiple services, making trace correlation essential The [OpenTelemetry Lambda](https://github.com/open-telemetry/opentelemetry-lambda) project provides Lambda layers that solve these challenges with automatic instrumentation and an embedded collector. ## How It Works The setup uses two Lambda layers working together: 1. **Instrumentation Layer** (language-specific) - Automatically instruments your code, captures context from upstream callers, creates spans, and instruments the AWS SDK 2. **Collector Layer** - Embeds a stripped-down OpenTelemetry Collector as a Lambda extension that processes and exports telemetry data The collector runs as an [AWS Lambda Extension](https://docs.aws.amazon.com/lambda/latest/dg/lambda-extensions.html), receiving telemetry from your instrumented function and forwarding it to groundcover. It also captures your function's `stdout` and `stderr` as logs. {% hint style="info" %} OpenTelemetry Lambda supports automatic instrumentation for **Python**, **Node.js**, **Java**, and **Ruby**. For **.NET** and **Go**, manual instrumentation is required. {% endhint %} ## Prerequisites Before you begin, ensure you have: * An AWS account with permissions to create/modify Lambda functions and layers * A groundcover account with a [BYOC endpoint](/architecture/byoc/ingestion-endpoints.md) * A groundcover [Ingestion Key](/use-groundcover/remote-access-and-apis/ingestion-keys.md) * Knowledge of your Lambda function's runtime and architecture (x86\_64 or arm64) ## Step 1: Create the Collector Configuration The OpenTelemetry Collector requires a configuration file that defines how telemetry is received and exported. ### Configuration File Create a file named `otel-collector-config.yaml`: ```yaml receivers: otlp: protocols: grpc: endpoint: localhost:4317 http: endpoint: localhost:4318 telemetryapi: exporters: otlphttp/groundcover: endpoint: https://{BYOC_ENDPOINT} compression: gzip headers: apikey: "{INGESTION_KEY}" # Optional: Add enrichment headers for better context in groundcover x-groundcover-service-name: "{SERVICE_NAME}" x-groundcover-env-name: "{ENVIRONMENT}" x-groundcover-source: "otel-lambda" service: pipelines: traces: receivers: - otlp - telemetryapi exporters: - otlphttp/groundcover metrics: receivers: - otlp exporters: - otlphttp/groundcover logs: receivers: - otlp - telemetryapi exporters: - otlphttp/groundcover ``` The `telemetryapi` receiver captures Lambda platform telemetry including: * **Cold start spans** - Records function initialization time (`platform.initRuntimeDone`) * **Function logs** - Captures `stdout` and `stderr` from your Lambda function {% hint style="warning" %} Replace the following placeholders: * `{BYOC_ENDPOINT}` - Your groundcover BYOC endpoint * `{INGESTION_KEY}` - Your ingestion key * `{SERVICE_NAME}` - Your Lambda function or service name * `{ENVIRONMENT}` - Your environment (e.g., `production`, `staging`) Find your endpoint and ingestion key in [Settings > Ingestion Keys](https://app.groundcover.com/settings?selectedTab=ingestion-keys). {% endhint %} ### Creating a Configuration Layer Package the collector configuration as a Lambda layer for easy reuse: 1. Create a zip file with `otel-collector-config.yaml` at the root (not inside a directory) 2. Open the [Lambda Layers console](https://console.aws.amazon.com/lambda/home#/layers) 3. Click **Create layer** 4. Enter a name (e.g., `otel-collector-config`) 5. Upload your zip file 6. Select the compatible architectures and runtimes 7. Click **Create** {% hint style="info" %} Alternatively, you can include the config file directly in your function bundle at the root level, or store it in S3 and reference via URI (ensure your Lambda's IAM role has `s3:GetObject` permission). {% endhint %} ## Step 2: Add the Lambda Layers Your Lambda function needs three layers: 1. **OpenTelemetry Instrumentation Layer** - Auto-instruments your code (language-specific) 2. **OpenTelemetry Collector Layer** - Processes and exports telemetry 3. **Configuration Layer** - Contains your collector config (from Step 1) ### Layer ARNs Replace `` with your AWS region and `` with the latest version from the [OpenTelemetry Lambda releases](https://github.com/open-telemetry/opentelemetry-lambda/releases). {% hint style="warning" %} Layer versions are updated frequently. Always check the [releases page](https://github.com/open-telemetry/opentelemetry-lambda/releases) for the latest versions before deploying. {% endhint %} #### Instrumentation Layers | Runtime | Layer ARN | Wrapper Path | | ---------------- | ---------------------------------------------------------------------------------- | ---------------------- | | **Node.js** | `arn:aws:lambda::184161586896:layer:opentelemetry-nodejs-:1` | `/opt/otel-handler` | | **Python** | `arn:aws:lambda::184161586896:layer:opentelemetry-python-:1` | `/opt/otel-instrument` | | **Java Agent** | `arn:aws:lambda::184161586896:layer:opentelemetry-javaagent-:1` | `/opt/otel-handler` | | **Java Wrapper** | `arn:aws:lambda::184161586896:layer:opentelemetry-javawrapper-:1` | `/opt/otel-handler` | | **Ruby** | `arn:aws:lambda::184161586896:layer:opentelemetry-ruby-:1` | `/opt/otel-handler` | #### Collector Layer | Architecture | Layer ARN | | ------------------- | -------------------------------------------------------------------------------------- | | **x86\_64 (amd64)** | `arn:aws:lambda::184161586896:layer:opentelemetry-collector-amd64-:1` | | **arm64** | `arn:aws:lambda::184161586896:layer:opentelemetry-collector-arm64-:1` | {% hint style="warning" %} Make sure the collector layer architecture matches your Lambda function's architecture. {% endhint %} ### Adding Layers via AWS Console 1. Open the [Lambda Console](https://console.aws.amazon.com/lambda/) 2. Select your function 3. In the **Layers** section, click **Add a layer** 4. Select **Specify an ARN** 5. Paste the appropriate ARN 6. Click **Verify**, then **Add** 7. Repeat for all three layers (instrumentation, collector, configuration) ### Adding Layers via Terraform ```hcl resource "aws_lambda_function" "my_function" { function_name = "my-function" runtime = "python3.12" handler = "index.handler" architectures = ["x86_64"] layers = [ # Instrumentation layer - check releases for latest version "arn:aws:lambda:us-east-1:184161586896:layer:opentelemetry-python-:1", # Collector layer (match your architecture) - check releases for latest version "arn:aws:lambda:us-east-1:184161586896:layer:opentelemetry-collector-amd64-:1", # Your configuration layer aws_lambda_layer_version.otel_config.arn ] } resource "aws_lambda_layer_version" "otel_config" { layer_name = "otel-collector-config" filename = "otel-config-layer.zip" compatible_runtimes = ["python3.12"] } ``` ### Adding Layers via AWS CDK ```typescript import { Function, Code, Runtime, LayerVersion, Architecture } from 'aws-cdk-lib/aws-lambda'; const fn = new Function(this, 'MyFunction', { code: Code.fromAsset('lambda'), runtime: Runtime.PYTHON_3_12, handler: 'index.handler', architecture: Architecture.X86_64, }); // Add instrumentation layer - check releases for latest version fn.addLayers(LayerVersion.fromLayerVersionArn( this, 'OtelInstrumentationLayer', 'arn:aws:lambda:us-east-1:184161586896:layer:opentelemetry-python-:1' )); // Add collector layer - check releases for latest version fn.addLayers(LayerVersion.fromLayerVersionArn( this, 'OtelCollectorLayer', 'arn:aws:lambda:us-east-1:184161586896:layer:opentelemetry-collector-amd64-:1' )); // Add your configuration layer fn.addLayers(LayerVersion.fromLayerVersionArn( this, 'OtelConfigLayer', 'arn:aws:lambda:us-east-1:123456789012:layer:otel-collector-config:1' )); ``` ## Step 3: Configure Environment Variables Set the following environment variables on your Lambda function: | Variable | Value | Description | | ------------------------------------ | ----------------------------------- | --------------------------------- | | `AWS_LAMBDA_EXEC_WRAPPER` | See wrapper path in the table above | Enables automatic instrumentation | | `OPENTELEMETRY_COLLECTOR_CONFIG_URI` | `/opt/otel-collector-config.yaml` | Path to collector config | {% hint style="info" %} The collector config can also be loaded from your function bundle (`/var/task/otel-collector-config.yaml`) or from S3 (`s3://.s3..amazonaws.com/config.yaml`). {% endhint %} ## Step 4: Deploy and Test 1. Deploy your Lambda function with the layers and configuration 2. Invoke the function (via AWS Console, CLI, or trigger) 3. Check groundcover for incoming telemetry data {% hint style="info" %} Allow 1-2 minutes for data to appear in groundcover after the first invocation. {% endhint %} ## Complete Example ### Collector Config (`otel-collector-config.yaml`) ```yaml receivers: otlp: protocols: grpc: endpoint: localhost:4317 http: endpoint: localhost:4318 telemetryapi: exporters: otlphttp/groundcover: endpoint: https://example.platform.grcv.io compression: gzip headers: apikey: "your-ingestion-key" x-groundcover-service-name: "my-python-lambda" x-groundcover-env-name: "production" x-groundcover-source: "otel-lambda" service: pipelines: traces: receivers: - otlp - telemetryapi exporters: - otlphttp/groundcover metrics: receivers: - otlp exporters: - otlphttp/groundcover logs: receivers: - otlp - telemetryapi exporters: - otlphttp/groundcover ``` ### Terraform Configuration ```hcl # Configuration layer resource "aws_lambda_layer_version" "otel_config" { layer_name = "otel-collector-config" filename = "otel-config-layer.zip" compatible_runtimes = ["python3.12"] } # Lambda function resource "aws_lambda_function" "my_python_lambda" { function_name = "my-python-lambda" runtime = "python3.12" handler = "index.handler" filename = "lambda.zip" role = aws_iam_role.lambda_role.arn timeout = 30 memory_size = 256 architectures = ["x86_64"] layers = [ # Check https://github.com/open-telemetry/opentelemetry-lambda/releases for latest versions "arn:aws:lambda:us-east-1:184161586896:layer:opentelemetry-python-:1", "arn:aws:lambda:us-east-1:184161586896:layer:opentelemetry-collector-amd64-:1", aws_lambda_layer_version.otel_config.arn ] environment { variables = { AWS_LAMBDA_EXEC_WRAPPER = "/opt/otel-instrument" OPENTELEMETRY_COLLECTOR_CONFIG_URI = "/opt/otel-collector-config.yaml" } } } ``` ## Enriching Telemetry Data groundcover supports enriching ingested data with additional context using HTTP headers in the collector configuration. Add enrichment headers in the exporter configuration: ```yaml exporters: otlphttp/groundcover: endpoint: https://{BYOC_ENDPOINT} compression: gzip headers: apikey: "{INGESTION_KEY}" x-groundcover-service-name: "my-lambda-function" x-groundcover-env-name: "production" x-groundcover-source: "otel-lambda" ``` ### Available Enrichment Headers | Header | Description | Example | | ---------------------------- | ---------------------- | ----------------------- | | `x-groundcover-service-name` | Service/workload name | `payment-processor` | | `x-groundcover-env-name` | Environment name | `production`, `staging` | | `x-groundcover-env-type` | Environment type | `lambda`, `vm` | | `x-groundcover-source` | Data source identifier | `otel-lambda` | For more details, see [Enriching 3rd Party Data](/integrations/data-sources/enriching-3rd-party-data.md). ## Sampling Configuration ### Always Sample (Development) ```bash OTEL_TRACES_SAMPLER=always_on ``` ### Ratio-Based Sampling (Production) Sample 10% of traces to reduce costs: ```bash OTEL_TRACES_SAMPLER=traceidratio OTEL_TRACES_SAMPLER_ARG=0.1 ``` {% hint style="info" %} For high-traffic Lambda functions, ratio-based sampling reduces costs while maintaining visibility. {% endhint %} ## Collecting Logs The `telemetryapi` receiver automatically captures your Lambda function's `stdout` and `stderr` output via the [Lambda Telemetry API](https://docs.aws.amazon.com/lambda/latest/dg/telemetry-api.html) and forwards them as logs to groundcover. To correlate logs with traces, use a logger that outputs trace context. For Python: ```python import logging from opentelemetry import trace logger = logging.getLogger(__name__) def handler(event, context): span = trace.get_current_span() trace_id = span.get_span_context().trace_id logger.info(f"Processing event", extra={"trace_id": format(trace_id, '032x')}) ``` ## Lambda Platform Metrics The OpenTelemetry layers capture application-level metrics. For Lambda platform metrics (invocation count, duration, throttles, etc.), use the [CloudWatch Metrics integration](/integrations/data-sources/aws/ingest-cloudwatch-metrics.md) to ingest the `AWS/Lambda` namespace. ## Troubleshooting ### Data Not Appearing in groundcover 1. **Check config path**: Verify `OPENTELEMETRY_COLLECTOR_CONFIG_URI` points to the correct location 2. **Verify credentials**: Confirm your ingestion key is valid 3. **Check Lambda logs**: Look for collector errors in CloudWatch Logs 4. **Enable debug logging**: Set `OPENTELEMETRY_EXTENSION_LOG_LEVEL=debug` 5. **Network connectivity**: Ensure Lambda can reach the groundcover endpoint (port 443) ### 403 Forbidden Errors * Verify your ingestion key is valid and not revoked * Ensure you're using a **Third Party** type ingestion key * Check the `apikey` header format in your collector config ### Architecture Mismatch If you see errors about missing libraries: * Ensure the collector layer architecture (`amd64` vs `arm64`) matches your function's architecture ## Additional Resources * [OpenTelemetry Lambda GitHub](https://github.com/open-telemetry/opentelemetry-lambda) * [OpenTelemetry Lambda Releases](https://github.com/open-telemetry/opentelemetry-lambda/releases) * [groundcover OpenTelemetry Integration](/integrations/data-sources/opentelemetry.md) * [groundcover Ingestion Keys](/use-groundcover/remote-access-and-apis/ingestion-keys.md) * [Ingest CloudWatch Metrics](/integrations/data-sources/aws/ingest-cloudwatch-metrics.md) - For Lambda platform metrics --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.groundcover.com/integrations/data-sources/aws/monitor-lambda-functions.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.