Ingest Logs Stored on S3

This feature is only available for enterprise plan as part of inCloud Managed.

Many AWS services allow saving logs or other data in S3 buckets for long term storage. It can often be very useful to read that data into groundcover, which is what this page is all about.

groundcover uses a lambda function called groundcover-aws-ingester that uses AWS trigger mechanism to run the lambda whenever a new file is put inside a bucket. This is a common way to utilize lambdas for these types of tasks. You can read more about here.

List of supported AWS services

The following list of services are supported for ingestion. Note the Displayed Source column, which will be filterable in the platform under the source filter.

Service Name
Displayed Source

AWS ELB

aws-elb

AWS CloudTrail

aws-cloudtrail

AWS S3

aws-s3

Installation

groundcover uses a provided CloudFormation stack to deploy the lambda function. This is the easiest and recommended method to deploy the lambda function. It also takes care of adding granular permissions on the required resources such as access to the buckets and secret (if configured).

We support multiple ways of deploying the stack:

Required configuration

When setting up the stack you will need to provide the following values:

Endpoint details

Choose one of the options below to configure the groundcover endpoint details. Using environment is more simple and requires no other configuration, while using a secret requires creation of the secret beforehand.

  • GCAPIKey - the API key used to ingest data into groundcover. Can be retrieved using: groundcover auth print-api-key

  • GCSite - your inCloud Managed site, provided to you during installation. Example: example.platform.grcv.io

Target buckets

  • BucketARNs - a comma-separated list of buckets that you wish to consume logs from. Example: arn:aws:s3:::my_awesome_bucket,arn:aws:s3:::my_other_awesome_bucket

Additional attributes

  • GCEnvName (optional) - if provided, logs collected will be tagged with this value as env , making them filterable in the platform accordingly.

  • LambdaTimeout (optional) - The amount of time (in seconds) that Lambda allows a function to run before stopping it. The default is 30 seconds. The maximum allowed value is 900 seconds.

  • LambdaMemorySize (optional) - The amount of memory available to the function at runtime. Increasing the function memory also increases its CPU allocation. The default value is 128 MB. The value can be any multiple of 1 MB up to 10240 MB (e.g 128, 256, 512...)

Installing using the AWS Console

There are two ways to install using CloudFormation:

Installing using Terraform

(Optional) Creating a secret

This step is only needed if you prefer to use AWS Secret Manager to store the groundcover endpoint details. If you wish to pass them by env, skip to the next step.

variable "groundcover_api_key" {
  type        = string
  description = "groundcover API key"
}

variable "groundcover_site" {
  type        = string
  description = "groundcover site"
}

locals {
  groundcover_endpoint_configuration = {
    apikey = var.groundcover_api_key
    site   = var.groundcover_site
  }
}

resource "aws_secretsmanager_secret" "groundcover_endpoint" {
  name        = "groundcover_endpoint"
  description = "groundcover endpoint configuration"
}

resource "aws_secretsmanager_secret_version" "groundcover_endpoint" {
  secret_id     = aws_secretsmanager_secret.groundcover_endpoint.id
  secret_string = jsonencode(local.groundcover_endpoint_configuration)
}

output "groundcover_endpoint_secret_arn" {
  value = aws_secretsmanager_secret.groundcover_endpoint.arn
}

Creating the CloudFormation stack

See above section on the relevant configuration parameters

groundcover uses the Terraform resource aws_cloudformation_stack to deploy the CloudFormation stack directly inside Terraform.

locals {
  groundcover_aws_ingester_buckets = ["my_awesome_bucket", "my_other_awesome_bucket"]
  stack_name                       = "groundcover-aws-ingester"
}

resource "aws_cloudformation_stack" "groundcover_aws_ingester" {
  name         = local.stack_name
  capabilities = ["CAPABILITY_IAM", "CAPABILITY_NAMED_IAM", "CAPABILITY_AUTO_EXPAND"]
  parameters   = {
    GCSecretARN      = "arn:aws:secretsmanager:::DEFAULT", # optional  required if not passing GCSite and GCAPIKey
    GCSite           = "example.platform.grcv.io",         # optional, required if not passing GCSecretARN
    GCAPIKey         = "api-key",                          # optional, required if not passing GCSecretARN
    BucketARNs       = join(",", formatlist("arn:aws:s3:::%s", local.groundcover_aws_ingester_buckets)),
    GCEnvName        = "",                                 # optional
    LambdaTimeout    = 30,                                 # optional
    LambdaMemorySize = 128,                                # optional
  }
  template_url = "https://groundcover-public-cloudformation-templates.s3.us-east-1.amazonaws.com/CF_groundcover_aws_ingester.yaml"
}

Updating the CloudFormation stack

This step is not needed on initial deployment; It's only relevant when a new version of the lambda function is released and you wish to upgrade to it. If this is your first time deploying the function, move on to Adding triggers

  1. Access the existing stack created in the previous steps.

  2. Click on Update in the top right corner

  3. Click Next once again to retain the existing configuration values

  4. Continue clicking Next to update the stack

Adding triggers

After deploying the lambda function, you will need to add triggers.

groundcover supports multiple types of triggers:

SNS trigger

SNS triggers are added in two parts:

  • Adding notifications from the buckets to an SNS topic

    • You will need to do this for each bucket specified in the configuration

  • Adding a trigger to the Lambda from the SNS topic

    • You will need to do this for each SNS topic

groundcover supports multiple methods of setting up the SNS trigger:

Setting up SNS triggers using the AWS Console

Adding the SNS trigger to the lambda

  1. Access the groundcover-aws-ingester lambda function in the UI, and browse to the Add Trigger section.

    1. Select SNS as the trigger type.

    2. Enter the SNS topic to trigger the lambda on.

    3. Click on Add.

  1. On each bucket, create an event notification to the SNS topic

    1. Select All object create events as the Event types

    2. Select the SNS topic from (1) as the destination

    3. Click on Save changes

Setting up SNS triggers using Terraform

groundcover uses the Terraform resource aws_s3_bucket_notification as seen below:

locals {
  groundcover_aws_ingester_buckets = ["my_awesome_bucket", "my_other_awesome_bucket"]
  stack_name                       = "groundcover-aws-ingester"
}

resource "aws_sns_topic" "s3_events" {
  name = "s3_events_topic"
  policy = <<-POLICY
{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": [
        "SNS:Publish"
      ],
      "Resource": "*"
    }
  ]
}
  POLICY
}

data "aws_lambda_function" "groundcover_aws_ingester" {
  function_name = "GroundcoverAWSIngester-${local.stack_name}"

  depends_on = [aws_cloudformation_stack.groundcover_aws_ingester]
}

resource "aws_lambda_permission" "groundcover_aws_ingester_allow_sns" {
  statement_id  = "AllowExecutionFromSNS"
  action        = "lambda:InvokeFunction"
  function_name = data.aws_lambda_function.groundcover_aws_ingester.arn
  principal     = "sns.amazonaws.com"
  source_arn    = aws_sns_topic.s3_events.arn
}

resource "aws_s3_bucket_notification" "groundcover_aws_ingester_bucket_notification" {
  for_each = toset(local.groundcover_aws_ingester_buckets)
  
  bucket   = each.key

  topic {
    topic_arn     = aws_sns_topic.s3_events.arn
    events        = ["s3:ObjectCreated:*"]
  }

  depends_on = [aws_lambda_permission.groundcover_aws_ingester_allow_sns]
}

resource "aws_sns_topic_subscription" "groundcover_aws_ingester" {
  topic_arn = aws_sns_topic.s3_events.arn
  protocol  = "lambda"
  endpoint  = data.aws_lambda_function.groundcover_aws_ingester.arn
}

S3 trigger

You will need to provide triggers on each bucket specified in the configuration.

After deploying the lambda function, you will need to add triggers on the S3 buckets you want to read logs from. groundcover supports multiple methods of setting up the triggers:

Setting up S3 triggers using the AWS Console

Access the groundcover-aws-ingester lambda function in the UI, and browse to the Add Trigger section.

Make sure to keep the default value of Event types - All object create events

Click on Addto create the trigger.

Setting up S3 triggers using Terraform

groundcover uses the Terraform resource aws_s3_bucket_notification as seen below:

locals {
  groundcover_aws_ingester_buckets = ["my_awesome_bucket", "my_other_awesome_bucket"]
  stack_name                       = "groundcover-aws-ingester"
}

data "aws_lambda_function" "groundcover_aws_ingester" {
  function_name = "GroundcoverAWSIngester-${local.stack_name}"

  depends_on = [aws_cloudformation_stack.groundcover_aws_ingester]
}

resource "aws_lambda_permission" "groundcover_aws_ingester_allow_bucket" {
  for_each = toset(local.groundcover_aws_ingester_buckets)

  statement_id  = "AllowExecutionFromS3Bucket-${each.key}"
  action        = "lambda:InvokeFunction"
  function_name = data.aws_lambda_function.groundcover_aws_ingester.arn
  principal     = "s3.amazonaws.com"
  source_arn    = "arn:aws:s3:::${each.key}"
}

resource "aws_s3_bucket_notification" "groundcover_aws_ingester_bucket_notification" {
  for_each = toset(local.groundcover_aws_ingester_buckets)
  
  bucket   = each.key

  lambda_function {
    lambda_function_arn = data.aws_lambda_function.groundcover_aws_ingester.arn
    events              = ["s3:ObjectCreated:*"]
  }

  depends_on = [aws_lambda_permission.groundcover_aws_ingester_allow_bucket]
}

Viewing the logs

Access the groundcover platform logs page to view your logs. You can filter based on the source of the data (see table) or based on the env name, if set.

Last updated