Ingest CloudWatch Logs Using AWS Firehose

This feature is only available for enterprise plan and, at this stage, requires public ingress endpoint enabled.

To be able to view your CloudWatch logs in the groundcover platform's Logs section, together with all other logs, and leverage the advanced capabilities of the platform to slice and dice your data, you'll need to follow these steps:

Setup a Firehose stream

1. Go to Amazon Data Firehose.

2. Click on Create Firehose stream

   1. Source: Direct PUT

   2. Destination: HTTP Endpoint

   3. Create a name for your stream, for example PUT-Groundcover-logs

   4. Destination settings:

      1. HTTP endpoint URL: Firehose logs endpoint, fetched using these docs

      2. Access key: groundcover API token, fetching using these docs

      3. Content encoding: GZIP

      4. Parameters:

         1. env_name - Specify your Environment name, it will show up in this environment in the application

   5. Backup settings:

      1. Choose a backup bucket, or create a new one.

3. Click Create Firehose stream

Create an IAM role and policy

1. Go to Amazon IAM

2. Click on Roles in the side bar

3. Click on Create Role

   1. Select Custom trust policy

   2. Paste the following policy:

      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Principal": {
              "Service": "logs.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
          }
        ]
      }

   3. Click on Next twice (we'll attach permissions later)

   4. Provide a name for the role

   5. Click on Create Role

4. Go to your newly created role

   1. In the Permissions section, click on Add permissions and then Create inline policy

   2. Click on JSON and paste the following:

      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": [
              "firehose:PutRecord",
              "firehose:PutRecordBatch"
            ],
            "Resource": "<YOUR_FIREHOSE_STREAM_ARN>"
          }
        ]
      }

   3. Click on Next

   4. Give the policy a name

   5. Click on Create Policy

Create a subscription filter

Now that we're all set up, we can add a subscription filter to the desired log group in CloudWatch. Using CLI

The following is an example of how to create a subscription filter through the AWS CLI:

  aws logs put-subscription-filter \
    --log-group-name "<GROUPONAME>" \
    --filter-name "<FILTERNAME>" \
    --filter-pattern "" \
    --destination-arn "<DESTINATIONARN>" \
    --role-arn "<ROLEARN>"

Using AWS Console

1. Go to the specific log group in CloudWatch and click on the Subscription filters tab.

   • Click on Create

   • Select Create Amazon Data Firehose subscription filter

2. Select the Firehose delivery stream created in the previous steps, as well as the IAM role.

3. Fill Configure log format and filters as you need.

4. Choose a name for the subscription filter, then click Start streaming.