AWS Builder Center

Builder Center

Using Firehose as a Lambda Log Destination

Learn how to use Amazon Data Firehose as an AWS Lambda log destination to stream logs to S3 or other platforms for centralized delivery.

Series: Lambda Logging [and available options] (3 articles)

  1. 1
    Lambda Logging in AWS [Default]
  2. 2
    Moving Lambda Logs to a Centralized Location [S3]
  3. 3
    Using Firehose as a Lambda Log Destination This article

Sunny Bhambhani

Using Firehose as a Lambda Log Destination

The last logging option available in the Lambda configuration is Amazon Data Firehose.
 
 
This option becomes interesting when CloudWatch Logs is not the final destination and you want Lambda logs to keep moving through a delivery pipeline.
 
Instead of just storing logs in one place, Firehose is built to receive data, optionally transform it, and then deliver it to another system.

How Firehose works at a high level

The flow is fairly straightforward:
  • Ingest: choose where the incoming data comes from, such as Amazon Kinesis Data Streams, Amazon MSK, or Direct PUT
  • Transform: optionally process or enrich the data before delivery
  • Deliver: send the data to a destination such as Amazon S3, Splunk, Dynatrace, and others
 
That flexibility is the main reason Firehose is useful. It can sit between the source and the final storage or observability platform without needing you to build the delivery logic yourself.
 
For this example, I kept things simple and used Amazon S3 as the destination.
 
If you want to see the full list of supported destinations, AWS documents them here:
https://docs.aws.amazon.com/firehose/latest/dev/create-destination.html

The Firehose stream

Before sending Lambda logs to Firehose, I created a delivery stream with the following setup:
  • Source: Direct PUT
  • Destination: Amazon S3
  • ARN: arn:aws:firehose:eu-central-1:ACCOUNT_ID:deliverystream/PUT-S3-QdzyI
Make sure the status is Active after it is created.

Destination settings

The S3 delivery stream was configured like this:
  • S3 bucket: s3-lambda-logs
  • S3 bucket error output prefix: fh_err_
  • S3 bucket prefix: fh_
  • S3 bucket and error output prefix time zone: UTC
  • New line delimiter: Enabled
 
This setup is useful because it separates normal output from delivery errors and keeps the objects written to S3 organized under a predictable prefix.

Where the logs land in S3

After the stream is configured and Lambda starts sending logs through Firehose, the delivered files appear in S3 under a path like this:
Amazon S3 -> Buckets -> s3-lambda-logs -> fh_2026/ -> 10/ -> 11/ -> 12/
 
That structure makes it easier to find logs by date and process them later with downstream tools.

A quick note about delivery log classes

  • If you send logs to Amazon S3 or Amazon Data Firehose, Lambda requires a log group with the delivery log class.
  • That is an important detail because this log class is designed for delivery, not for the full CloudWatch Logs experience. In other words, you trade some CloudWatch features for simpler forwarding.
  • For example, delivery log class does not support standard CloudWatch Logs features such as Logs Insights and Live Tail.
  • It is also worth remembering that log events are only kept in CloudWatch Logs for two days when this class is used. That retention period is fixed and cannot be changed.

Function

The function remains the same which we discussed in the first article.
 
1
2
3
4
5
6
7
8
9
10
def lambda_handler(event, context):
print("Dummy Lambda output")
return {
"statusCode": 200,
"body": "Dummy Lambda output",
}


if __name__ == "__main__":
print(lambda_handler({}, None))

In this example:

  • A new log group is created named dummy_lambda_logs_to_firehose
  • I also created an IAM role for the Lambda function. The trust relationship and permissions are defined below:
    • 03_iam_role_trust_relationships.json
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "logs.amazonaws.com"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringLike": {
"aws:SourceArn": "arn:aws:logs:eu-central-1:ACCOUNT_ID:*"
}
}
}
]
}
 
  • 03_iam_role_permissiones.json
 
1
2
3
4
5
6
7
8
9
10
11
12
13
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"firehose:PutRecord",
"firehose:PutRecordBatch"
],
"Resource": "arn:aws:firehose:eu-central-1:ACCOUNT_ID:deliverystream/*"
}
]
}
 
Notes:
  • Log delivery to Amazon Data Firehose requires using a log group with the delivery log class. This enables efficient routing, but it does not support standard CloudWatch Logs features such as Logs Insights and Live Tail.

Final thought

If CloudWatch Logs is where you troubleshoot, S3 is where you archive, then Firehose is the option you use when logs need to travel.
 
For a simple demo, using Firehose with S3 keeps the architecture easy to understand.
  • In a larger environment, the same pattern can be extended to feed analytics, security, or observability platforms without changing the Lambda function itself.
 
Feel free to add your thoughts, happy learning!
# aws-lambda# ai-assisted# logging# cloud-operations

Series: Lambda Logging [and available options] (3 articles)

  1. 1
    Lambda Logging in AWS [Default]
  2. 2
    Moving Lambda Logs to a Centralized Location [S3]
  3. 3
    Using Firehose as a Lambda Log Destination This article
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article