Beyond Field Redaction: Customize AWS WAF Logs with Lambda Transformation

What if you only need 40% of those fields? Learn how to use Lambda transformation with Kinesis Data Firehose to create custom JSON logs, reduce storage costs by 50-80%, and simplify AWS WAF log analysis.

chethan

📌 Empowering Business: Cloud Network & Edge Services📌 🎯 Subject Matter Expert: WAF & Route 53 🎯

Published Feb 6, 2026

Last modified Jun 7, 2026

Introduction

AWS WAF (Web Application Firewall) generates detailed logs that capture extensive information about web requests and the rules that evaluate them. While this rich data is valuable for security analysis, different organizations have different needs—some may require only specific fields tailored to their business requirements and investigation workflows. AWS WAF provides a field redaction feature that masks sensitive data, but what if you want to go further—either removing fields entirely or creating a custom JSON structure with only the fields relevant to your specific use case?

🎯 Benefits

💰 Reduce storage costs by 40-80%
⚡ Faster log analysis with cleaner data
🛠️ Custom schemas for your SIEM tools

Understanding AWS WAF Field Redaction

Before diving into custom solutions, it's important to understand what AWS WAF offers natively.

Built-in Field Redaction:

AWS WAF includes a field redaction feature that allows you to mask specific fields in your logs. You can redact:

URI path
Query string
Single header
HTTP method

When redacted, these fields appear as REDACTED in the logs instead of showing actual values. For example, if you redact the Query string field, all rules using the Query string match component will show REDACTED in the logs.

🔍 When Field Redaction Isn't Enough?

Field redaction is excellent for masking sensitive data while maintaining log structure, but it has limitations:

Redacted fields still exist in the log structure—Value masked but Key namespace field still present.
You cannot remove entire field categories like ruleGroupList, ja3Fingerprint, or labels
You cannot create a custom JSON structure with only your required fields

If your goal is to reduce log volume, simplify log analysis, or create a custom schema optimized for your specific use case, you need a different approach.

💡 One Solution: Lambda Transformation with Kinesis Data Firehose

AWS WAF support Data Firehose as a logging destination, you can enable Lambda transformation to customize records before they're stored in Target. This approach allows you to:

✅ Remove unnecessary fields entirely

✅ Keep only specific fields you need

✅ Restructure the JSON to match your requirements

✅ Reduce storage costs by decreasing log volume

✅ Simplify downstream analysis with cleaner data

Note: While this article focuses on the Kinesis Data Firehose + Lambda approach, other solutions exist, such as post-processing with S3 Event Notifications + Lambda, AWS Glue ETL jobs, or Amazon Athena CTAS queries. The Firehose transformation approach is especially beneficial when streaming to SIEM tools or when immediate cost savings are required.

How It Works

The Lambda function intercepts each log record, transforms it according to your specifications, and returns the modified record to Firehose for delivery to S3.

Option 1: Keep Only Specific Fields

This approach creates a new JSON structure containing only the fields you explicitly specify. It's the most efficient method for reducing log volume and creating clean, purpose-built logs.

Use case: Security Team only need timestamp, action, rule ID, URI, and HTTP method for investigations.

Implementation Guide

Step 1: Create the Lambda Transformation Function

Here's a sample Lambda function that decode the payload and keep only required fields.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
import json
import base64

def lambda_handler(event, context):
    output_records = []
    
    for record in event['records']:
        try:
            # Decode the base64 encoded data from Firehose
            payload = base64.b64decode(record['data']).decode('utf-8')
            
            # Parse JSON
            log_entry = json.loads(payload)
            
            # Create custom log with only required fields
            cleaned_log = {
                'timestamp': log_entry.get('timestamp'),
                'action': log_entry.get('action'),
                'terminatingRuleId': log_entry.get('terminatingRuleId'),
                'terminatingRuleType': log_entry.get('terminatingRuleType'),
                'webaclId': log_entry.get('webaclId'),
                'httpRequest': {
                    'uri': log_entry.get('httpRequest', {}).get('uri'),
                    'httpMethod': log_entry.get('httpRequest', {}).get('httpMethod'),
                    'country': log_entry.get('httpRequest', {}).get('country'),
                    'host': log_entry.get('httpRequest', {}).get('host')
                }
            }
            
            # Convert back to JSON and encode
            cleaned_payload = json.dumps(cleaned_log) + '\n'
            encoded_data = base64.b64encode(cleaned_payload.encode('utf-8')).decode('utf-8')
            
            # Return successful transformation
            output_records.append({
                'recordId': record['recordId'],
                'result': 'Ok',
                'data': encoded_data
            })
            
            print(f"Successfully processed record {record['recordId']}")
            
        except json.JSONDecodeError as e:
            print(f"JSON decode error for record {record['recordId']}: {str(e)}")
            output_records.append({
                'recordId': record['recordId'],
                'result': 'ProcessingFailed',
                'data': record['data']
            })
            
        except Exception as e:
            print(f"Error processing record {record['recordId']}: {str(e)}")
            output_records.append({
                'recordId': record['recordId'],
                'result': 'ProcessingFailed',
                'data': record['data']
            })
    
    print(f"Processed {len(output_records)} records")
    return {'records': output_records}

Step 2: Configure Lambda Function

Create a new Lambda function with Python 3.x runtime
Set appropriate timeout: Recommended 3-5 minutes
Configure memory: Start with 512 MB and adjust based on log volume
Create an IAM role with permissions for CloudWatch Logs

Step 3: Enable Lambda Transformation in Firehose

Open your Kinesis Data Firehose delivery stream
Navigate to Configuration → Transform and convert records
Enable Data transformation
Select your Lambda function
Set *Buffer size (*1-3 MB recommended) and Buffer interval (60-300 seconds)
Save changes

Step 4: Test and Monitor

Test with sample data using the Lambda console
Monitor CloudWatch Logs for transformation errors
Check S3 output to verify the custom log format
Monitor Firehose metrics**** for processing failures

Example output:

1
2
3
4
5
6
7
8
9
10
11
12
13
{
"timestamp": 1699564800000,
"action": "BLOCK",
"terminatingRuleId": "RateLimitRule",
"terminatingRuleType": "RATE_BASED",
"webaclId": "arn:aws:wafv2:us-east-1:123456789012:regional/webacl/example/a1b2c3d4",
"httpRequest": {
"uri": "/api/login",
"httpMethod": "POST",
"country": "US",
"host": "example.com"
}
}

Option 2: Remove Specific Fields

If you prefer to start with the complete WAF log structure and remove only certain fields, this approach works well when you need most of the standard fields but want to eliminate specific categories.

Use case: You want standard WAF logs but without detailed rule group information and specific Header fields to reduce log size moderately.

Sample Lambda Code Core Logic:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# Parse JSON
log_entry = json.loads(payload)

# Remove ruleGroupList
log_entry.pop('ruleGroupList', None)

# Remove nonTerminatingMatchingRules
log_entry.pop('nonTerminatingMatchingRules', None)

# Remove specific headers (Content-Type and Accept-Encoding)
if 'httpRequest' in log_entry and 'headers' in log_entry['httpRequest']:
    log_entry['httpRequest']['headers'] = [
        header for header in log_entry['httpRequest']['headers']
        if header['name'] not in ['Content-Type', 'Accept-Encoding']
    ]

# Convert back to JSON
cleaned_payload = json.dumps(log_entry) + '\n'

Conclusion

While AWS WAF's built-in field redaction is valuable for masking sensitive data, Lambda transformation with Kinesis Data Firehose unlocks powerful customization capabilities. You can:

Create minimal JSON structures with only required fields
Significantly reduce storage costs
Simplify log analysis and investigation
Maintain flexibility to adjust fields as requirements evolve

The solution integrates seamlessly with your existing WAF logging infrastructure. By choosing the right transformation option for your use case, you can optimize both cost and operational efficiency while maintaining the security visibility you need.

Additional Resources:

# aws-waf # logging # aws-lambda # cost-optimization # aws-edge-services

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

Beyond Field Redaction: Customize AWS WAF Logs with Lambda Transformation

chethan

Introduction

🎯 Benefits

Understanding AWS WAF Field Redaction

Built-in Field Redaction:

🔍 When Field Redaction Isn't Enough?

💡 One Solution: Lambda Transformation with Kinesis Data Firehose

How It Works

Option 1: Keep Only Specific Fields

Implementation Guide

Option 2: Remove Specific Fields

Conclusion