AWS Builder Center

Builder Center

The 4-Cent AWS Bill That Exposed Hidden Serverless Costs

The 4-Cent AWS Bill That Exposed Hidden Serverless Costs

Deep Dive into Lambda Event Source Mapping polling behavior, which uncovered hidden costs affecting thousands of serverless systems

Harpreet Siddhu

AWS Community Builder | AWS Road to re:Invent Hackathon Champion

The Discovery That Started Everything

While monitoring my personal AWS account, I discovered something that challenged my understanding of serverless economics. Despite processing fewer than 100 messages across my three SQS queues, I was being charged for exceeding my SQS free-tier limit.
The bill? Just 4 cents for SQS. But the implications were enormous.
This seemingly trivial charge led me to uncover a critical cost pattern in serverless architectures that affects hundreds of developers building event-driven systems every day. What I found reveals a fundamental gap between the serverless promise of "pay-per-use" and the operational reality of Lambda Event Source Mappings.


My Setup

My architecture was a typical serverless pattern I use for testing AI-based insurance quote processing workflows:
Architecture:
SNS SQS and Lambda Architecture Diagram

  • SNS topics feeding into SQS queues
  • Lambda functions connected via Event Source Mappings
  • Three queues:  home-insurance-quotes ,  life-insurance-quotes , and  vehicle-insurance-quotes 
The Unexpected Reality: When I analyzed my CloudWatch metrics, here's what I discovered:
Queue
Empty Receives/Hour
Monthly Total
home-insurance-quotes
402
289,440
life-insurance-quotes
636
457,920
vehicle-insurance-quotes
~500
360,000
Total
~1,538
1,107,360
The Math That Revealed the Problem:
  • 1,107,360 SQS API calls for processing just 97 actual messages
  • 107,360 requests over the 1M free tier limit
  • $0.04 charge from polling overhead alone
  • 11,415:1 ratio of empty polls to actual message processing

The Technical Reality I Uncovered

AWS Lambda Internal Architecture showing ESM Polling Workers

This research revealed how Lambda Event Source Mappings actually operate behind the scenes:

Continuous Polling Architecture

When you create an ESM between Lambda and SQS, AWS Lambda service initiates dedicated polling processes that:
AWS architecture diagram zoomed into Event Source Mapping behavior

  1. Poll continuously, regardless of queue activity, based on short or long polling.
  1. Generate ReceiveMessage API calls even for completely empty queues
  1. Scale polling frequency based on internal algorithms
  1. Maintain 24/7 polling to ensure low-latency message processing

My Real-World Measurements

Using CloudWatch metrics analysis:
Plain text
# Command I used to investigate each queue
aws cloudwatch get-metric-statistics \
  --namespace AWS/SQS \
  --metric-name NumberOfEmptyReceives \
  --dimensions Name=QueueName,Value=home-insurance-quotes \
  --start-time 2024-XX-01T00:00:00Z \
  --end-time 2024-XX-31T23:59:59Z \
  --period 3600 \
  --statistics Sum
Key Findings:
  • Consistent polling rates: 402-636 empty receives per hour per queue
  • 24/7 operation: Polling continued even during zero-activity periods
  • Predictable costs: Each queue contributed ~$0.01-0.02 monthly in polling overhead

Why This 4-Cent Discovery Matters

While 4 cents seems trivial, I found out a pattern with significant implications:

The Hidden Infrastructure Layer

Serverless architectures include several "always-on" components that can confuse you with the "pay-per-use" only service model:
  • Event Source Mappings (continuous polling) - used for services like Kinesis, DynamoDB Streams, and SQS, act as "always-on" components.
  • VPC networking (persistent connections when used) - underlying networking, VPC Endpoints, can carry, at minimum, hourly or per-GB data processing charges, even if the Lambda is not running.
  • CloudWatch Logs (storage accumulation) - By default, CloudWatch Logs are set to "Never Expire," meaning logs accumulate indefinitely. While you only pay for the ingestion and storage of logs, the accumulation of old, unnecessary logs can lead to significant, persistent storage costs.

Scaling Impact Analysis

Single Developer Impact:
  • 3 queues = $0.04/month in polling overhead
  • Minimal but measurable cost for zero business value
Small Team Impact:
  • 3 environments × 3 queues = $0.36/month
  • 9x cost multiplication for dev/test/prod
Enterprise Impact:
  • 10 microservices × 3 environments × 2 queues each = $2.40/month per service
  • $24/month across 10 services just for polling overhead
  • Compounds with architectural complexity
The issue isn't the 4-cent cost—it's the lack of visibility into this operational behavior. Most developers are unaware that ESMs poll continuously regardless of message volume.

Extended Cost Impact Across All ESM-Supported Services

While my research focused on SQS, it's important to understand that this polling cost pattern extends across all seven AWS services that support Lambda Event Source Mappings.
According to AWS documentation, the following services use the same continuous polling architecture:
Services Using Event Source Mappings:
  • Amazon SQS (Simple Queue Service) - $0.40 per million requests
  • Amazon Kinesis Data Streams - $0.014 per million GetRecords requests
  • Amazon DynamoDB Streams - $0.02 per 100,000 read request units
  • Amazon MSK (Managed Streaming for Apache Kafka) - Polling overhead on cluster compute costs
  • Self-managed Apache Kafka - Network and compute costs for continuous polling
  • Amazon MQ (ActiveMQ/RabbitMQ) - Polling costs based on broker instance hours
  • Amazon DocumentDB - Change stream polling costs
Compound Cost Implications:
For organizations using multiple ESM-supported services, the polling costs compound significantly. While the charges remain relatively small, they represent a hidden operational tax on serverless architectures that most development teams are completely unaware of.
More importantly, this pattern reveals a gap between serverless marketing and operational realities—costs that exist regardless of actual business logic execution.

My Solution Framework

Based on this research, I developed practical approaches to manage these hidden costs:

1. Cost Visibility and Monitoring

Critical Metric Monitoring:
Plain text
import boto3
from datetime import datetime, timedelta

def analyze_esm_polling_costs():
    cloudwatch = boto3.client('cloudwatch')
    
    # Get empty receives for the last 30 days
    response = cloudwatch.get_metric_statistics(
        Namespace='AWS/SQS',
        MetricName='NumberOfEmptyReceives',
        StartTime=datetime.utcnow() - timedelta(days=30),
        EndTime=datetime.utcnow(),
        Period=86400,
        Statistics=['Sum']
    )
    
    total_empty_receives = sum(point['Sum'] for point in response['Datapoints'])
    free_tier_usage = (total_empty_receives / 1000000) * 100
    excess_cost = max(0, (total_empty_receives - 1000000) * 0.0000004)
    
    return {
        'total_empty_receives': total_empty_receives,
        'free_tier_usage_percent': free_tier_usage,
        'monthly_polling_cost': excess_cost,
        'approaching_limit': free_tier_usage > 80
    }

2. Architectural Alternatives

For Low-Volume Use Cases: After discovering this pattern, I migrated to SNS → Lambda direct integration:
SNS Lambda Architecture Diagram

Results:
  • Eliminated 1,107,360 monthly SQS API calls
  • Reduced monthly costs by $0.04 (100% of polling overhead)
  • Maintained event-driven architecture benefits
  • Simplified operational monitoring
Note: For high-burst or critical workloads where you need to control consumption rate, SQS is still better, but for simple, low-volume alerts or events, the direct integration is a superior, optimized pattern for cost savings.

3. Development Environment Controls (Sample Template)

Conditional ESM Activation (Environment-Aware Infrastructure) :
Plain text
# SAM template with intelligent ESM management
Parameters:
  Environment:
    Type: String
    AllowedValues: [dev, test, prod]
    
Conditions:
  EnableESM: !Or
    - !Equals [!Ref Environment, prod]
    - !Equals [!Ref Environment, test]  # Only when actively testing
  
Resources:
  EventSourceMapping:
    Type: AWS::Lambda::EventSourceMapping
    Condition: EnableESM
    Properties:
      EventSourceArn: !GetAtt ProcessingQueue.Arn
      FunctionName: !Ref ProcessorFunction

4. When SQS + ESM Is Justified

Based on my cost analysis, SQS with ESMs is justified when you need:
  • Dead letter queue functionality - SQS provides robust DLQ capabilities with configurable retry counts and message retention (SNS DLQs are more limited and require SQS queues anyway)
  • Message persistence and durability - SQS stores messages reliably until processed, while SNS is fire-and-forget with no message persistence
  • Visibility timeout and retry control - SQS allows fine-grained control over message processing timeouts and automatic retry mechanisms
  • Decoupling with buffering - SQS provides true decoupling by buffering messages when consumers are unavailable (SNS requires all subscribers to be available)

The Broader Implications for Serverless Economics

This 4-cent investigation revealed fundamental challenges in serverless cost modeling:

The "Pay-Per-Use" Paradox

While Lambda functions follow true pay-per-execution pricing, supporting services introduce continuous operational costs:
  • ESM polling generates costs regardless of message volume
  • VPC networking maintains persistent connections
  • CloudWatch integration accumulates storage costs

Recommendations & Best Practices

Essential Monitoring and Visibility

  • Monitor the NumberOfEmptyReceives CloudWatch Metric
  • AWS identifies this as the primary indicator of polling cost issues
  • Set up CloudWatch alarms when the empty receives approach 80% of the service limits
  • Review this metric regularly across all ESM-supported services (SQS, Kinesis, DynamoDB Streams, MSK, MQ, DocumentDB)
  • Implement Cost Tracking
  • Use AWS Cost Explorer to track ESM-related charges separately
  • Tag Event Source Mappings for better cost allocation and analysis
  • Monitor trends to identify cost increases before they impact budgets

Configuration Optimization

  • Enable Long Polling (SQS)
  • Configure ReceiveMessageWaitTimeSeconds to 20 seconds (maximum)
  • AWS documentation confirms this "helps reduce the cost of using Amazon SQS by eliminating the number of empty responses."
  • Lambda automatically uses long polling for SQS Event Source Mappings
  • Optimize Batch Processing
  • Use appropriate batch sizes to reduce the number of API requests
  • Configure MaximumBatchingWindowInSeconds to balance latency vs. cost efficiency
  • Leverage batch operations (SendMessageBatch, DeleteMessageBatch) when possible

Lifecycle Management

  • Remove Unused Resources
  • Delete SQS queues, Kinesis streams, and other ESM sources when no longer needed
  • Implement automated cleanup for temporary or test resources

Implementation Checklist****

Before deploying any Event Source Mapping:
  • Configure appropriate polling settings (long polling for SQS)
  • Set up the NumberOfEmptyReceives monitoring and alerts
  • Implement proper batch sizing for your use case
  • Plan for lifecycle management and cleanup
  • Document expected costs and optimization strategies
  • Consider architectural alternatives and justify ESM usage

Tools and Implementation Guide

I have implemented these cost-optimization strategies, and I've created a collection of ready-to-use tools and sample templates. All code is available in my GitHub repository and can be adapted for any AWS environment.

Getting Started

Prerequisites:
  • AWS CLI configured with appropriate permissions
  • Python 3.10+ with boto3 installed
  • CloudWatch read permissions for SQS metrics
Quick Setup:
plaintext
# Clone the repository
git clone https://github.com/hsiddhu2/aws-esm-cost-optimizer.git
cd aws-esm-cost-optimizer

# Install dependencies
pip install -r requirements.txt

# Configure your AWS credentials
aws configure

Usage Examples

1. Analyze Your Current ESM Costs:
Plain text
# Run the cost analysis script
python esm_cost_analyzer.py

# Output example:
# Total empty receives: 1,107,360
# Free tier usage: 110.7%
# Monthly polling cost: $0.04
# Recommendation: Consider SNS direct integration
2. Set Up Monitoring Alerts:
Plain text
# Deploy CloudWatch alarms
aws cloudformation deploy \
  --template-file templates/esm-monitoring.yaml \
  --stack-name esm-cost-monitoring \
  --parameter-overrides AlertEmail=your-email@domain.com
3. Implement Conditional ESMs:
Plain text
# Deploy environment-aware ESM template
sam deploy \
  --template-file templates/conditional-esm.yaml \
  --stack-name insurance-quotes-processor \
  --parameter-overrides Environment=dev

Repository Structure

The complete toolkit includes:
  • Cost analysis scripts - Python tools for measuring ESM overhead
  • CloudFormation templates - Infrastructure as Code for monitoring
  • SAM templates - Serverless application examples with cost optimization
  • Migration guides - Step-by-step instructions for SNS integration
  • Monitoring dashboards - CloudWatch dashboard configurations

Community Contributions

This repository is open for community contributions. If you discover additional ESM cost patterns or optimization techniques, please submit a pull request. Together, we can help the AWS community build more cost-effective serverless architectures.
Repository:  https://github.com/harpreetsiddhu/aws-esm-cost-optimizer 

Conclusion

What started as a 4-cent billing anomaly became an investigation that revealed important patterns in serverless cost modeling. While the absolute cost was minimal, the discovery highlighted a critical gap between serverless promises and operational realities.
This investigation demonstrates that effective serverless cost management requires understanding not just functional behavior, but operational characteristics of the services we build upon. The "pay-per-use" promise remains valid, but we must understand what constitutes "use" in event-driven architectures.
By sharing this discovery and the solutions I developed, I hope to help other developers make more informed architectural decisions and build cost-effective serverless applications. Sometimes the most valuable insights come from the smallest unexpected charges in our AWS bills.

This investigation was conducted using my personal AWS account with real workloads running in us-east-1. All data and findings are based on actual CloudWatch metrics and billing analysis. The ESMs were subsequently disabled after direct SNS integration to save free-tier-usage and additional cost.
# aws-community# aws-lambda# serverless# cost-optimization# event-driven
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article