AWS Builder Center

Builder Center

QueueMesh: Automated Serverless Backpressure for AWS Lambda & SQS

Built for the First Commit hackathon, QueueMesh is a zero-code, sub-second circuit breaker that dynamically throttles SQS-to-Lambda concurrency to protect downstream APIs from HTTP 429 retry storms.

Rajal

Building systems that combine applied AI with production-grade engineering.
This project was built for the Ship It track during the WeMakeDevs "First Commit" Hackathon.
Check out our 3-Minute Demo Video: https://www.youtube.com/watch?v=5yX5kDagP4I
GitHub Repository: https://github.com/Rajal-ui/QueueMesh

The "Infinite Scale" Problem

AWS Lambda and Amazon SQS are incredible because they scale infinitely. But what happens when the downstream services you rely on don't?
Imagine your SQS queue suddenly receives 10,000 messages. AWS Lambda rapidly scales out to hundreds of concurrent workers to process them. If those workers are making calls to a third-party API (like Stripe, Twilio, or SendGrid) or a legacy SQL database, that massive spike in traffic acts like a self-inflicted DDoS attack.
The downstream API gets overwhelmed and returns HTTP 429 (Too Many Requests). Because of standard SQS behavior, the failed messages go back into the queue for a retry. Your Lambdas keep scaling, the API keeps rejecting them, and you enter a destructive retry storm that leads to API key suspensions, DLQ (Dead-Letter Queue) spills, and 3 AM pager alerts for Site Reliability Engineers (SREs).
Currently, fixing this requires manual intervention to dial down the queue concurrency, which takes 15 to 30 minutes. We wanted to fix this in under a second.

Enter QueueMesh

QueueMesh is an automated, real-time backpressure controller for AWS event pipelines. It sits above your worker Lambdas like a smart circuit breaker.
It monitors downstream HTTP response codes in real-time. If it detects an HTTP 429 spike, it dynamically reaches into your AWS infrastructure and dials down the MaximumConcurrency of your SQS Event Source Mapping on the fly.
The best part? It requires ZERO changes to your application source code.

The AWS-Native Architecture

We built QueueMesh using a 100% serverless AWS stack deployed via AWS SAM:
  1. Amazon SQS & AWS Lambda (Worker): The core event pipeline. The worker processes the queue and calls the external API.
  2. CloudWatch Embedded Metric Format (EMF): To avoid slowing down the worker Lambda, we use CloudWatch EMF. The worker simply prints a JSON log to stdout containing the HTTP response codes (e.g., Downstream429Count). CloudWatch parses this asynchronously with zero performance penalty.
  3. AWS Lambda (Controller) & Boto3: The "brain" of QueueMesh. When CloudWatch detects an error spike, it triggers this Python 3.11 Lambda.
  4. Amazon DynamoDB: A single-table state ledger that tracks the live health of the queue (STATE#CURRENT) and logs immutable audit trails of every infrastructure change (AUDIT#TIMESTAMP).

The Core Engine: AIMD Math

QueueMesh uses the Additive Increase / Multiplicative Decrease (AIMD) algorithm—the same math that prevents congestion on the global internet (TCP/IP).
  • Emergency Drop (Multiplicative Decrease): If the 429 error rate exceeds 5%, the Controller cuts the SQS MaximumConcurrency by 50% instantly (e.g., 100 workers -> 50 -> 25).
  • Safe Ramp-Up (Additive Increase): Once the downstream API stabilizes and success rates hit 98%, QueueMesh safely adds 5 workers at a time until baseline capacity is restored.

Hackathon Execution & Learnings

During the hackathon, we deployed this live in us-east-1 and ran load tests. The results blew us away.
QueueMesh was able to detect an error spike, query DynamoDB, calculate the AIMD math, and execute the boto3.client('lambda').update_event_source_mapping API call in exactly 309 milliseconds. We successfully protected downstream APIs in less than half a second.
What we loved: CloudWatch EMF is a game-changer for serverless observability. Being able to emit custom telemetry via standard JSON logs without making blocking API calls makes ultra-fast infrastructure loops possible.
QueueMesh is open-source, cost-efficient (running almost entirely on the AWS Free Tier), and ready to protect enterprise pipelines. Let us know what you think in the comments!
#WeMakeDevs #Hackathon #AWS
# amazon-sqs# aws-sam# distributed-architecture# event-driven# aws-serverless-application-repository
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article