Beyond the Timeout: Architecting Resilient Asynchronous AI Agents with Amazon Bedrock, SQS, and AWS Lambda
Learn how to decouple long-running LLM inferences from synchronous API routes using Amazon SQS and AWS Lambda. Implement partial batch failure reporting and Dead Letter Queues to prevent API timeouts, control compute costs, and protect Bedrock rate quotas.
The most common trap developers fall into when building AI features is treating Large Language Model (LLM) inference like a standard database query. You send a prompt from your API backend, await the response, and return it to the client.
But LLMs are inherently non-deterministic in their latency. As generation length increases, Time To First Token (TTFT) and total generation time can easily spike. If you are routing this through Amazon API Gateway, you will inevitably hit the hard 29-second integration timeout. The connection drops, the user sees a 504 Gateway Timeout, but your backend might still be processing the LLM request—wasting expensive compute and API tokens.
To build production-grade AI agents, we must move from synchronous waiting to asynchronous event-driven pipelines.
The Asynchronous AI Architecture
Event-driven AI architecture using Amazon SQS to buffer LLM requests and prevent API timeouts.
The pattern is simple but robust:
- Ingest & Acknowledge: Your API (e.g., FastAPI or API Gateway directly) receives the user's prompt, generates a unique
taskId, pushes the payload to Amazon SQS, and immediately returns a202 Acceptedwith thetaskId. - Buffer & Isolate: SQS durably holds the request. If there are massive traffic spikes, SQS buffers the load, preventing Amazon Bedrock throttling (RateExceeded exceptions).
- Process & Generate: An AWS Lambda worker picks up the message, invokes Amazon Bedrock, and streams or stores the result into Amazon DynamoDB.
- Retrieve: The client polls an endpoint with the
taskId(or receives a WebSocket push) to get the final generated response.
Implementing the Worker Lambda (Python)
When wiring SQS to Lambda, failure isolation is critical. If a batch contains 5 messages and 1 fails (e.g., due to a malformed prompt or a transient Bedrock timeout), you do not want the Lambda to throw a blanket exception and reprocess the 4 successful messages.
We solve this using Partial Batch Failure Reporting (
ReportBatchItemFailures).Here is the robust Python implementation for the worker Lambda using Boto3:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
import json
import boto3
import os
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
bedrock_runtime = boto3.client('bedrock-runtime', region_name=os.environ.get('AWS_REGION', 'us-east-1'))
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table(os.environ.get('DYNAMODB_TABLE_NAME'))
def lambda_handler(event, context):
batch_item_failures = []
logger.info(f"Processing batch of {len(event['Records'])} AI requests.")
for record in event['Records']:
message_id = record['messageId']
try:
# 1. Parse the incoming SQS payload
body = json.loads(record['body'])
task_id = body['task_id']
user_prompt = body['prompt']
# 2. Construct the Bedrock API call
bedrock_payload = {
"anthropic_version": "bedrock-2023-05-31",
"max_tokens": 1000,
"messages": [
{"role": "user", "content": user_prompt}
]
}
# 3. Invoke Amazon Bedrock
response = bedrock_runtime.invoke_model(
modelId='anthropic.claude-3-haiku-20240307-v1:0',
contentType='application/json',
accept='application/json',
body=json.dumps(bedrock_payload)
)
response_body = json.loads(response['body'].read())
generated_text = response_body['content'][0]['text']
# 4. Save the generated result to DynamoDB
table.put_item(
Item={
'taskId': task_id,
'status': 'COMPLETED',
'result': generated_text
}
)
logger.info(f"Successfully processed task: {task_id}")
except Exception as e:
logger.error(f"Failed to process message {message_id}: {str(e)}")
# Append only the failed message ID for SQS targeted reprocessing
batch_item_failures.append({"itemIdentifier": message_id})
# 5. Return partial failures to SQS
return {
"batchItemFailures": batch_item_failures
}Why this pattern wins in production
Defeats API Timeouts: Your frontend users never see a hanging loading spinner that suddenly crashes into a 504 error. The backend acknowledges the request instantly, resulting in a snappy UX.
Built-in Throttling Protection: Foundation models have strict Tokens-Per-Minute (TPM) and Requests-Per-Minute (RPM) limits. By configuring the Lambda event source mapping's
BatchSize and MaximumConcurrency, SQS acts as a shock absorber. It trickles requests to Bedrock exactly at the rate your quota allows.Cost Control via DLQs: By pairing this with an SQS Dead Letter Queue (DLQ), poisoned prompts that consistently fail (e.g., triggering Bedrock's content moderation filters) are routed out of the pipeline after 3 retries. This prevents infinite retry loops that silently drain your AWS compute budget.
Building AI agents isn't just about crafting the perfect prompt; it's about engineering the infrastructure to handle the unpredictable nature of generative models. Decoupling the generation via SQS ensures your applications remain resilient, scale effortlessly, and handle failures gracefully.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article