Your AI Response Takes 45 Seconds. Should You Add SQS?
A slow LLM response doesn't automatically mean you need asynchronous architecture. Let's break down when streaming is the right answer—and when it's time to introduce queues and workflows.
Your AI Response Takes 45 Seconds. Should You Add SQS?
Your AI application works perfectly.
Until the model takes 45 seconds to respond.
Then someone says:
“API Gateway has a timeout. Just add SQS.”
Problem solved?
Not necessarily.
This is where AI architecture gets interesting.
Because a slow response doesn't always mean you need asynchronous processing.
Sometimes you need streaming.
Sometimes you need SQS.
And sometimes you're solving the wrong problem entirely.
⚡ First Question: Is This Interactive?
Imagine a user asks:
“Explain why my AWS deployment failed.”
They are sitting in front of your application waiting for an answer.
If the model takes 20–40 seconds to generate the response, making the user stare at a loading spinner isn't a great experience.
Instead:
1
2
3
4
5
6
7
8
9
10
11
User
↓
API Gateway
↓
Lambda
↓
Amazon Bedrock
↓
Token / response chunks
↓
User sees response progressively
That's a streaming problem, not necessarily a queue problem.
Amazon Bedrock supports streaming model responses through APIs such as
InvokeModelWithResponseStream, allowing applications to receive output incrementally instead of waiting for the entire response.And AWS API Gateway supports response streaming for REST APIs, specifically including generative-AI use cases and long-running operations. AWS documents streaming as a way to reduce time-to-first-byte and exceed the normal 29-second integration timeout in supported configurations.
So the first architectural question shouldn't be:
“How do I get around the timeout?”
It should be:
“Does the user actually need to wait for the complete result?”
🧠 Streaming vs. Asynchronous Processing
Here's how I think about it.
Interactive AI
1
2
3
4
5
6
7
8
9
10
11
User
↓
API Gateway
↓
Lambda
↓
Bedrock
↓
Streaming response
↓
User
Use this when the user is actively waiting for the result.
Examples:
- AI chatbot
- Coding assistant
- Resume assistant
- Customer support assistant
- Interactive AI agent
Background AI
Now imagine the user uploads:
500 PDF documents
and asks your application to:
- extract information,
- summarize them,
- classify them,
- generate embeddings,
- store results.
Do you really want the HTTP request to remain open?
Probably not.
A better pattern is:
1
2
3
4
5
6
7
8
9
10
11
User
↓
API
↓
SQS
↓
Lambda / Worker
↓
Bedrock
↓
S3 / DynamoDB
The API can respond quickly:
“Your job has been accepted.”
The processing continues independently.
That's where asynchronous architecture starts making sense.
🚦 My Rule of Thumb
I use a very simple mental model:
| Requirement | Architecture |
|---|---|
| User is waiting | Streaming |
| User needs progressive output | Streaming |
| Job can finish later | Async |
| Large document processing | Async |
| Batch AI processing | Async |
| Reliable retries required | Queue |
| Multiple dependent steps | Workflow |
| User needs an immediate answer | Don't start with SQS |
The point isn't that one architecture is universally better.
The workload decides.
🔥 But Here's Where It Gets Interesting
Suppose your AI agent does this:
1
2
3
4
5
6
7
8
9
10
11
12
13
User Question
↓
Retrieve documents
↓
Call Bedrock
↓
Call another API
↓
Analyze result
↓
Call Bedrock again
↓
Generate final answer
Now we're no longer dealing with a simple model invocation.
We're dealing with a workflow.
At that point, simply increasing timeouts may hide the real architectural problem.
You may need:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
┌───────────────┐
│ API Gateway │
└───────┬───────┘
↓
┌───────────────┐
│ Lambda │
└───────┬───────┘
↓
┌───────────────┐
│ Step Functions│
└───────┬───────┘
↓
┌────────┼────────┐
↓ ↓ ↓
Lambda Bedrock SQS
↓ ↓ ↓
└────────┼────────┘
↓
Result
AWS Step Functions is designed for workflows with multiple steps and can run Standard workflow executions for long durations.
Now you're not adding services because they're fashionable.
You're adding them because the problem changed.
🏗️ This Is What I Mean by “Earn Complexity”
I used this idea in my previous article:
Earn your architecture.
Here's what I mean.
Start:
1
2
3
4
5
API Gateway
↓
Lambda
↓
Bedrock
Don't immediately add:
1
2
3
4
5
6
7
8
SQS
Step Functions
EventBridge
DynamoDB
OpenSearch
ECS
Kubernetes
...
Instead, ask:
What problem am I solving?
Then let the architecture evolve.
Stage 1 — Simple AI Request
1
2
3
4
5
API Gateway
↓
Lambda
↓
Bedrock
Perfect for validating the idea.
Stage 2 — Streaming
1
2
3
4
5
6
7
API Gateway
↓
Lambda
↓
Bedrock
↓
Streaming
The user gets progressive output.
Stage 3 — Background Processing
1
2
3
4
5
6
7
API Gateway
↓
SQS
↓
Worker
↓
Bedrock
Now the workload doesn't need to remain connected to the user.
Stage 4 — Complex Workflow
1
2
3
4
5
6
7
8
9
API
↓
Step Functions
↓
┌─────────┬──────────┬─────────┐
│ │ │ │
Lambda Bedrock SQS Database
│ │ │ │
└─────────┴──────────┴─────────┘
Now complexity has been earned.
⚠️ One More Thing: Don't Confuse “Slow” With “Long-Running”
This distinction is easy to miss.
A response taking 35 seconds doesn't automatically mean it's a background job.
If the user is still waiting for the answer, you may simply have a latency problem.
But if the operation takes several minutes and the user doesn't need the result immediately, you have an asynchronous processing problem.
Those are different architectural problems.
And they deserve different solutions.
☁️ My AI Architecture Checklist
Before adding SQS, I ask:
1. Does the user need the result immediately?
Yes → consider streaming.
2. Can the job finish independently?
Yes → consider asynchronous processing.
3. Does it need retries?
Yes → a queue may help.
4. Does it have multiple dependent steps?
Yes → consider workflow orchestration.
5. Am I adding this service because I actually need it?
If not → don't add it yet.
🎯 The Bigger Lesson
Cloud architecture isn't about memorizing which AWS service solves which problem.
It's about recognizing the shape of the problem.
A chatbot waiting for a response has a different architecture requirement from a 10-minute document-processing pipeline.
Both may use Amazon Bedrock.
Both may use Lambda.
But they don't necessarily need the same architecture.
That's why I like this principle:
Don't architect for the architecture diagram. Architect for the user experience and workload.
💬 Now I'm Curious
Suppose you're building an AI chatbot where:
Bedrock sometimes takes 40–60 seconds to complete.
You have two options:
A.
Stream the response to the user.
B.
Put the request onto SQS and make the processing asynchronous.
Which would you choose?
And more importantly:
What would make you switch from A → B?
I'm especially interested in hearing how experienced AWS builders draw that boundary.
👇 Drop your reasoning in the comments.
There isn't one architecture that fits every workload.
The interesting part is deciding when the architecture needs to change.
Build simple.
Measure the workload.
Earn the complexity. 🚀
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article