AWS Builder Center

From AWS Labs to Production Thinking: 12 Lessons I Learned Building and Breaking Serverless Systems

Learn the services. Build the projects. Prepare for certifications. But after building and breaking multiple AWS architectures, I realized something. Knowing what Lambda, DynamoDB, SQS, API Gateway, CloudWatch, IAM, and RDS do is not enough. The real challenge is understanding: What happens when these services interact under failure, scale, security constraints, and real traffic? That became the biggest shift in my AWS learning journey.

1. Serverless Does Not Mean Stateless

One of my early architectural lessons was understanding the difference between stateful and stateless systems.
A stateful application may depend on information stored inside a particular machine or execution environment.
That creates problems when traffic moves between instances or when infrastructure scales.
A stateless architecture moves the required state outside the compute layer.
For example:
Client → API Gateway → Lambda
Lambda should not depend on one specific server's memory to understand the user's session.
Instead, state can be handled through services such as Cognito, DynamoDB, tokens, or other external stores.
The important lesson was:
Compute should be replaceable.
If one execution environment disappears, the application should still be able to continue.



2. Synchronous APIs Can Become Bottlenecks

One of my projects started with a simple synchronous architecture:
Client → API Gateway → Lambda → S3
The client waited while Lambda processed the request.
It worked.
But then I asked:
What happens when the workload becomes expensive and traffic suddenly increases?
That led me to redesign the workflow:
Client → API Gateway → SQS → Lambda → S3
Now the client receives a job identifier while processing happens asynchronously.
This changed the architecture from:
“Wait for the work to finish.”
to:
“Accept the work and process it safely in the background.”
That is one of the most useful patterns I learned for handling workloads that don't need an immediate response.



3. Scaling Lambda Does Not Automatically Mean Scaling Your Database

This became one of my most important serverless lessons.
Lambda can scale very quickly.
A relational database cannot necessarily accept thousands of new connections at the same rate.
Imagine 1,000 concurrent Lambda invocations trying to establish database connections.
The compute layer may scale successfully while the database becomes the bottleneck.
I tested this architecture using:
Lambda → RDS Proxy → RDS PostgreSQL
RDS Proxy allowed many Lambda requests to share a controlled database connection pool.
The result was a major architectural lesson:
Always analyze the scalability of the entire dependency chain, not just the compute layer.



4. Concurrency Is a Design Problem

Another experiment involved Lambda concurrency and throttling.
I intentionally pushed concurrency beyond the configured limit.
The function started returning:
HTTP 429 – Too Many Requests
Initially, I expected AWS X-Ray to show me what happened.
It didn't.
Why?
Because a throttled invocation can be rejected before Lambda actually starts executing the function.
That means there may be:
  • No application execution
  • No application logs
  • No X-Ray trace
CloudWatch's Lambda Throttles metric becomes critical.
This taught me something extremely important:
The absence of telemetry can itself be a troubleshooting clue.



5. Observability Is Not Just Logging

Before working deeply with CloudWatch, I thought observability mostly meant:
“Add logs.”
My projects changed that definition.
A production system needs different sources of evidence.
CloudWatch Logs can tell me:
What happened?
X-Ray can help answer:
Where did the request spend time or fail?
CloudTrail can answer:
Which AWS API action happened, and who/what performed it?
Metrics can answer:
How frequently is the problem occurring?
These tools complement each other.
Good observability isn't about producing more data.
It is about producing the right evidence to investigate a failure.



6. IAM Is Part of the Architecture

Some of my longest debugging sessions involved IAM.
At first, an error such as:
AccessDeniedException
looks like a simple permissions problem.
But AWS authorization can involve:
  • Identity policies
  • Resource-based policies
  • Trust policies
  • Service principals
  • Session permissions
  •  iam:PassRole 
  • Resource ARNs
  • Explicit denies
One project taught me an especially important distinction.
When CodeDeploy or another AWS service needs to use a role, the role's permissions alone are not enough.
The correct service must also be allowed to assume or pass that role.
The lesson:
IAM isn't something you configure after building the architecture. IAM is part of the architecture.



7. Design for Failure, Not Just Success

One of my favorite experiments was implementing a circuit breaker.
The basic idea:
CLOSED → OPEN → HALF-OPEN
When the downstream service is healthy:
🟢 CLOSED
Requests are allowed through.
When failures cross a threshold:
🔴 OPEN
Requests fail fast without continuously hitting the unhealthy dependency.
After a cooldown:
🟡 HALF-OPEN
A controlled request tests whether recovery has occurred.
This taught me that resilience isn't simply adding retries.
Sometimes the best action is:
Stop calling the broken service.



8. Retries Without Protection Can Make an Outage Worse

Retries sound helpful.
If a request fails, try again.
But imagine a downstream service is already overloaded.
10,000 clients retry simultaneously.
Now the unhealthy service receives even more traffic.
This is where patterns such as:
  • Exponential backoff
  • Dead-letter queues
  • Circuit breakers
  • SQS buffering
  • Idempotency
become important.
The objective isn't simply:
“Make the request succeed.”
The objective is:
“Recover without creating a larger failure.”



9. Idempotency Is Critical in Distributed Systems

Distributed systems can deliver the same message more than once.
If the same order-processing request is executed twice, the result could be disastrous.
I implemented idempotency using DynamoDB conditional writes.
The basic idea was:
  1. Receive request.
  1. Generate or extract an idempotency key.
  1. Attempt to create the key.
  1. If it already exists, reject the duplicate.
  1. Process only the first valid request.
This changed the way I think about APIs.
A request isn't necessarily executed exactly once simply because the client sent it once.
Distributed systems require you to design for duplicates.



10. Caching Is About Reducing Work

Another project focused on performance.
Instead of allowing every request to reach the database, I explored:
Client → CloudFront → Application → Redis → Database
The goal wasn't simply to make the application “faster.”
The deeper goal was:
Do not perform expensive work when the answer is already available.
I also learned that caching introduces its own engineering problems:
  • Cache invalidation
  • TTL selection
  • Cache keys
  • Stale data
  • Cache misses
  • Consistency
Caching is powerful.
But poorly designed caching can create a different class of problems.



11. Infrastructure as Code Changed My Deployment Mindset

Initially, creating AWS resources manually felt straightforward.
Create the service.
Configure it.
Test it.
Move on.
But manual infrastructure becomes difficult to reproduce.
CloudFormation and AWS SAM changed that process.
Now infrastructure becomes:
Code → Review → Deploy → Test → Repeat
If something breaks, I can inspect the stack instead of trying to remember every console setting.
IaC isn't only about automation.
It gives infrastructure:
  • Repeatability
  • Version control
  • Consistency
  • Reviewability
  • Recoverability



12. The Most Important Lesson: Break Your Own Systems

This is probably the biggest lesson from my AWS journey.
A successful deployment doesn't prove that a system is production-ready.
So I started intentionally introducing failures.
I tested scenarios such as:
  • Lambda throttling
  • IAM permission removal
  • Database pressure
  • Function timeouts
  • Deployment failures
  • Circuit breaker activation
  • Dead-letter queue behavior
  • Invalid requests
  • Broken integrations
The question changed from:
“Does my application work?”
to:
“What happens when something stops working?”
That is a completely different engineering mindset.



What Changed After These Projects?

When I started learning AWS, I focused heavily on services.
Now I focus more on relationships.
I don't just ask:
“What does Lambda do?”
I ask:
“What happens to Lambda when traffic spikes?”
I don't just ask:
“How do I connect Lambda to RDS?”
I ask:
“What happens to database connections when Lambda scales to hundreds or thousands of concurrent executions?”
I don't just ask:
“How do I deploy an application?”
I ask:
“How do I deploy it, monitor it, roll it back, and troubleshoot it when the deployment fails?”
And I don't just ask:
“How do I secure the application?”
I ask:
“Can the infrastructure itself prevent an unauthorized request from reaching the data?”
That shift—from service knowledge to systems thinking—has probably been the most valuable part of my AWS journey.



Final Thought

I started this journey because I wanted to learn AWS and prepare for certifications.
But the projects gradually became something bigger.
I began building architectures.
Then I started testing them.
Then I started breaking them.
Then I started investigating why they broke.
And eventually, I started designing them with those failures in mind.
That's when AWS stopped feeling like a collection of cloud services.
It started feeling like engineering.
I'm still learning.
There are still hundreds of concepts I need to understand.
But every project is helping me move from:
“I know how to use AWS.”
toward:
“I understand how to engineer systems on AWS.”
And that's the skill I'm ultimately trying to build.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article