My First VPC Architecture Disaster (And Everything I Learned From It)

I have a confession to make.

When I built my first real VPC architecture on AWS, I drew a diagram first. A proper one. With colour coding, flow arrows, subnet labels, and a legend. I even used a ruler.

I was a Network SPOC for a year. Diagrams were my love language. And this one was beautiful.

The architecture, however, was an absolute disaster.

Here is what I designed with complete confidence:

Three VPCs. One for production. One for development. One for shared services. Each with public and private subnets across two Availability Zones. A NAT Gateway in each (I had learned my lesson from Week 1 — at significant cost). Security groups for every layer. Route tables that I had colour-coded by function.

It looked like the kind of diagram they put on AWS certification study guides. I was proud of it.

I showed it to a colleague. She looked at it for thirty seconds and said: "How are these VPCs talking to each other?"

I said: "Through the internet, obviously."

She said: "...Why?"

I did not have a good answer.

Here is the thing about coming from on-premises networking: when you need two networks to talk to each other, you connect them. Physically. With a cable. Or a leased line. Or a very expensive piece of hardware that a vendor will try to upsell you on for three months.

In AWS, I assumed the same logic applied. Three VPCs. They are all mine. They can all see each other through the internet. Simple.

What I had actually built was three completely isolated islands with no secure way to communicate — and then routed all inter-service traffic out through the public internet and back in again.

My production database was, technically, accessible from the development environment. Via the public internet. With a security group I had labelled "temporary - fix later."

Reader, I want you to know that "temporary - fix later" is the most dangerous phrase in any engineer's vocabulary, regardless of which side of the on-premises/cloud divide you sit on.

Looking back, I had made three separate mistakes that combined into something spectacular.

Mistake One: I treated VPCs like physical networks.

In on-premises networking, two networks on the same switch fabric can route to each other easily. I assumed VPCs worked the same way — that being in the same AWS account meant they were somehow "nearby." They are not. Each VPC is completely isolated by default. Neighbouring VPCs have no idea the others exist, and they are fine with that.

Mistake Two: I had never heard of VPC Peering.

This is the part where I admit that I had been so focused on learning compute and storage concepts that I skipped an entire section of the AWS networking documentation. VPC Peering is exactly what it sounds like — a connection between two VPCs that allows them to route traffic to each other privately, without going through the public internet.

It is not automatic. You have to set it up. But once you do, your VPCs can communicate as if they are on the same private network.

I had built three isolated VPCs and then routed traffic between them through the public internet, when the private solution was sitting in the documentation three pages from where I had stopped reading.

Mistake Three: I had created a masterpiece.

The diagram was so detailed, so colour-coded, so well-labelled that I had convinced myself the architecture must be correct. The diagram looked professional, therefore the architecture was professional.

This is the trap of knowing how to present things clearly. A well-drawn bad architecture is still a bad architecture. It just takes longer to explain why.

My colleague did not say anything after I explained my reasoning. She just looked at me for a moment with the expression of someone who has seen this particular mistake before.

Then she pointed me to the VPC Peering documentation.

Four hours later, I had redesigned the architecture. The three VPCs now spoke to each other over private peering connections. The "temporary - fix later" security group had been deleted. The route tables made sense in a way they had not before.

The new diagram was less colourful. It had fewer arrows. The legend was shorter.

It was also actually correct, which — I have learned — is worth more than beautiful.

The deeper lesson was not about VPC Peering specifically. It was about how AWS thinks about isolation.

In on-premises networking, connectivity is the default. Everything is connected unless you explicitly block it. You build walls.

In AWS, isolation is the default. Nothing is connected unless you explicitly allow it. You build doors.

This is the mental shift that nobody tells you about when you move from networking to cloud. I had spent years building walls. I needed to start building doors.

Once I understood that AWS starts from "nothing can talk to anything" and works outward from there, the architecture decisions started making sense. VPC Peering, Transit Gateway, PrivateLink, security groups — they are all doors of different sizes, with different price tags, for different use cases.

My on-premises brain wanted to start with everything connected and lock down what needed to be private. My AWS brain needed to start with everything locked and open only what needed to communicate.

Same goal. Completely different starting point.

I now draw diagrams differently.

Before I draw a single subnet, I ask: "Who needs to talk to whom, and should that conversation be private or public?" I map the communication requirements first. The architecture follows.

My diagrams are less beautiful. They are also less wrong, which turns out to matter considerably more at 2 AM when something stops working.

If you are building your first multi-VPC architecture, here is the honest version of the advice:

Start small. One VPC. Get it right. Add complexity only when you have a specific reason for it.

And if two things need to talk to each other privately — please, for the love of your AWS bill — use VPC Peering. It is in the documentation. I have checked.

What was your first AWS architecture mistake? The one you look back on now and can finally laugh about? Tell me in the comments — I promise I will not judge you. I routed production traffic through the public internet because I didn't read three more pages of documentation.

We are all learning here. 👇