The Transformation Diaries Week 5: Production Is Different… Apparently
Week 5 of The Transformation Diaries is here. This week, Narayan digs into EC2 sizing, RDS, 24×7 environments, NAT Gateways, VPC interface endpoints and backup policies, while asking a deceptively simple question: Does non-production really need this or did it simply inherit it? Week 5: Production Is Different… Apparently.
Series: The Transformation Diaries - 52 Weeks Journey (10 articles)
- …
- 5The Transformation Diaries Week 5: Production Is Different… Apparently This article
By Friday of the previous week, Narayan had started to understand the story behind Westbridge's growing AWS bill.
Some of it was good growth. More customers, more products and more workloads naturally meant more cloud consumption.
Some of it wasn't.
More than 100 NAT Gateways, 300+ VPC interface endpoints, surprisingly high CloudWatch costs and resources nobody could confidently explain had already given Narayan and Raghu plenty to investigate.
But one number kept catching Narayan's attention.
Non-production.
For something everyone insisted was different from production, it was doing a surprisingly good job of costing like it.
Monday Started with Non-Production
Narayan started Monday with a simple request.
“I want to understand how we build Dev and Test compared with Production.”
Raghu knew exactly who they needed.
“Talk to Tom.”
Tom Walker was one of Westbridge's Principal Cloud Engineers. He had spent years working across its AWS estate and knew an uncomfortable amount about what lived where.
A meeting appeared in Tom's calendar.
Non-Production Architecture Review.
Tom accepted it.
Reluctantly.
Meet Tom Walker
Tom arrived with his laptop and the unmistakable expression of an engineer who had been invited to a meeting to explain decisions made over several years in sixty minutes.
Narayan explained what they had found during the cost review.
Tom listened.
Then came the NAT Gateways.
The endpoints.
The oversized resources.
Tom leaned back.
“So we're doing one of those exercises where we find everything that's wrong with Westbridge's AWS Ecosystem?”
Narayan smiled.
“No. I'm trying to understand why it's there.”
“That's usually what people say before telling us it's wrong.”
Raghu wisely concentrated on his coffee.
Narayan understood the reaction.
Tom and his team had kept this environment running through migrations, incidents, deadlines, acquisitions and requests that often arrived with the word urgent attached. It was easy to look at an architecture years later and question it. It was harder to remember the constraints under which it had been created.
So Narayan didn't defend himself.
“Show me.”
Tom opened the first account.
That changed the conversation.
Production-Like or Production-Sized?
They picked one application and compared its environments.
Production made sense.
Multiple EC2 instances. Appropriate capacity. A resilient database configuration. Monitoring, backups and the usual supporting infrastructure expected for an important customer-facing workload.
Then Tom opened Test.
Narayan looked at the EC2 configuration.
Then back at Production.
Then at Test again.
“Same instance size?”
“Yep.”
They opened Dev.
Almost the same story.
Tom explained that the environments had intentionally been kept consistent.
“We want non-prod to be production-like. Otherwise you test something completely different from what you eventually deploy.”
Narayan agreed.
That was good engineering.
But he wasn't sure Westbridge was distinguishing between architecture and capacity.
“Same architecture makes sense,” Narayan said. “But does Test need the same compute capacity?”
Tom paused.
Narayan continued.
“Production-like or production-sized?”
That was a slightly different question.
A test environment might need the same operating system, runtime, application architecture and deployment pattern as Production.
It didn't automatically need the same number or size of EC2 instances.
Architectural parity didn't require capacity parity.
Tom looked at the utilisation graphs.
The silence lasted slightly longer this time.
Apparently Dev Doesn't Sleep
The next surprise was uptime.
A number of Dev and Test environments were running 24 hours a day, seven days a week.
Narayan looked at their utilisation overnight.
Almost nothing.
Weekends weren't particularly exciting either.
“Who uses this at three in the morning?”
Tom looked at the graph.
“AWS, apparently.”
It was the first time Narayan heard the sarcasm without the defensive edge.
There were legitimate reasons for some environments to remain available. Automated testing happened overnight. Integration environments had dependencies. Teams occasionally worked outside normal hours.
But those were exceptions.
Somehow, 24×7 had become the default.
AWS already provided several ways Westbridge could automate stopping and starting appropriate non-production resources. The technology wasn't particularly difficult.
The harder question was deciding which environments genuinely needed to be available all the time.
Narayan added another item to the list.
Non-prod operating hours define the requirement.
Even the Database Was Production-Like
Then they reached the databases.
One non-production RDS instance looked remarkably familiar.
Similar sizing.
Multi-AZ.
Running continuously.
Narayan looked at Tom.
“High availability?”
Tom nodded.
“For Dev?”
Tom looked at the configuration again.
“Developers are highly available.”
Raghu laughed.
There were environments where Multi-AZ absolutely made sense outside Production. Resilience testing, release validation or important shared integration environments could justify it.
But that wasn't the point.
The point was that nobody had deliberately decided which environments needed it.
They had inherited the production pattern.
Narayan wasn't suggesting they remove Multi-AZ everywhere.
He was suggesting they should be able to answer one simple question:
“What failure are we protecting this environment from, and does the business need us to?”
Tom nodded.
This time without an argument.
Production Had One Exit. Non-Prod Had Plenty
Then they returned to networking.
This was where things became genuinely interesting.
Production largely followed Westbridge's intended network model. Internet-bound traffic was routed towards a centralised egress and inspection layer, giving the organisation a smaller number of places from which outbound connectivity could be governed and monitored.
Non-production looked very different.
Across the estate, dozens of non-production VPCs had their own NAT Gateways and decentralised internet egress.
It explained a good portion of the 100+ NAT Gateways Narayan had found the previous week.
But cost was only half the issue.
Some of those paths potentially bypassed the central inspection and monitoring controls Westbridge expected internet traffic to traverse.
Narayan looked at the two architectures side by side.
“Which one is supposed to be the more controlled environment?”
“Production.”
“And which one uses centralised egress?”
“Production.”
Narayan pointed at Non-Prod.
“So why doesn't this?”
Tom stared at the diagram.
“I'm starting to understand why you keep asking annoying questions.”
Narayan smiled.
“Progress.”
Non-Prod Was More Secure Than Production
The VPC interface endpoints made even less sense.
Westbridge had more than 300 of them.
When Tom and Narayan started mapping where they were deployed, an unexpected pattern emerged.
The majority were in non-production environments.
Production used relatively few.
Narayan initially assumed there must be a security or compliance requirement requiring private connectivity to AWS services.
Tom went looking for it.
Nothing.
They checked again.
There was no broad written requirement stating that these non-production workloads needed private connectivity to those AWS services. Many of the VPCs weren't isolated from outbound connectivity anyway.
Tom opened the Terraform.
There it was.
The same collections of interface endpoints appeared repeatedly across non-production VPCs.
A reusable module.
Narayan smiled.
“Didn't we meet this module last week?”
“Unfortunately, I think you two are becoming friends.”
The irony was difficult to miss.
Production, the environment carrying real customer workloads had relatively few or almost no interface endpoints.
Non-production had hundreds of them, providing private connectivity that nobody could clearly demonstrate was required.
Non-production had somehow become more privately connected than Production and Westbridge was paying considerably more for the privilege.
Tom stared at the screen.
“So technically…”
Narayan knew what was coming.
“…we've made Dev more secure than Prod?”
Raghu interrupted.
“Let's not put that sentence in Sarah's inbox.”
For the first time all week, all three laughed.
“Seven Years.” “Why?” “Compliance.”
Backups were next.
Narayan expected this one to be straightforward.
It wasn't.
“What are our backup requirements for non-production?”
Tom gave the technically correct answer.
“Depends on the application.”
So they asked around.
One team said non-production should be backed up.
Another said it should use the default.
Someone else said backups needed to be retained for seven years.
Narayan stopped.
“Why seven?”
“Compliance.”
“Which compliance requirement?”
Nobody immediately knew.
They searched through policies and standards.
There were plenty of legitimate requirements around production data and regulated information.
What they couldn't immediately find was a policy saying every non-production environment needed seven years of backup retention.
That didn't mean seven years was wrong.
It meant they didn't yet know whether it was right.
Someone mentioned the default again.
Narayan shook his head.
“Default is a configuration. It's not a business requirement.”
Tom looked at him.
“I'm beginning to dislike your questions again.”
“Again?”
“Don't ruin the progress.”
Production Had Requirements. Non-Production Had Inherited Assumptions.
By Thursday afternoon, the pattern was becoming difficult to ignore.
Nobody at Westbridge had intentionally designed non-production to be expensive.
Most of the individual decisions had sensible origins.
Keep environments consistent.
Make databases resilient.
Protect the data.
Use private connectivity.
Keep environments available.
Standardise infrastructure through reusable code.
Every one of those statements sounded reasonable.
The problem appeared when reasonable principles became universal defaults.
Production architecture had been copied.
Modules had been reused.
Defaults had been inherited.
And somewhere along the way, “Non-production should be production-like” had started meaning “Build most of Production again.”
Narayan wrote three words on the whiteboard.
Need. Want. Inherited.
Tom looked at them.
“You really like those three words.”
“They're proving useful.”
Narayan added another question underneath.
What does this environment actually need to prove?
That was the question Narayan wanted Westbridge to start with. Did it need production-level availability and capacity? Did it really need to run 24×7? What were the actual requirements for backups, connectivity, monitoring and recovery?
Tom looked at the whiteboard. “You realise the answer to some of those might still be ‘same as Production’?”
“Absolutely.”
“You're not going to argue with us?”
Narayan smiled.
“Give me the requirement and I'll happily lose the argument.”
And that was really the point. Some non-production environments might genuinely need to look almost exactly like Production.
Westbridge just needed a better reason than ‘because Production does.’
By Friday, Tom Had Started Asking the Questions
On Friday morning, Tom appeared beside Narayan's desk carrying his laptop.
No meeting invite.
No architecture workshop.
“I found another one.”
Narayan looked up.
“Another what?”
Tom turned the screen around.
Another non-production AWS account.
Another collection of resources.
Another configuration that had apparently been inherited from somewhere else.
Narayan started looking through it.
Tom stopped him.
“Before you ask, no, I can't find a requirement for it either.”
Narayan smiled.
Something had changed during the week.
On Monday, Tom had thought Narayan was there to find faults.
By Friday, Tom was finding the questions himself.
Not because the architecture was terrible.
Not because the engineers who built it didn't know what they were doing.
But because Westbridge had started doing something organisations often stopped doing once standards, modules and processes became established.
It had started questioning its own assumptions.
Tom looked at the growing list.
“So what now? Another cost review?”
Narayan shook his head.
Over the last few weeks they had talked about operations, cost, security, networking, resilience, ownership and architecture.
The problems were becoming connected.
Raghu looked at the whiteboard.
“You're thinking Well-Architected, aren't you?”
Narayan nodded.
“I think we're finally ready to have that conversation.”
Tom closed his laptop.
“Six pillars?”
“Six pillars.”
Tom sighed.
“And I thought three words were bad.”
Narayan laughed.
Monday was going to be interesting.
Next week: Week 6 - The First Well-Architected Conversation.
Series: The Transformation Diaries - 52 Weeks Journey (10 articles)
- …
- 5The Transformation Diaries Week 5: Production Is Different… Apparently This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article