The Transformation Diaries Week 7: Six Pillars, Fifty Problems
Week 7 of The Transformation Diaries is here. This week, Westbridge planned to continue its Well-Architected review. Production had other plans. 😄 A routine firewall change exposes something much bigger: one CloudFormation stack, 50+ Security Groups, 50+ older applications, shared VPCs, and a blast radius nobody quite expected. The deployment was successful. Production disagreed. Week 7: Six Pillars, Fifty Problems
Series: The Transformation Diaries - 52 Weeks Journey (10 articles)
- …
- 7The Transformation Diaries Week 7: Six Pillars, Fifty Problems This article
The Transformation Diaries Week 7: Six Pillars, Fifty Problems
Narayan arrived at Westbridge on Monday expecting to continue the Well-Architected conversations from the previous week.
He had spent part of Sunday going through his notes. Six AWS Well-Architected pillars. The additional Value Realisation lens they had discussed. A growing list of observations that needed to be turned into something Westbridge could actually act on.
His first workshop was scheduled for 9:30.
At 9:12, it disappeared from his calendar.
The 11:00 meeting disappeared shortly afterwards.
Then the afternoon workshop.
Narayan looked across at Raghu.
"Either everyone has suddenly become very efficient, or I'm being avoided."
Raghu barely looked up from his laptop.
"Production."
That was enough.
Monday's Well-Architected Review Didn't Happen
Within minutes, the office had changed.
Priya was already on an incident bridge. Mark was coordinating with Operations. Sarah's team was checking whether there was any security involvement. Engineers who had been difficult to get into a workshop last week were suddenly all available on the same Teams call.
Just not for Narayan.
He messaged Tom.
Narayan: "I'm guessing our 10 o'clock isn't happening?"
A reply appeared almost immediately.
Tom: "Production has decided to run its own Well-Architected Review."
Narayan smiled.
Narayan: "How's it going?"
There was a pause.
Tom: "Poorly."
The humour stopped there.
One of Westbridge's older customer-facing applications was unavailable.
The application was running. The database was healthy. The load balancer was healthy. There had been no obvious infrastructure failure.
But requests between parts of the application were failing.
Something could no longer talk to something else.
And nobody yet knew why.
Production Is Down
Narayan stayed away from the incident bridge initially.
This wasn't the moment for an external consultant to arrive with a whiteboard and start asking philosophical questions about architecture.
The Westbridge teams knew their environment. They needed space to restore service.
Eventually, Raghu pulled him into the call.
Tom was sharing his screen.
"Application team says nothing changed."
Someone from Operations responded.
"Nothing changed on the instances."
"Database?"
"Healthy."
"Load balancer?"
"Healthy."
"Any deployments?"
"No application deployments."
Narayan listened.
Then someone mentioned that the Network Operations team had made a firewall change earlier that morning.
For a different application.
Priya stopped them.
"Different application?"
"Yes."
"Then why are we talking about it?"
Nobody answered immediately.
Tom started looking through the Security Groups.
A few minutes later he stopped scrolling.
"Found it."
"What?"
"A Security Group rule changed about twenty minutes before the alerts started."
Priya frowned.
"Nobody changed this application's Security Group."
Tom looked at the screen again.
"Exactly."
That changed the conversation.
Nobody Changed This Application
The Security Group wasn't managed by the application's Terraform.
In fact, it wasn't managed by the application team at all.
It belonged to an older CloudFormation stack maintained by Westbridge's Network and Firewall Operations team.
That wasn't immediately surprising.
Westbridge had been using AWS for years. Different generations of architecture had left different generations of automation behind them.
Most of the organisation had gradually standardised on Terraform.
This part hadn't.
Narayan asked the obvious question.
"How many Security Groups does this stack manage?"
Tom opened the stack.
There was a pause.
Then some more scrolling.
Then considerably more scrolling.
"More than fifty."
Narayan thought he'd misheard.
"Security Group rules?"
"No. Security Groups."
"And they're all for this application?"
Tom shook his head.
"Different applications."
"How many?"
Another pause.
"More than fifty."
Raghu leaned back.
"Well, at least we've found today's blast radius."
UPDATE_COMPLETE
The change itself had been straightforward.
The Network and Firewall Operations team had received a request to modify connectivity for one of the older applications. The change had gone through the CloudFormation stack that had been managing these controls for years.
CloudFormation had processed the update.
The stack status was green.
UPDATE_COMPLETE.
From CloudFormation's perspective, the deployment had succeeded.
Production had a slightly different opinion.
As the team worked through the change, they found that a Security Group relationship used by another application had been affected. What looked like a firewall change for one workload wasn't isolated to that workload.
The configuration was shared.
The ownership was shared.
And now the impact was shared too.
Someone on the call said what everyone was thinking.
"Can we just roll the stack back?"
Tom didn't answer immediately.
He looked again at the resources controlled by it.
More than fifty Security Groups.
More than fifty older applications.
Some of them with dependencies nobody on the incident call fully understood.
"I really don't want our recovery plan to be 'let's change fifty other things and see what happens.'"
Nobody argued.
The immediate fix was eventually isolated and connectivity restored.
Production came back.
The incident was over.
The questions weren't.
One Stack. 50+ Security Groups. 50+ Applications.
Later that afternoon, Narayan, Raghu and Tom went back through the architecture.
The CloudFormation template had not started life as a monster.
It had grown into one.
Years earlier, the logic had been understandable.
Firewall rules were considered a network responsibility. The Network and Firewall Operations team needed control. Security Groups were AWS's mechanism for controlling network access, so centralising their management had seemed sensible.
New applications arrived.
More Security Groups were added.
More rules followed.
More dependencies appeared.
Eventually, Westbridge had effectively recreated its traditional firewall operating model inside AWS.
Application teams requested connectivity.
A central network team implemented it.
And a central infrastructure stack accumulated the rules.
Narayan looked at the diagram.
"We've moved the firewall to AWS. I'm not sure we've moved the firewall operating model."
Tom nodded.
"There was a reason it was built this way."
"I believe you."
Tom looked slightly surprised.
Narayan continued.
"The question isn't whether the reason existed. It's whether the reason still exists."
That was becoming a familiar question at Westbridge.
The Firewall We Rebuilt in AWS
The incident had exposed more than a bad rule.
Westbridge's newer applications were increasingly managed differently.
Terraform was becoming the organisation's IaC standard. Application infrastructure was moving closer to the teams responsible for operating it. Pipelines, reviews and deployment practices were gradually becoming more consistent.
But here was a critical piece of production infrastructure operating under a different model.
Different tooling.
Different ownership.
Different change processes.
Different assumptions.
CloudFormation wasn't the problem.
There was nothing inherently wrong with CloudFormation because the rest of Westbridge happened to prefer Terraform.
The problem was standardisation.
Westbridge had moved forward, while parts of its production operating model had remained where they were.
The shared VPCs added another dimension.
More than fifty older applications lived across shared network environments, and security controls for many of them were coupled together.
That meant a change intended for one application could have consequences somewhere completely different.
Shared infrastructure had slowly become shared risk.
And suddenly the discussion from the previous week's Well-Architected review didn't feel theoretical anymore.
Shared VPC. Shared Security Groups. Shared Blast Radius.
Raghu drew the current model on the whiteboard.
At the top sat the Network and Firewall Operations team.
Below it sat the large CloudFormation stack.
Below that were Security Groups spreading across dozens of applications.
It was horizontal ownership.
One central layer reaching across everything.
Narayan drew another model beside it.
This one was vertical.
An application.
Its compute.
Its load balancer.
Its database.
Its application-specific Security Groups.
Its Terraform.
Its pipeline.
Its team.
That didn't mean every networking or security control suddenly belonged to application developers.
Westbridge would still need centrally managed network foundations, enterprise security policies and organisational guardrails.
But there was a difference between central governance and central ownership of every application rule.
Where practical, application-specific security controls could live with the application infrastructure that depended on them.
A change to Application A should not require modifying a stack containing the security configuration of Applications B through Z.
Tom looked at the two diagrams.
"So basically, smaller blast radius."
"That's one benefit."
"And if an application team changes its architecture, its Security Groups change with it."
"Potentially, yes."
Tom pointed at the existing stack.
"And we need to untangle that."
Narayan looked at the fifty-plus applications attached to it.
"Carefully."
CloudFormation Wasn't the Problem
The Well-Architected workshop finally happened on Wednesday.
It just wasn't the workshop Narayan had planned.
The incident was now sitting in the middle of it.
Operational Excellence was no longer a discussion about best practices. Westbridge had just experienced what happened when ownership, change processes and operational boundaries were unclear.
Security wasn't simply about whether the correct ports were open. It was also about how security controls were owned, changed, reviewed and governed.
Reliability wasn't simply Multi-AZ or redundant infrastructure. The blast radius of a routine change had just taken a production application offline.
Other findings from the broader review touched Performance Efficiency, Cost Optimisation and Sustainability.
Nobody needed another presentation explaining the pillars.
Westbridge was starting to see how they interacted.
A decision made for Security could affect Reliability.
A decision made for operational control could make change harder.
A decision that made sense seven years ago could become technical debt without anybody deliberately creating technical debt.
Narayan looked at the growing findings list.
Last week, six pillars had looked reasonably tidy on a whiteboard.
Reality was considerably less organised.
Six Pillars, Fifty Problems
By Thursday, the review had accumulated findings across the estate.
Some were significant.
Some were small.
Some could be fixed quickly.
Others, like a CloudFormation stack controlling Security Groups across more than fifty applications, could take months to safely unwind.
Tom looked at the list.
"How many have we got?"
Narayan checked.
"Thirty-seven."
Tom looked disappointed.
"That's annoying."
Narayan laughed. "Why?"
"I was hoping for fifty. 'Six Pillars, Thirty-Seven Problems' doesn't have the same ring to it."
But beneath the humour was a much harder question.
Which problems mattered most?
The production incident would obviously receive attention.
But should Westbridge prioritise the most technically imperfect architecture?
The biggest security risk?
The largest cost opportunity?
The biggest operational burden?
The fastest improvement?
Or the change that created the most value for the business?
The seventh pillar they had discussed the previous week was suddenly back in the conversation.
Not because Westbridge needed another checklist.
Because Value Realisation gave them another way to think about priority.
The goal wasn't to make every Well-Architected finding disappear.
The goal was to decide which changes would meaningfully improve Westbridge.
The incident made that distinction painfully real.
CloudFormation had reported:
UPDATE_COMPLETE.
Technically, the change had succeeded.
The business outcome had failed.
That was probably the simplest explanation of Value Realisation Narayan could have asked for.
A Successful Deployment, An Unsuccessful Outcome
By Friday, nobody was talking about whether Westbridge had thirty-seven findings or fifty.
They had something more useful.
Context.
They understood why some of the problems existed.
They had seen how Security, Reliability and Operational Excellence could collide during one seemingly routine change.
They had identified areas where architecture needed to change.
And they had started separating technical perfection from business priority.
Tom stood beside Narayan looking at the CloudFormation stack one more time.
"So we're going to fix the thing that caused the outage?"
"Yes."
Tom waited.
Narayan looked again at the fifty-plus Security Groups and the applications depending on them.
"Carefully."
Tom nodded.
"Good. Because I'd rather not have another Well-Architected Review like Monday's."
Narayan smiled.
Neither would he.
There was only one problem now.
Westbridge had spent seven weeks discovering what needed to change.
They couldn't change everything at once.
Someone had to decide what came first.
Next Week: The Transformation Plan That Survived Contact with Reality
Series: The Transformation Diaries - 52 Weeks Journey (10 articles)
- …
- 7The Transformation Diaries Week 7: Six Pillars, Fifty Problems This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article