The Transformation Diaries Week 9: You Can't Transform What You Can't See
Welcome to Week 9 of The Transformation Diaries! One forgotten AWS Lambda. Years of unnecessary logging. One investigation that reduced CloudWatch costs by 60%. But the discovery raised a bigger question: What else are we paying for that nobody actually needs? Sometimes transformation isn't about building something new. It's about questioning what's already there! #TheTransformationDiaries #AWS #FinOps #CloudOptimisation #AWSAmbassador #AWSCommunityBuilder
Series: The Transformation Diaries - 52 Weeks Journey (10 articles)
- …
- 9The Transformation Diaries Week 9: You Can't Transform What You Can't See This article
Eight weeks into Westbridge's transformation, the team had finally agreed on what needed to change.
They had a roadmap. They had priorities. They even had something that looked suspiciously like agreement between Engineering, Operations, Security and Finance.
Now came the difficult part.
Actually doing it.
And as Narayan was about to discover, sometimes the biggest opportunities aren't hidden behind complicated architectures or expensive modernisation programmes.
Sometimes they're sitting quietly in an AWS account, doing absolutely nothing useful.
Twenty-two million times a month.
The First Week of Actual Transformation
Monday morning.
Tom Walker walked into the meeting room carrying his laptop and a coffee.
"Right," he said, dropping into his chair. "Eight weeks of workshops, reviews, diagrams and meetings. Are we finally allowed to touch AWS?"
Narayan smiled.
"We've been touching AWS for eight weeks, Tom."
"Yes, but mostly with PowerPoint."
Raghu laughed from the other side of the table.
For the first time since the engagement began, Westbridge had something resembling an executable transformation plan.
The priorities were reasonably clear.
Network and Security Simplification would address the sprawling VPC estate, the hundred-plus NAT Gateways, hundreds of interface endpoints and the legacy Security Group management model that had contributed to the recent production outage.
Engineering would begin identifying opportunities to improve infrastructure ownership and standardisation.
Operations would work with the application teams to establish a better understanding of the environment, including what was running, why it existed and what could safely be changed.
And Finance wanted to see results.
Preferably before the next budget meeting.
But Narayan had one concern.
"We've agreed on what we want to change," he said. "Before we start, we need to understand how we'll measure whether those changes are actually making things better."
Tom sighed.
"Please don't tell me we're scheduling another workshop."
"No. I want us to start with something we already know is costing money."
He opened the cost analysis from the previous weeks.
CloudWatch.
The logging costs had stood out during their earlier review. By Week 8, the team had narrowed the problem to excessive debug logging and identified it as a potential quick win.
Now they needed to establish the cause, understand the impact and confirm that any changes were safe.
Tom leaned forward.
"Finally. A problem that doesn't involve fifty Security Groups."
Raghu looked up.
"Give it time."
The CloudWatch Bill That Wouldn't Behave
Westbridge's AWS bill had plenty of interesting stories to tell.
Some were expected.
EC2 costs reflected a large estate of workloads that had accumulated over the years. RDS costs included databases provisioned for performance and availability requirements that hadn't always been revisited.
Networking costs were another story entirely, with NAT Gateways and interface endpoints distributed across dozens of VPCs.
But CloudWatch was different.
Its costs had grown to a level that didn't seem consistent with the organisation's monitoring requirements.
And while everyone agreed that logging was important, nobody could explain why it was costing so much.
Narayan opened AWS Cost Explorer and started breaking down the CloudWatch charges by usage type.
He wasn't particularly interested in the total cost at this stage.
He wanted to understand what was driving it.
CloudWatch includes several chargeable capabilities, including metrics, alarms, dashboards and Logs. Within CloudWatch Logs, ingestion and storage can become significant cost drivers, especially when applications produce large volumes of data.
The analysis pointed towards log ingestion.
A lot of log ingestion.
Narayan and Tom moved into CloudWatch Logs and began examining log groups, ingestion patterns and the services generating the highest volumes.
Some log groups were expected. Production applications, infrastructure components and security-related workloads naturally generated operational data.
But one log group looked particularly unusual.
Its name suggested a Lambda function associated with Westbridge's logistics platform.
And the volume of data it was producing was extraordinary.
Tom stared at the numbers.
"That's a lot of logs for one function."
Narayan opened the Lambda metrics.
"Wait until you see how often it's running."
Twenty-Two Million Invocations. For What?
22M+
Monthly Lambda invocations
S3
Event source
DEBUG
Logging level
Westbridge operated a large logistics and infrastructure business.
Across its distribution centres, warehouses and transport operations, thousands of shipments moved through its systems every day.
Every parcel scan, warehouse movement, shipment status update and delivery event contributed to the operational data flowing through the organisation.
Some of that information was captured as small event files uploaded to Amazon S3.
The process was entirely legitimate.
The files fed downstream systems responsible for shipment tracking, operational reporting and reconciliation.
And at Westbridge's scale, those events added up quickly.
Hundreds of thousands of files a day.
Millions every week.
More than twenty-two million in a month.
Narayan traced the Lambda's trigger to an S3 event notification configured to invoke the function whenever a new object was created under the logistics event prefix.
Every time a shipment event file arrived, the function ran.
And every time it ran, it generated logs.
Not just a simple message confirming that an event had arrived.
Detailed debug information, including event metadata and portions of the incoming payload.
Tom scrolled through the code.
"Okay. So what does it actually do with the file?"
Narayan continued reading.
"Nothing."
"Nothing?"
"It logs the event."
"And then?"
"That's it."
Tom sat back.
"Twenty-two million times?"
"More than twenty-two million."
"Well, at least it's consistent."
They looked further into the function's configuration.
There was no database update. No transformation of the event. No downstream API call.
The Lambda received an S3 event, produced verbose logs and completed successfully.
Over and over again.
The function was working exactly as written.
That was becoming the problem.
A Lambda Nobody Remembered
The function had an interesting name.
westbridge-logistics-event-debugger
That final word caught Narayan's attention.
Debugger.
He checked the deployment history, configuration and available documentation.
The Lambda appeared to have been created several years earlier, during an early integration exercise for the logistics event platform.
At the time, developers had been testing how shipment event files arrived in S3.
They needed a simple way to inspect incoming events, understand their structure and troubleshoot the integration.
Creating a Lambda that logged each event was a perfectly reasonable way to do that.
For a test.
The problem was that the test had never really ended.
The actual logistics ingestion platform had evolved. New components had been introduced. The production processing pipeline had matured.
But the original event notification and debugging function remained.
As shipment volumes increased, so did the number of invocations.
And because verbose debug logging was still enabled, the volume of logs increased with it.
The function had become a permanent passenger on a production data pipeline.
Nobody was actively using it.
Nobody was monitoring its business value.
And because it wasn't failing, nobody had much reason to investigate it.
Tom shook his head.
"So this thing has been running for years?"
"Looks that way."
"And it's not actually doing anything?"
"It's generating logs."
"Which we're paying for."
"Yes."
Tom paused.
"That's an impressively expensive way of proving S3 works."
Narayan laughed, but he wasn't ready to remove it.
The code suggested that the function wasn't doing anything useful.
That didn't prove nothing depended on it.
Someone might still be using those logs for troubleshooting. Another process might rely on information being captured there. Or the function might have an undocumented purpose.
They needed someone who understood the application.
Meet Justin Richards: The Developer Who Knew Where to Look
Justin Richards joined the call later that afternoon.
He was an application developer working with Westbridge's logistics engineering team.
Unlike some of the infrastructure discussions Narayan had been involved in, Justin didn't begin by defending an architecture or explaining why something couldn't be changed.
He simply asked what they'd found.
Narayan shared his screen.
"We've traced a significant amount of CloudWatch logging to this Lambda. It's being invoked more than twenty-two million times a month."
Justin leaned closer.
"Logistics event debugger?"
"Yes."
Justin smiled.
"Oh, that's old."
Tom looked at him.
"How old?"
"Very old."
"Technical term?"
"Officially, yes."
Justin explained that the logistics platform had gone through several changes over the years.
The original integration used S3 to receive shipment event files from warehouse and transport systems. During testing, developers had used simple event listeners to inspect incoming files and validate the payload format.
The current processing pipeline no longer relied on that debugging function.
But Justin didn't assume it was safe to remove simply because it looked obsolete.
Together, they traced the S3 event notification, reviewed the Lambda's permissions and checked its behaviour against the current ingestion architecture.
Justin also confirmed with the application team that nobody depended on the debug logs for operational reporting or troubleshooting.
The actual event-processing pipeline had its own logging, monitoring and error-handling mechanisms.
The old Lambda was simply receiving a copy of the S3 event.
Justin opened the function's code.
"Look at this. It logs the entire event, reads the object metadata, prints the debug payload and exits."
Tom looked at Narayan.
"Twenty-two million times."
Justin laughed.
"You're going to keep saying that, aren't you?"
"Until somebody explains why we're paying for it."
Narayan asked Justin a question.
"If we remove the S3 notification for this function, what breaks?"
Justin considered it.
"Based on what we've checked, nothing in the current application. But I'd like to validate the event configuration and run the normal regression checks before we change production."
"Exactly what I was hoping you'd say," Narayan replied.
Tom looked disappointed.
"I was hoping he'd say we could delete it immediately."
"That's why you're not allowed near the delete button," said Raghu, who had joined the call halfway through.
The team followed the change process, validated that the legitimate S3 ingestion notifications remained intact, removed the obsolete Lambda's event subscription and disabled the function.
They monitored the production pipeline afterwards.
Shipment events continued to arrive.
The application continued processing them.
Operational monitoring remained healthy.
And one very busy Lambda finally stopped working.
For the first time in years.
One Forgotten Function. Sixty Percent Less CloudWatch Spend.
The impact was almost immediate.
The log group that had been generating millions of unnecessary debug messages suddenly went quiet.
And the numbers told an extraordinary story.
A 60% reduction in Westbridge's organisation-wide CloudWatch costs. Approximately USD 80,000 in annualised savings.
From disabling one forgotten Lambda.
Tom stared at the cost report.
"Wait. We just saved eighty thousand dollars a year?"
"That's what the numbers suggest," Narayan replied.
"By turning off one Lambda?"
"Yes."
Tom leaned back.
"Remind me why we spent eight weeks creating a transformation roadmap?"
Even Narayan laughed.
But then another thought crossed his mind.
The Lambda had been running for years. As shipment volumes grew, so did its unnecessary logging costs.
If it was costing USD 80,000 annually today, how much had Westbridge spent on it over the years?
The historical figure could be significantly higher, although nobody had calculated it yet.
Justin smiled.
"So, no refactoring? No new architecture? No six-month modernisation programme?"
"Not this time," Narayan replied.
"Are we sure we're doing transformation correctly?" Tom asked.
But there was an important distinction.
Twenty-two million invocations weren't the problem. Twenty-two million unnecessary invocations were.
The legitimate shipment processing continued without interruption. Only the obsolete debugging function had stopped.
The team would validate the annualised savings against subsequent billing data, but the reduction in log ingestion was already clear.
One forgotten Lambda. Twenty-two million unnecessary invocations a month. USD 80,000 in annual savings.
Sometimes the biggest transformation wins don't come from building something new.
They come from questioning why something old is still running.
And that raised a much bigger question.
What else was Westbridge paying for that nobody actually needed?
If One Lambda Was Forgotten, What Else Have We Forgotten?
The result quickly reached Daniel Foster.
The CFO had spent much of the previous weeks asking why Westbridge's AWS bill kept growing despite repeated optimisation efforts.
For once, there was a measurable improvement he could point to.
Rachel arranged a short meeting with Daniel, Mark, Priya, Sarah, Raghu, Narayan and the engineering team.
Daniel opened with a simple question.
"Just so I understand this correctly. We had a function running twenty-two million times a month, generating logs nobody needed, and we've been paying for it for years?"
Narayan nodded.
"That's what the investigation found."
Daniel looked around the room.
"And nobody knew?"
Justin answered before anyone else could.
"The function was created for a legitimate reason. The application changed over time, but the old component wasn't retired. It's not something you'd necessarily notice from the application's behaviour."
Daniel nodded.
He wasn't looking for someone to blame.
He was trying to understand how it could happen.
"Okay. Then I have another question."
Raghu leaned back.
"I have a feeling I know where this is going."
Daniel looked at Narayan.
"How many more do we have?"
The room went quiet.
Not because anyone knew the answer.
Because nobody did.
Mark spoke first.
"We've got hundreds of applications and years of infrastructure changes. There will be things out there that aren't being used the way they were originally intended."
Priya added, "And not everything unused is expensive. Some things cost very little. Others could be creating significant costs or operational risks."
Sarah raised another concern.
"Cost isn't the only issue. What permissions did that Lambda have? Was it logging information we shouldn't have been retaining? What other old components still have access they no longer need?"
That changed the conversation.
The forgotten Lambda wasn't just a cost optimisation story anymore.
It raised questions about lifecycle management, security, ownership, observability and technical debt.
Narayan looked at the group.
"We've spent years getting better at deploying infrastructure. But how good are we at knowing when something should stop existing?"
Tom glanced at Justin.
"Apparently, not twenty-two-million-times-a-month good."
Even Daniel laughed.
But the question stayed with them.
What else was Westbridge running simply because nobody had ever asked whether it was still needed?
The Cloud Doesn't Come with a S
pring-Cleaning Service
The discussion continued into the afternoon.
Westbridge had spent years making it easier to provision infrastructure. Developers could experiment, deploy and build without waiting weeks for hardware.
But there was one thing nobody had made equally easy.
Getting rid of things when they were no longer needed.
Tom started listing examples from their AWS estate.
Unattached EBS volumes. Forgotten snapshots. Development environments running 24/7. Test databases that had quietly become permanent. Old Lambda functions, unnecessary logging and resources nobody remembered creating.
And, of course, the hundred-plus NAT Gateways and three hundred-plus VPC interface endpoints they'd already discovered.
"Sounds like we've got plenty to delete," Tom said.
"Not necessarily," Narayan replied.
A rarely used disaster recovery component might be critical. A monthly reconciliation job might appear idle for twenty-nine days but be essential on the thirtieth.
Low utilisation doesn't automatically mean low value.
"The question isn't whether something is running," Narayan explained. "It's whether it still serves a purpose, and whether we're delivering that purpose efficiently."
Justin nodded.
"When developers build something, we create test resources, enable debug logging and try different approaches. That's normal."
"And then?" Priya asked.
"We move on to the next feature. Sometimes the temporary stuff doesn't."
That was the real issue.
Westbridge had processes for creating infrastructure, but very few for reviewing and retiring it.
Narayan suggested making lifecycle management part of normal engineering practices: clear ownership, regular reviews, sensible logging and retention standards, cost anomaly detection, and safe decommissioning.
Not another one-off cleanup exercise.
A continuous practice.
Tom looked at the whiteboard.
"So we're adding decommissioning to our engineering standards?"
"Yes."
"Fantastic. More work."
Raghu smiled.
"Think of it as fewer things to maintain later."
Tom paused.
"That's annoyingly reasonable."
Cloud optimisation wasn't just about reducing costs. It was about regularly questioning whether the environment still reflected what the business actually needed.
The Network Project Wasn't Quite So Simple
The CloudWatch investigation had delivered a relatively straightforward win.
The network simplification project was proving more complicated.
Tom and Raghu had been reviewing the NAT Gateways, interface endpoints and legacy Security Group arrangements identified during the earlier assessments.
The inventory was available.
They could see which accounts and VPCs contained the resources, how they were configured and what they were costing.
But understanding their purpose was another matter.
Some NAT Gateways supported workloads with well-understood outbound connectivity requirements.
Others appeared to exist because older VPC templates had deployed them by default.
Several interface endpoints had been created consistently across non-production environments, even when there was no clear evidence that those environments required private connectivity to the relevant AWS services.
And the legacy CloudFormation stack controlling Security Groups across dozens of older applications remained a particularly sensitive area.
Nobody wanted another outage.
Tom highlighted one NAT Gateway.
"This one's barely showing any traffic. Could be an easy candidate."
Narayan looked at the associated route tables.
"Do we know what uses it?"
"Not yet."
"Then it's a candidate for investigation, not deletion."
Tom sighed.
"I miss the Lambda."
Raghu laughed.
"The Lambda was easy because Justin knew what it did, and we could verify nothing depended on it."
That was the difference.
For the network estate, they had technical inventory but incomplete dependency information.
VPC Flow Logs could help reveal traffic patterns, and configuration data could show how routing and network controls were set up.
But those tools couldn't automatically explain why a particular connection existed or whether it supported a critical business process.
A low-traffic network path might be unnecessary.
Or it might support an important monthly integration.
A Security Group might appear overly permissive.
Or it might be supporting an undocumented dependency between older applications.
And a VPC endpoint might look redundant until someone discovered a workload that genuinely required private connectivity.
The team agreed to build a more dependable view of network dependencies before making changes.
They would combine AWS configuration data, traffic evidence, IaC repositories and application knowledge.
They would identify candidates, validate them with the relevant teams and sequence changes carefully.
The network project hadn't stalled.
It had simply reached the point where assumptions were no longer good enough.
Narayan looked at Tom.
"We're still going to simplify this environment."
"I know."
"We're just not going to break it while doing so."
Tom smiled.
"That's a surprisingly unpopular requirement in some projects."
You Can't Transform What You Can't See
By Friday, Westbridge had its first measurable win of Act II.
One forgotten Lambda disabled. Twenty-two million unnecessary monthly invocations eliminated. Approximately USD 80,000 in annualised savings.
And something arguably more important.
An organisation-wide conversation about what else might be hiding in plain sight.
But the week had exposed a bigger problem.
Westbridge wasn't short of tools. It had CloudWatch, CloudTrail, AWS Config, Cost Explorer, IaC repositories and monitoring platforms.
There was plenty of data. What was missing was the connection between that data and the business.
CloudWatch could show which Lambda generated the logs. Cost Explorer could show what they cost. S3 notifications could explain why the function kept running.
But none of them could answer the most important question.
Was it still needed?
That required someone like Justin, who understood the application, its history and its purpose.
The network simplification project was proving harder precisely because those answers weren't always available.
Tom joined Narayan as he was closing his laptop.
"Not a bad week."
"No, not bad."
"One Lambda down. Only several thousand resources left to question."
Narayan laughed.
Tom glanced at the whiteboard, still covered with application names and dependencies.
"Surely someone owns all these applications?"
Narayan looked at him.
"That's what we need to find out."
"Please tell me that's not another eight-week exercise."
"Hopefully not."
"Your confidence is inspiring."
Eight weeks ago, Westbridge was asking what needed to change.
Now it was learning that knowing what exists isn't the same as understanding why it exists.
And before you can modernise, optimise or retire something, you need to understand its purpose.
You can't transform what you can't see.
And seeing something doesn't mean you understand it.
Next Week: Who Actually Owns This Application?
The CloudWatch investigation had succeeded because Narayan and Tom found the right person.
Justin knew the application, understood the history and could help establish whether the Lambda was still needed.
But as Westbridge started reviewing other workloads, the team encountered a different problem.
Some applications had changed hands several times.
Some had been inherited through organisational changes.
Others had technical contacts, but nobody clearly accountable for their business purpose, cost or lifecycle.
And a few seemed to belong to everyone.
Which, as Raghu would soon point out, usually meant they belonged to no one.
Week 10: Who Actually Owns This Application?
Because before you can decide what to modernise, optimise or retire, you need to know who's responsible for making that decision.Spotify
Series: The Transformation Diaries - 52 Weeks Journey (10 articles)
- …
- 9The Transformation Diaries Week 9: You Can't Transform What You Can't See This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article