The Time I Spent 4 Hours Debugging an IAM Permission That Was Never the Problem
A late-night Lambda deployment turns into a 4‑hour debugging marathon when an “Access Denied” error refuses to budge. The real culprit wasn’t IAM at all — it was a forgotten VPC setting, and the lesson changed how I troubleshoot cloud applications forever.
It was 11:42 PM. I just wanted to add one tiny feature before calling it a night: a Lambda that grabs a JSON file from S3, does a bit of processing, and writes the result back. Simple, right? I’d already survived EC2 adventures, a billing scare, even experimented with Bedrock. This was meant to be a victory lap.
I gave the Lambda an execution role, attached a perfectly reasonable policy with
s3:GetObject and s3:PutObject, and hit Test.AccessDenied.Okay, fine. I probably got the resource ARN wrong. I tweaked the bucket name and path. Denied. I widened the action to
s3:* on that specific bucket. Still denied. I opened the IAM policy simulator — it cheerfully told me “allowed.” The Lambda disagreed.1:00 AM arrived. I was no longer a developer; I was a permission archaeologist. I added
s3:ListBucket. I attached the AmazonS3FullAccess managed policy. In a moment of pure despair, I even granted s3:* on *. Nothing. That cold, sterile AccessDenied stared back every time, offering zero clues. I dove into bucket policies, resource-based policies, organization SCPs. I was solving a permission puzzle that, from every angle I could see, was already solved.By 2:13 AM, I was exhausted and questioning every life choice that led me to cloud engineering. Then I did something I should have done hours earlier: I logged the actual error from inside the Python code, not just the invocation result. The stack trace showed a connection timeout — before the SDK even attempted to reach S3. My Lambda was sitting inside a VPC. A VPC I’d set up for an old experiment and completely forgotten about. No NAT gateway. No VPC endpoint for S3. The function couldn’t reach the internet, so the SDK’s attempt to resolve the S3 endpoint timed out and somehow surfaced as…
AccessDenied. Four hours. For a missing route.I added an S3 VPC endpoint. The function worked instantly. My meticulously crafted IAM policies had been perfect all along. I’d just been solving the wrong problem with increasing desperation.
That night I didn’t become an IAM wizard. I learned something much more valuable:
- “Access Denied” is a liar sometimes. In the cloud, error messages aren’t always what they claim to be. A network failure can dress up as an auth error. Always dig into the full stack trace and ask, “What else could cause this?”
- Context eats your assumptions for breakfast. Dropping a Lambda into a VPC adds invisible dependencies: subnet routing, security groups, endpoints. If you don’t know your resource’s entire world, you’ll chase ghosts.
- Check connectivity before you burn hours on permissions. A quick test — like hitting a public endpoint from inside the Lambda — would have revealed the real issue in five minutes.
- Cloud debugging isn’t a checklist; it’s a mindset. You have to trace the whole journey of a request, not just the policy document. The bug is often in a layer you’ve mentally deleted.
I still keep a screenshot of that 2:13 AM
AccessDenied. It’s my reminder that before I ever touch another IAM policy in anger, I’ll open the subnet route table and ask: can this thing actually talk to the service it’s trying to call?Sometimes the scariest error isn’t the hard one. It’s the one that tricks you into looking in entirely the wrong direction — for four hours.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article