How One IAM Policy Broke Everything (And Why the Silence Was the Scariest Part)

I want to tell you about the quietest disaster of my career.

Not the loudest. I have had loud disasters. Loud disasters are almost enjoyable — alarms go off, everyone scrambles, there is a certain electricity to the chaos. You know something is wrong. You can fight it.

This was not that.

Here is what I did.

We had a service role that needed access to an S3 bucket. Straightforward. I had done this before. I wrote the policy, attached it to the role, deployed the change, and went to make a cup of tea.

When I came back, the deployment pipeline had gone quiet.

Not failed. Not errored. Just... quiet. Jobs that should have completed had not completed. Jobs that should have started had not started. The dashboard looked the way a dashboard looks when everything is fine, except nothing was fine.

I checked CloudWatch. No errors.
I checked the pipeline logs. No errors.
I checked the service. Running normally, apparently.

Everything looked fine. Nothing was working.

This is the specific cruelty of IAM mistakes: they do not always announce themselves.

When you break a server, the server tells you. When you misconfigure a network rule, the connection refuses. When you make a mistake in code, the code throws an exception and points you directly at the line.

When you get an IAM policy wrong, sometimes AWS just quietly denies the request and moves on. No fanfare. No alarm. The service makes a call, the call is silently rejected, and the service either retries forever or simply does nothing. Both look identical from the outside.

I was not looking for an IAM problem. I was looking for a broken service, a failed deployment, a configuration error. I spent thirty minutes in completely the wrong place.

I found it eventually — buried in a CloudTrail log that I had not thought to check, because CloudTrail was not part of my usual debugging workflow at the time.

There it was. Hundreds of Access Denied events, all from the service role I had just modified, all pointing at the same S3 bucket I had just given it access to.

I had given it access. I had checked the policy. The bucket name was correct. The ARN was correct.

The problem was a single asterisk.

I had written the resource as the bucket ARN: arn:aws:s3:::my-bucket-name

What I needed was the bucket ARN and its contents: arn:aws:s3:::my-bucket-name/*

One character. A single wildcard. The policy allowed the role to perform actions on the bucket. It did not allow the role to perform actions on anything inside the bucket. So every attempt to read or write an object was silently denied, and the service sat there making quiet, polite requests into a void that never responded.

I added the /*. I redeployed. Everything started working immediately.

Forty-five minutes of debugging. One character.

The /* issue is not a bug. It is, in fact, IAM working exactly as intended.

IAM makes a distinction between the bucket itself (the container) and the objects inside it (the contents). These are different resources with different ARNs, and they require separate permissions. Giving someone access to a bucket does not give them access to what is in it, any more than giving someone a key to a building gives them access to every filing cabinet inside.

Once I understood this, a lot of things clicked. IAM is not trying to be user-friendly. IAM is trying to be exact. Every permission is explicit. Every resource boundary is real. The principle of least privilege is not a suggestion — it is baked into the structure of how resources are identified.

This precision is also why IAM mistakes are so quiet. The system is doing exactly what you told it to do. You told it to allow access to the bucket. It allowed access to the bucket. You did not tell it to allow access to objects in the bucket. So it did not.

It was never confused. I was.

First: I added CloudTrail to my debugging checklist. Before this incident, I would check application logs, service logs, and CloudWatch before thinking about CloudTrail. Now CloudTrail is second on the list, right after application logs. If something is behaving strangely and there are no errors, the answer is almost always in CloudTrail.

Second: I started using IAM Access Analyzer. AWS provides a tool that analyses your policies and tells you what they actually grant access to — not what you intended, but what the policy says. Running your policies through Access Analyzer before deployment catches exactly this kind of mistake. It is free. I now use it every time.

Third: I learned to test IAM changes in isolation before deploying to a real workflow. The AWS Policy Simulator lets you run a policy against a specific action and resource and see whether it would be allowed or denied. Five minutes of testing would have caught my missing /* before it ever touched production.

The scariest part of this incident was not the forty-five minutes of debugging. It was the realisation that I could have deployed that policy into a critical production system, watched everything appear to run normally, and had no idea that data was silently not being written — until something downstream finally noticed it was missing.

IAM mistakes do not just cause failures. They can cause invisible failures. Systems that appear healthy but are quietly not doing the thing they are supposed to do.

This is why IAM is not a "set it and forget it" part of your AWS architecture. It is something you test, validate, and monitor — because the cost of getting it wrong is not always a red alarm. Sometimes it is just silence.

And silence, I have learned, is the most dangerous sound in a distributed system.

Have you had an IAM moment that humbled you? Tell me in the comments. I promise the /* story is not even close to the most embarrassing one I have heard.