When My AI Agent Confidently Got It Wrong

The answer looked perfect. Clear steps, a friendly tone, even a little summary at the end. My Bedrock agent had just told a colleague exactly how to request access to a shared S3 bucket. The only problem was that the process it described did not exist. Not an old version, not a slightly wrong version. It simply made it up, and it sounded more sure of itself than I usually do.

This was a few weeks into my RAG experiments. The knowledge base had our team docs, the agent had instructions, and in testing it had answered most of my questions well. So I shared it with a couple of teammates and asked them to try it.

One of them asked about bucket access. Our docs had a page on IAM roles and a separate page on our ticketing process, but nothing that connected the two for that specific bucket. Retrieval pulled back a bit of each. The model filled the gap between them with a process that sounded completely reasonable: raise a ticket, tag the platform team, wait for approval, get a role attached. It read like something we would do. We just don't.

When code breaks, it usually breaks loudly. An error message, a failed deploy, a red line somewhere. This was the opposite. Nothing failed. The answer was polite, well formatted, and wrong, and the only reason we caught it was that my teammate knew enough to be suspicious.

That was the part that stuck with me. If someone newer had asked, they would have followed those steps, raised a ticket nobody expected, and probably felt silly for doing it wrong. The agent would not have felt anything at all.

The fix was not one big thing, it was a few small ones that I now treat as basics.

First, I added the line I wrote about last week: if the answer is not in the retrieved content, say so and do not guess. That alone would have caught this one.

Second, I asked the agent to name its source document for each step. A made-up step has no source, so it stands out immediately.

Third, I stopped testing with questions I already knew the answer to. I asked teammates to try questions from their real work, especially the awkward ones that fall between two docs. Those gaps are exactly where the agent is most likely to invent something.

And finally, I filled the gap in the docs. The agent was wrong, but it was wrong because we had never written down that process anywhere. Fixing the source fixed the answer.

I used to think the risk with AI agents was that they would not know enough. Now I think the bigger risk is that they never sound unsure. A confident wrong answer is harder to spot than a vague one, so the job is to make it easy for the agent to say "I don't know" and easy for people to check where an answer came from.

My agent still gets things wrong sometimes. The difference is that now, when it does, it usually tells me.