A confidence score needs a business rule, not just a number
A Business Analysis look at AI confidence scores: what the number measures, who owns the threshold, and how to write testable acceptance criteria for it.
In a lot of AI demos I've watched as a student, someone points at a number on screen and says "the model is 0.87 confident." Everyone nods, and nobody asks what the product is supposed to do with 0.87. From a Business Analysis point of view, that number isn't a feature yet. It only becomes one when the requirements say what happens above it, below it, and when it's missing.
Where the number actually comes from
The first question I'd put in the BRD is boring but important: what is this score measuring? In a generative AI app it could be a retrieval relevance score from a search step, a classifier probability, a rule-based check, or a model rating its own answer. Those are very different things. A high retrieval score means the documents looked related, not that the final answer is correct. A self-rated score from the same model that wrote the answer is the weakest signal of the four.
If the requirements just say "show confidence," the team will pick whichever number is easiest to get, and the business will read it as "probability the answer is right."
Thresholds are business decisions
Once we know what the score is, someone has to choose the cut-offs. That isn't a data science detail. It decides how much work goes to humans, how often users see "I'm not sure," and how many wrong answers go out with a confident tone.
Questions I'd want answered before the threshold goes in a config file:
- What happens below the threshold: refuse, answer with a warning, or route to a person?
- Is the threshold the same for every task, or stricter for actions that change data or money?
- Who owns changing it, and does a change go through the same review as a prompt or model change?
- What do we do when no score is returned at all, because a step timed out or a tool failed?
The last one gets skipped most often. "Missing" should never quietly be treated as "high."
What I'd write as acceptance criteria
Instead of "system uses confidence scores," something testable:
- Given an answer whose score is below the agreed threshold for this task type, when it's shown to the user, then the UI labels it as low confidence and offers the defined next step (source links, retry, or human review), and the event is logged with the score and threshold version.
- Given an action that changes customer data, when the score is below the stricter action threshold or missing, then the action is not executed automatically and is queued for approval.
- Given the threshold value is changed, when the change is deployed, then it's recorded with an owner, a reason, and a before/after sample of how many requests would have been routed differently.
Don't show the raw number by default
One more product question: should users see the score at all? A raw "0.87" invites people to trust it like a test result. Often a plain label such as "checked against 3 sources" or "needs review" is more honest and easier to act on. That choice belongs in the requirements too, not in whoever builds the UI last.
I'm still learning how teams on AWS wire this up in practice, so I'm curious: if you've shipped something with Bedrock or another model service, where did your confidence signal come from, and who owned the threshold?
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article