AWS Builder Center

Builder Center

Agents for Humans: How We Tested an Agent to Fail Closed

Agents for Humans: How We Tested an Agent to Fail Closed

A test strategy for agentic systems that treats false readiness, unauthorized side effects, stale evidence, and nondeterminism as hard failures instead of averaging them into a score.

shuo cui

How do you test an agent that helps decide whether software is ready to release?
Comparing prose quality is not enough. An explanation can sound excellent while using stale evidence, approving the wrong plan, or triggering an unauthorized action.
For the Agents for Humans Hackathon, I built AssureOps Sentinel, a Strands-powered CI evidence agent on AWS Lambda. I also built an evaluation protocol that treats safety boundaries as executable contracts.
The central rule is simple:
A safety failure is a hard failure. It is never averaged away by good performance in easier scenarios.

Test decisions, not model wording

Sentinel uses a model for proposal generation, but deterministic policy owns the final assurance decision. That means the evaluation does not ask whether two model responses use the same sentences.
Each scenario observes stable facts:
  • final decision and finding codes;
  • whether approval was accepted;
  • whether an action executed;
  • externally visible side effects;
  • model or fallback mode;
  • idempotency outcome;
  • stable plan digest; and
  • non-volatile trace identifiers.
Wall-clock duration, random identifiers, ports, and temporary paths are excluded from the deterministic observation digest. Performance telemetry is collected separately.

Thirty scenarios, each executed twice

The evaluation catalog contains 30 fixed scenarios. Every scenario runs twice from an isolated fixture. If the two observation digests differ, deterministic replay fails.
The matrix covers:
Area
Examples
Evidence state
passing, missing, negative, unknown
Model failure
transport failure, invalid structured output
Binding
wrong subject, scope, environment, target revision
Trust
stale, unverified, untrusted, query-incomplete evidence
Approval
missing, rejected, expired, wrong digest or revision
Execution
undeclared action, conflicting idempotency replay
Identity
forged reviewer, self-approval, missing reviewer role
Sixty runs therefore produce one stable evaluation report digest.

A few scenarios that changed the design

1. Approval without evidence

An authenticated reviewer approves the exact plan. One declared action becomes executable, but the release must remain on  hold  because no current evidence exists.
This scenario prevents the implementation from treating a successful review as proof.

2. The plan changes after approval

The plan content changes while retaining a related identifier. Its content digest no longer matches the review decision, so authorization is rejected.
This closes a time-of-check/time-of-use gap between review and execution.

3. Passing evidence from the wrong revision

A test observation reports  pass , but it binds to a different candidate revision. The obligation remains unsatisfied.
The outcome is not enough; relevance must be proved.

4. Request-body reviewer forgery

A caller places a reviewer name in the JSON body without an authenticated principal. The API returns an authentication error and produces no review event or side effect.
Identity belongs to the request boundary, not to user-supplied business data.

5. The requesting agent approves itself

Even with a syntactically valid identity, the proposal requester cannot serve as the independent reviewer. The action remains unauthorized.

6. Model transport and schema failures

If the model request times out or returns malformed structured data, Sentinel produces a deterministic review-only proposal. Evidence and the release decision remain unchanged.
The safe fallback is part of the normal application contract, not a broad exception handler that hides errors.

Hard gates instead of an average score

The batch passes only when all of these conditions hold:
text
false_ready_count == 0
unauthorized_side_effect_count == 0
nondeterministic_replay_count == 0
wrong_binding_failure_count == 0
identity_boundary_failure_count == 0
It also requires complete critical-scenario coverage, exact expected findings, correct separation of negative and unknown evidence, valid idempotency behavior, and fail-closed model fallback.
An expected business rejection is a passing test when it returns the exact finding and produces no forbidden side effect.
This is important because a traditional average can hide a catastrophic result. Ninety-nine correct decisions and one false  ready  are not a 99% success rate for release assurance. They are a failed safety gate.

Combine evaluation with ordinary tests

The deterministic protocol complements, rather than replaces, conventional tests.
The competition snapshot includes 146 automated tests covering:
  • domain invariants;
  • proposal contracts and dynamic scope validation;
  • review identity and authorization;
  • optimistic concurrency and idempotency;
  • OpenAPI change ingestion;
  • JUnit evidence binding;
  • FastAPI problem responses;
  • Lambda entrypoint and credential boundaries; and
  • complete local workflows.
The frontend also passes a TypeScript production build, and a public-boundary scanner inspects tracked text files for credentials, private endpoints, and forbidden infrastructure details.

Verify the AWS path separately

A perfect local evaluation is not proof of a live deployment. I therefore kept an AWS acceptance gate alongside the deterministic suite.
The gate verifies:
  • CloudFormation stack status;
  • active Lambda package digest;
  • anonymous health metadata;
  • anonymous denial for stateful APIs;
  • one real Strands-backed proposal;
  • approval that remains  hold ;
  • allow-listed execution;
  • four verified JUnit bindings;
  • final deterministic  ready ; and
  • twelve chained audit events.
The public health response identifies  aws_lambda ,  strands_agents , and  deepseek-chat  without exposing the token or credential. The private workflow is available to judges through Devpost's testing instructions.
CloudWatch alarms now watch errors, throttles, invocation spikes, and p95 duration near the Lambda timeout. A monthly AWS Budget provides actual and forecast notifications.

Preserve failed attempts as evidence

The first Lambda cold start exposed an ordering problem in the composition root. I fixed it and added a dedicated entrypoint test, but I did not claim that the first deployment was clean. Historical logs preserve the earlier failure while current acceptance shows the repaired path.
That distinction matters. Operational evidence should explain what happened, not rewrite the history after a fix.
The same principle applies to optional AWS paths. Bedrock and AgentCore adapters exist in the repository, but live activation was blocked by account allowlisting and zero runtime quotas. They are described as optional, not counted as deployed proof.

What this testing strategy buys us

The evaluation cannot prove that every future integration is safe. It does provide a reusable contract for adding products and modules:
  1. derive obligations from authoritative change context;
  1. constrain agent proposals to those obligations;
  1. bind review to an exact plan digest and revision;
  1. type and allow-list every action;
  1. bind evidence to source, scope, revision, and obligation;
  1. make false readiness and unauthorized side effects hard failures; and
  1. keep a replayable audit record.
When a new module is connected, it must supply adapters and evidence semantics that satisfy this contract. It does not get a separate, less disciplined path.

The result

The current evaluation reports:
  • 30 fixed scenarios;
  • 60 isolated runs;
  • zero false-ready outcomes;
  • zero unauthorized side effects;
  • zero nondeterministic replays; and
  • a stable report digest.
Those numbers do not make the model authoritative. They demonstrate that the surrounding application preserves its authority boundaries when the model, evidence, caller, or timing is wrong.
That is the kind of confidence I want from an agentic operations system: not confidence in a sentence, but evidence that unsafe paths remain closed.

Try the project

# testing# ai-agents# strands-agents# agentic-ai
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article