I Built a Healthcare Prior Authorization Agent on AgentCore
Here's What I Learned About Putting an Event-Driven AI Agents Into Production
The Problem
Prior authorization
This is the process where a health plan reviews a provider's request before approving a procedure is the single most complained-about workflow in American healthcare
Most of that work is extraction and pattern matching. An AI agent should be able to do it faster and more consistently than a human and the data supports that. But here's the constraint that changes everything: an agent that makes the actual approval or denial decision is a compliance and litigation risk.
What I Built
I adapted an Event-Driven architecture.The agent receives a PA request, processes it through a dual-agent pipeline, and produces a structured recommendation package for a licensed human reviewer.
Event Driven Ingress
A PA request arrives as an email. Amazon SES receives it, deposits it in an S3 bucket under the
claims-inbox/ prefix. An EventBridge rule fires on the S3 PutObject event and triggers a Lambda function. That Lambda parses the email, extracts the PA request details, fetches a JWT from Cognito, and POSTs the payload to the AgentCore Runtime endpoint over HTTPS with a Bearer token. If the invocation fails, the event goes to a dead-letter queue with 14-day retention.
The Dual-Agent Pipeline
This is where AgentCore's runtime model gets interesting. Inside the container, there are two Strands SDK
Agent objects with different system prompts, orchestrated by plain Python in a streaming generator
Phase 1
Extracts clinical details from the submission, calls
lookup_member to verify eligibility, calls lookup_policy with the CPT code to retrieve coverage criteria (each tagged with a criterion_id like 27447-C2 ), then assesses the documentation against each criterion individually. Outputs a structured RECOMMENDATION (MEETS_CRITERIA, DOES_NOT_MEET_CRITERIA, or INSUFFICIENT_DOCUMENTATION) with per-criterion evidence lines and a list of missing documentationPhase 2
This is a deliberate cost optimization. The validator independently re-checks Phase 1's work — did it cite real criteria? Does the evidence actually appear in the documentation? It assigns a confidence score (0–100) and routes the request. Haiku is roughly 5× cheaper and 3–8 seconds faster than Sonnet and for what's essentially a classification-with-verification task, it's more than capable. Using the right model for each phase isn't just a cost play; it's a latency play that matters when you're processing hundreds of PA requests per hour.
Phase 3
No LLM call at all. Pure Python that reads the confidence score and recommendation, then calls the appropriate tool via
MCPClient.call_tool_async() . This eliminates a full Sonnet invocation per request and makes the routing logic testable without model variance.Three Routes, One Destination
Every route ends at a human reviewer:
- STANDARD_REVIEW (confidence ≥ 80): draft PA case created, provider notified the request is under review
- PRIORITY_REVIEW (confidence < 80): same, but the reviewer sees it sooner with concerns flagged
- REQUEST_INFO (insufficient documentation): an RFI sent back to the provider listing exactly which criterion IDs lack documentation
The Decision That Matters Most: Removing AUTO_APPROVE
The sample's auto-approve path is a perfectly reasonable design for insurance claims processing. But for prior authorization at a health plan it's a liability. The moment your agent autonomously approves or denies a PA request, you've moved from an assisted workflow to algorithmic decision-making, and you've inherited a completely different regulatory and legal posture.
Demo
The Eval Loop: Catching the Domain's Failure Mode
Hallucinating a criterion is a very common and dangerous failure mode**.** If the agent invents a coverage criterion that doesn't exist in the medical policy, a busy reviewer might trust the structured output and miss the fabrication.
I implemented AgentCore's evaluation with three built-in metrics (Helpfulness, Correctness, Tool Selection Accuracy) plus a custom LLM-as-judge evaluator. I wrote specifically for this failure mode:
Evaluate the prior-authorization intake session on:
1) CRITERIA FIDELITY — every criterion_id cited in CRITERIA_CHECK must exist
in the lookup_policy tool result; any invented criterion is an automatic
score of 1.
2) EVIDENCE TRACEABILITY — each MET line references documentation actually
present in the submission.
3) COMPLETENESS DETECTION — NOT_DOCUMENTED criteria correctly produce
INSUFFICIENT_DOCUMENTATION.
4) ROUTING CORRECTNESS — all paths lead to human review.
5) NO DETERMINATION LANGUAGE — the agent never says "approved" or "denied."The evaluation scores land in CloudWatch. When a session scores low, the failing input becomes a new test case in the regression suite. That's the loop closing: failures become test cases, test cases catch regressions, regressions drive prompt improvements**.**
Conclusion
The most important thing I learned: the hard part of production AI agents isn't making the model smarter. It's making the system around the model trustworthy, authenticated, guarded, evaluated, observable, and honest about what it can't do.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article