AWS Builder Center

Builder Center

Confident Event-Driven Agents using RLCD

Confident Event-Driven Agents using RLCD

A year ago I glued Amazon Nova Micro to a hardcoded glucose check and called it clinical decision support, turns out it was one if-statement and a prompt duplicating the same rules. Here's what happened when I swapped it for a calibrated classifier (Jev/RLCD), and let a confidence score catch the one case I was quietly getting wrong.

Marcos Henrique Santos de Lima

It has been 1 year since I first wrote about event-driven agents , where I presented a PoC using VoltAgent to orchestrate Amazon Nova Micro over Bedrock, read lab results from S3, and decide triage urgency. It still works in production, but I'm no longer part of the team that maintains it, so I was revisiting this PoC, since a friend asked me how to improve it 😆
And yes, AI at the time was like a mirror programmed to return you "beautiful" smiles, but nowadays it seems like a bible maker with a lot of claude-isms that you get buried in when you need to rely on it as a decision maker (they changed nova to opus, so it got better but worse at the same time).

The mirror that smiles back

Mostly of the huge labs, like OpenAI and Anthropic, were using RLHF , and this diagram is a very rough, simplified version of how this works and why this is always pleasant with ya:
Since this is trained to be kind and lovable by always telling you that you are the smartest guy in the world and your system is the most scalable and most reliable by using one proxy route in your API Gateway with 400+ Lambdas inside because you want to avoid CloudFormation's 500-resource limitation  in your stack (nested stacks  be like 🤡)...
So yeah, they had the brilliant idea to use RLAIF , which is basically AI that validates  AI, and yeah, this has its own advantages  , but I always remember The Terminator  xD

ok, what is the problem with RLHF? and where RLCD helps?

We don't have one, but the problem is poorly relying on it too much for everything, and most of the agent-ready software, despite being written by another agent-ready SDLC, doesn't necessarily need it. As I talked about one year ago we need deterministic workflows, decisions based on numbers, gauges and metrics... so here RLCD  shines. Within this approach, we can measure the level of confidence we are about to make a decision based on, welcome back, classifiers xD

New architecture

Basically, instead of using Nova, I envisioned using JEV  to see if I would see some improvements, since what I need here is a classifier on roids
Same shape as a year ago, S3 still kicks off EventBridge, EventBridge still wakes up a Lambda, DynamoDB still holds patient history. What changed is what happens inside the coordinator, and that's the part worth actually showing you instead of just telling you.

The old code, receipts included

Turns out "deterministic workflows, decisions based on numbers, gauges and metrics" wasn't quite what I shipped a year ago. What actually existed was one hardcoded glucose check:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
function checkCriticalValues(labData: LabData): CriticalCheck {
const glucose = labData.lab_results?.glucose?.value || 0;
if (glucose > 300) {
return {
is_critical: true,
action: 'emergency_appointment',
specialist: 'endocrinologista',
reasoning: `Hiperglicemia crítica: ${glucose}mg/dL (>300). Risco de cetoacidose.`
};
}
if (glucose < 50) {
return {
is_critical: true,
action: 'emergency_appointment',
specialist: 'endocrinologista',
reasoning: `Hipoglicemia severa: ${glucose}mg/dL (<50). Risco de coma.`
};
}
return { is_critical: false };
}
...plus a free-text Bedrock prompt for literally everything else, with the same thresholds duplicated as prose the model had to re-derive on its own every call:
1
2
3
4
5
6
7
CRITICAL DECISION RULES:
- Glucose > 300mg/dL = URGENT + createEvent(alert)
- Glucose < 50mg/dL = URGENT + createEvent(alert)
- HbA1c > 10% = PRIORITY + createEvent(appointment)
- Creatinine > 3.0mg/dL = URGENT + createEvent(alert)
- Multiple critical values = PRIORITY
- Normal/stable values = ROUTINE or observation
Two brains, same rules, written twice, no confidence scores, and nothing stopping them from disagreeing on the same patient. That's the "poorly relying on it too much" I mentioned, but I was the one relying

The new code 

src/triage.ts now makes a single call to Jev's systemOne(). One call, four typed answers, each with a confidence score attached, which is no prose to parse, no second brain that might disagree:
1
2
3
4
5
6
7
8
9
10
11
12
13
const { answers } = await getClient().systemOne({
state, // lab_results, patient_info, exam_date, patient_history
questions: {
urgency: choice("What urgency level does this lab result require?", {
urgent: "Immediate action within 0-24h: life-threatening critical values...",
priority: "Action within 1-7 days: significant abnormalities...",
routine: "Normal follow-up within 30-90 days: normal or stable values.",
}),
specialist: choice("Which specialist should review this patient's results?", { /* ... */ }),
event_type: choice("What type of medical event should be created for this case?", { /* ... */ }),
needs_event: noul("This case requires creating a medical event rather than pure observation.", { /* ... */ }),
},
});
choice() pins the answer to a closed set instead of a free-floating string I'd have to fuzzy-match. noul() gives a calibrated 0-1 probability instead of a yes/no I'd otherwise infer from tone. That's the "classifier on roids" bit, yes... same judgment call, none of the parsing problem.
The old hardcoded check didn't disappear like a Houdini, it just got demoted to what it should've been the whole time: a deterministic backstop that only fires if the Jev call itself throws, never to second-guess a live answer.
1
2
3
4
5
6
7
8
9
10
11
12
13
const certain = { urgency: 1, specialist: 1, event_type: 1, needs_event: 1 };

if (glucose !== undefined && glucose > 300) {
return {
urgency: "urgent",
specialist: "endocrinologista",
event_type: "alert",
needs_event: true,
confidence: certain, // deterministic, so certain by construction
reasoning: `Jev indisponível; backstop determinístico ativado: hiperglicemia crítica ${glucose}mg/dL (>300).`,
};
}
// ...same shape for glucose < 50 and creatinine > 3.0
And the confidence number Jev returns no longer gets thrown away in a log line. TriageDecision carries it as a real field, and src/index.ts actually reads it:
1
2
3
4
5
6
7
8
9
// event_type is the answer that showed run-to-run instability on cases with
// several simultaneous abnormal values -- exactly when confidence dropped
// below ~0.5. Publish the safer "review" instead of acting on an unstable
// answer; the stored decision still records what Jev actually said.
const EVENT_TYPE_CONFIDENCE_THRESHOLD = 0.5;
const eventType =
decision.confidence.event_type < EVENT_TYPE_CONFIDENCE_THRESHOLD
? "review"
: decision.event_type;

Benchmarks (actually run, not vibes dude)

I didn't want to just claim the new setup was better, so I ran it: 5 fixture lab results, each pushed through decideTriage() four times against live Jev, same input every call.
FixtureAvg latencyConsistent across 4 runs?event_type confidence
Normal glucose296 msYes0.91
High (non-critical) glucose132 msYes0.24
Critical high glucose (>300 mg/dL)128 msYes1.00
Critical low glucose (<50 mg/dL)185 msYes0.99
Multiple abnormal values, no dominant one179 msUrgency yes, event_type flipped0.26
100-230 ms a call, one 564 ms cold-start blip on the first request. The clear-cut cases never wavered, same urgency, specialist, and needs_event every single time, 0.98–1.00 confidence on the two critical-glucose fixtures. That's the behavior you actually want from something deciding if a patient needs an emergency alert.
The one fixture with several abnormal values and nothing dominant is where it got interesting: event_type flip-flopped between "review" and "appointment" across identical repeated calls, and that instability landed exactly where confidence bottomed out, 0.24-0.26, against 0.91-1.00 everywhere else. My old hardcoded check had no way to say "I'm not sure," it just answered. A free-text prompt would've buried that same uncertainty in a paragraph nobody was parsing. Here it showed up as a number I could actually act on.... which is the whole EVENT_TYPE_CONFIDENCE_THRESHOLD thing above.

Jev vs. Nova Micro: price and perf

Couldn't re-run the actual old Bedrock path for a clean A/B (no real credentials in the box I did this in, and the code's gone anyway), so this is published numbers plus my own Jev measurements, flagging estimates as estimates instead of pretending it's a controlled benchmark:
  • Price: Nova Micro runs $0.04/M input tokens, $0.14/M output tokens (Artificial Analysis ). Jev runs $0.042/M input tokens and free output, since it returns a typed answer instead of generated text (official pricing ); TypeSafe's own number for a typical decision: ≈$0.0004.
  • Latency: Nova Micro's own benchmarked time-to-first-token is 0.85s (P50, 72h rolling) at ~265 tokens/sec output - and the old prompt wanted a full paragraph back, so a ~300-token answer tacks on another ~1.1s after that first token. My measured Jev latency for a complete decision: 100–230 ms.
  • Vendor claims, clearly labelled as vendor claims: TypeSafe says 70–500 ms responses, "20–200x faster and 40-400x cheaper than comparable LLMs," explicitly "pending independent benchmarks" (daily.dev roundup ). My own number lands inside their claimed range, for what that's worth coming from the vendor.
Rough bottom-up math using real character counts from this repo, old prompt's fixed instructions ≈2,300 chars (≈580 tokens), new Jev question schema ≈1,400 chars (≈350 tokens), plus ~200 tokens of lab data either way: Nova Micro ≈ $0.00007/call, Jev ≈ $0.00002/call output-free. Jev comes out ~3x cheaper on my napkin math, which is oddly below TypeSafe's own quoted $0.0004/decision average, so their benchmark decision probably carries a much bigger state than five lab-result fixtures. Not going to pretend that discrepancy isn't there.

What actually improved

  • One decision maker instead of two. No more hardcoded checks and free-text prompts quietly disagreeing on the same patient.
  • A confidence number that means something. Low confidence on event_type predicted the exact case where the answer was unstable, not a coincidence, that's what a calibrated classifier is supposed to do.
  • Speed, for what it's worth. 100–230 ms for a full decision vs. Nova Micro's ~2s to finish a comparable free-text answer, end to end.
  • An audit trail that doesn't lie. The stored record keeps what Jev actually said even when the published event gets downgraded to "review" for being low-confidence — you can see both, instead of only the sanitized one.
Still not done: 0.5 as the confidence threshold is a guess, not a derived number, and I haven't run this against anything close to the volume TypeSafe's own $0.0004 figure implies. Next post, maybe.

Sources

# agentic-ai# agent-hooks# ai-agents# serverless
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article