Delphi: A Learned Cross-Model Router on Amazon Bedrock AgentCore
Build a learned query router that picks a foundation model for each request, run it serverless on Amazon Bedrock AgentCore, and test whether routing pays off.
Introduction
The simplest design sends every request to one foundation model: pick a strong model and use it for everything. It has two costs.
First, you overpay. A cheaper model answers part of the traffic as well as the strong one. In the first of the two test runs I describe below, Nova Lite answered every fact question correctly, as Claude Sonnet 5.5 did, at $0.000023 per question against Claude's $0.001942.
Second, you lose quality. No single model is best at everything. A model with a lower average still wins on some prompts. In the same tests, Nova Lite averaged 94.10 against Claude's 95.77, yet scored 100.0 on arithmetic to Claude's 93.8.
A router addresses both costs. It sits in front of the models, reads each request, and decides which model answers it. The set of models it can choose from is the candidate pool.
Others have built routers. Open-source projects such as RouteLLM and LLMRouter offer them. Amazon Bedrock has a native router, Intelligent Prompt Routing . It routes within one model family, for example between a smaller and a larger Claude model. It does not choose between model families such as Nova and Llama.
I found no public reference architecture that combines four things. It would need a router that learns to predict quality for each request, and routing across model families and providers. It would also need serverless hosting on Amazon Bedrock AgentCore, and a test harness that reports when routing does not pay off.
The test harness is the part I care about most. It compares the router with sending everything to one model, and reports a fail when the router loses.
This post walks through Delphi, my implementation of all four. It covers the tests the router has to pass, how the router is built, and what happened when I ran it against ten real Bedrock models. The short answer: on my test questions, the router gave the same quality as Claude Sonnet 5.5 at 80% lower cost, and one cheaper model on its own did as well for less.
The Oracle of Delphi answered questions by consulting the right voice for each one. A router does the same with a pool of models. The test questions keep the theme, with prophecies, facts, lore and arithmetic drawn from Greek myth.
The Three Parts
Delphi has three parts:
1/ The offline pipeline builds the data the router learns from. It sends every question in a dataset to every candidate model, scores each answer, and records each answer's cost. It runs offline.
2/ The evaluation harness decides whether routing was worth it. It runs statistical tests against three hypotheses and reports a negative result when routing does not win.
3/ The router agent reads a prompt, predicts which candidate model will answer it best, and sends the prompt to that model. It runs on AgentCore Runtime as a serverless container. The routing decision is a small CPU calculation inside the container, so it adds no network call.
The diagram shows how the three parts fit together on AWS.
The Response Matrix
The offline pipeline produces one table, the response matrix. Each row is a question. Each column is a candidate model. Each cell holds the quality score of that model's answer to that question, and the dollar cost of the answer.
Quality is a score from 0 to 100. Answers with a known correct answer score 100 if right and 0 if wrong. Short free-text answers get partial credit for each key word they include. Open-ended answers have no single right answer, so a separate model, the judge, scores them from 0 to 10, and I multiply by 10. The judge comes from outside the candidate pool, so no model marks its own family's work.
Three reference values come from the matrix.
The oracle. For each question, take the best score in its row. This is the score you would get if you knew in advance which model was best for that question. The average of these row maximums is the oracle quality. No router can beat the oracle.
The best single model. Average each column. The column with the highest average is the best single model: the one you would deploy if you sent everything to one model. I describe how the pipeline splits the questions into training, validation and test sets in How the Router Works. A router has to match it at lower cost, or beat it.
Random routing. Pick a model at random for each question and average the scores. Random routing is the floor: the score of a router that has learned nothing.
Here is an example with two queries and two models:
| Model A | Model B |
|---|---|---|
Query 1 | 90 | 60 |
Query 2 | 50 | 80 |
Model A and Model B both average 70, so the best single model scores 70. The oracle picks A for query 1 and B for query 2, and scores (90 + 80) / 2 = 85. The oracle gap is the oracle minus the best single model: 85 − 70 = 15 points. The oracle gap is the most a router can add in quality over one model.
The Three Hypotheses
Delphi tests three claims. The pass marks of these tested claims are my own judgement calls, so set your own for your workload.
H1, cost dominance. The router matches the best single model at lower cost. To pass, it must score no more than 2 points below that model, even allowing for chance variation, and cost at least 30% less per question.
H2, quality dominance. The router gives better answers than every model in the pool. It must beat each model with statistical confidence.
H3, learnability. The router captures at least 60% of the gap between random routing and the oracle. The measure is the capture ratio:
capture = (router − random) / (oracle − random)
A capture of 0 means the router scores the same as random. A capture of 1 means it matches the oracle. H3 passes when the capture ratio on the test questions is at least 0.6.
Why the oracle gap limits H2 (quality dominance)
H2 can only pass if the models make different mistakes. I call these errors non-nested: model A fails queries that model B gets right, and B fails queries that A gets right. A router wins by sending each query to the model that gets it right.
When one model's mistakes are a subset of another's, the errors are nested. Take the extreme case, where model A is the best model on every query. The best score in each row is then A's score, so the oracle equals A's average and the oracle gap is zero. The best possible router picks A every time and ties with it.
So the oracle gap sets an upper limit on H2. If the gap is near zero, no router can pass H2 on that pool.
The oracle gate
This sets the order of work. Before training any router, Delphi builds the response matrix and computes the oracle gap. This check is the oracle gate. If the gap is 3 points or more, the gate passes and H2 is worth testing. If the gap is below 3 points, the gate fails. The project then tests H1, routing for cost, or adds models from other families to the pool.
The gate is the first of the four tests Delphi runs, ahead of H1, H2 and H3. It uses the test questions, so it needs only the response matrix and no trained router. In the run below the gate was borderline, so I ran all four tests anyway.
How the Router Works
What the router predicts
For each prompt, the router predicts a quality score for every candidate model, then picks the highest. The response matrix supports this design, because every training question has a measured score for every model. The router picks on predicted quality alone and not on price.
By comparison, RouteLLM uses a different signal: which of two models, one strong and one weak, people preferred for a prompt. That suits a choice between two models. Delphi chooses from a model pool and has a score for every question and model, so I wrote three small router families of my own. Each is similar to a router in LLMRouter.
Turning a prompt into numbers: feature hashing
All three router families work on numbers, so the router first turns the prompt into a list of 512 numbers with feature hashing. It splits the prompt into words and word pairs, and a hash function maps each one to a position in the list. Similar prompts produce similar lists, with no embedding model to call or load, which would be costly.
Train, validation, test
The pipeline splits the questions into three groups. The routers learn from the training questions. I use the validation questions to pick the best router family. I score that router on the test questions once, at the end. I keep the test questions out of the choice of family, so the final score shows how the router does on questions it has never seen, rather than one that suits those questions by chance. A check also confirms that no test question is a near copy of a training question.
The three router families
Each family takes those 512 numbers and predicts a quality score for each model.
Linear. One ridge regression per model. Each is a weighted sum of the prompt's features, with a penalty on large weights that stops it fitting noise in the training data.
Prototype. One list of numbers per model, the average of the training questions where that model scored above its own average. The router scores each model by how similar the prompt is to its prototype.
kNN (k nearest neighbours). The router finds the 15 training questions most similar to the prompt and averages each model's scores on them.
I cannot know in advance which family suits a workload, so all three compete. The family with the highest capture ratio on validation data wins.
Testing on Real Models
The tests in this section ran in the offline pipeline, calling Bedrock directly.
The questions
I wrote a generator that makes questions on a Greek myth theme, in four topics. Each topic tests a different skill, so a model that is strong in one can be weak in another.
Topic | What it tests | Example |
|---|---|---|
Prophecy | Short creative writing that follows a rule | "Xanthe, a lovesick poet, comes to the Oracle seeking a prophecy about a broken oath. Deliver a cryptic yet wise prophecy. Give it in exactly three lines." |
Facts | Multiple-choice recall | "Which Titan stole fire for humanity?" with four lettered options |
Arithmetic | Word problems of one to three steps | "Kleon buys 12 jars at 7 obols each and pays a 5-obol toll. How many obols in total? Answer with a number." |
Lore | Short free-text answers | "What did Prometheus steal from the gods?" (answer: fire) |
Prophecies have no single right answer, so the judge scores them. For the judge I used GPT-5.4, which sits outside every model family in the pool. It scores each prophecy from 0 to 10 on whether it stays on topic, sounds cryptic, gives real advice and follows its rule, such as exactly three lines or at most 25 words. The other three topics have a known answer, so the pipeline checks each model's final answer against it.
The questions are synthetic, so I know every answer, and none of them comes from customer data. They are also short and themed, so they are easier than a production workload.
The models
I ran the pipeline against ten Bedrock models from seven families, calling each through the Converse API. The pool runs from 3 and 4 billion parameter models up to Claude Sonnet 5.5, with a 180 times spread in cost per question. Cost per question is the input and output tokens Bedrock reports, multiplied by each model's list price. It excludes the judge.
I wrote down the tests and pass marks for each run before making any calls, so I could not pick a test to suit the result.
The experiment has two steps. The first trains the router and runs the planned tests. The second freezes that router and tests it on new questions.
Step 1: the training run
Every model answered 1,000 questions, which I split as described in Train, validation, test: 491 for training, 202 for validation and 307 for test. I trained all three router families on the training questions. On validation, kNN won, with a capture ratio of 0.656, ahead of prototype at 0.623 and linear at 0.577. So the router in both steps is the kNN router. It is the slowest family, because it searches the stored training questions. Even so, in this offline run 90% of its routing decisions took 6.6 ms or less, against my budget of 150 ms.
The table shows how each model and the router scored on the 307 test questions:
Model | Quality | Cost per question |
|---|---|---|
Gemma 3 4B | 85.90 | $0.000011 |
Ministral 3B | 88.86 | $0.000012 |
Nova Micro | 90.65 | $0.000013 |
Nova Lite | 94.10 | $0.000023 |
Qwen3 32B | 93.39 | $0.000033 |
Gemma 3 27B | 96.74 | $0.000039 |
Llama 3.3 70B | 91.50 | $0.000086 |
DeepSeek V3.2 | 96.78 | $0.000141 |
Nova Pro | 91.66 | $0.000300 |
Claude Sonnet 5.5 | 95.77 | $0.001942 |
Delphi router | 97.95 | $0.000349 |
Random routing (floor) | 93.03 | |
Oracle (ceiling) | 99.71 | |
The most expensive model was not the best. Gemma 3 27B matched Claude at about a fiftieth of the price.
I set four tests before the run: the gate, then H1, H2 and H3. Only H3 passed.
The gate failed by a hair. It needs the best possible router to beat the best single model by at least 3 points. On the 307 test questions the gap was 2.93. Across all 1,000 questions it was 3.09, so with this many questions the gate is a close call.
The router learned. It captured 74% of the possible gain, so H3 passed. It did not beat every model. DeepSeek V3.2 and Gemma 3 27B scored as well, so H2 failed. H1 failed because the best single model was already cheap. That was Gemma 3 27B, at $0.000039 per question, and the router could not undercut it by 30%.
The router did beat Claude Sonnet 5.5, scoring 97.95 against 95.77 at 82% lower cost. Claude is the most expensive model in the pool, the kind of model the introduction describes sending everything to. But I chose that comparison after seeing the results, and a comparison chosen after the fact can flatter the router. So step 2 tested it on new questions.
Step 2: the confirmation run
I did not retrain the kNN router from step 1. I checked its files had not changed, and wrote 520 new questions it had never seen. All ten models answered them, so I could compare the router with each one. Before the calls, I wrote down one test: the router must score no more than 2 points below Claude Sonnet 5.5, at no more than 70% of Claude's cost.
| Quality | Cost per question |
|---|---|---|
Claude Sonnet 5.5 | 95.48 | $0.001754 |
Delphi router | 96.63 | $0.000346 |
Gemma 3 27B alone | 96.90 | $0.000038 |
The test passed. The router scored 1.15 points above Claude at a fifth of the price, an 80% saving, close to step 1's 82%. Allowing for variation across the 520 questions, the router could be at worst 0.54 points below Claude, well inside the 2-point limit.
The router does not look at price. The saving comes from topics where cheap models scored best:
Topic | Router | Claude | Where the router sent the questions |
|---|---|---|---|
Prophecy | 93.1 | 89.6 | DeepSeek V3.2 54, Gemma 3 27B 47, Claude 29 |
Facts | 99.2 | 99.2 | Nova Lite 117, Nova Pro 13 |
Arithmetic | 100.0 | 95.4 | Nova Lite 130 |
Lore | 94.2 | 97.7 | Nova Lite 66, Nova Pro 26, Nova Micro 20, others 18 |
It paid for large models mainly on prophecy. It sent all arithmetic and almost all facts to Nova Lite, at about $0.00002 per question. On lore it sent most questions to small models and scored 3.5 points below Claude.
The result has three limits. The router matched Claude but was not shown to beat it: the difference could be 0.54 points worse to 2.90 points better. It did not beat every model, so H2 failed again. And Gemma 3 27B alone scored as well at a ninth of the router's cost, cutting cost against Claude by 97.8% to the router's 80%.
One caveat applies: the new questions come from the same generator, so this tests more questions of the same kind.
Running It on AgentCore
Everything so far ran offline, as Python scripts on my laptop calling Bedrock directly. This section moves the router into the cloud, onto AgentCore Runtime.
The router in the sample is a different router from the kNN router I tested above. It chooses among three models. I describe the simulated models it learns from under Deploy It Yourself, below.
The router you deploy is trained on scores from the simulated models, so its choices do not reflect real model quality. The one I deployed is a linear router. To route on your own workload, build the response matrix for your questions and retrain it.
On the request path, a client sends a prompt to AgentCore Runtime. The router agent turns the prompt into numbers, predicts each model's quality, and calls the highest-scoring model through Bedrock Converse. It then writes a decision record to CloudWatch, a log entry holding the chosen model, the predicted scores, the tokens and the cost.
Offline, the scripts build the response matrix and write the trained router to S3, where the agent loads it at cold start. I ran the scripts on my laptop. The diagram in The Three Parts shows SageMaker or EC2, which suits a larger workload.
To compare routing with a single model, Delphi deploys two arms, each a separate copy of the system. The router arm routes. The pinned baseline arm always calls one fixed model.
I have not yet compared the two arms on live traffic. The results in this post come from the offline runs.
Some key design decisions to note: The router is a library inside the agent, so a routing decision takes 0.28 ms at the median. Both arms run the same container image and code, so any measured difference between them comes from routing.
All three families use only numpy. It is quick and runs on a CPU, which is all the router needs. I left out PyTorch, because it would make the container larger and slower to start, which adds both time and cost.
Deploy It Yourself
The code is in the aws-samples repository: sample-agentcore-delphi . Start with the offline pipeline. It needs only Python and no AWS account, because it uses simulated models in place of Bedrock. Each simulated model is a few lines of code with an accuracy I set per topic. It runs the gate, the training and the tests.
When you want to see routing run on AgentCore, follow
docs/deployment.md in a dedicated AWS account. The AWS CDK app builds two arms in us-west-2:
Delphi-router runs the learned router across Nova Lite, Nova Pro and Llama 3.3 70B. Delphi-pinned-nova-lite always calls Nova Lite, as the baseline. Both run the same container image, each in its own CDK stack with a least-privilege role. Each arm can call only the models in its own pool. The router loads its trained artifact from S3 through a pointer file, router/current.json , at cold start. Every request writes a decision record to CloudWatch. A command-line tool, delphi-verify , calls a deployed arm and prints the routing decision, the chosen model and the cost.When you finish, follow the guide's teardown steps.
Conclusion
Routing fixed one of the two costs from the introduction. Against Claude Sonnet 5.5, the router gave the same quality at 80% lower cost. It did not give better answers than every model in the pool, and the harness reported that as a fail.
Three findings stand out. Compute the oracle gap before you train a router. In an earlier round on three models, the router learned the pattern and still beat no model measurably. Measure every model before you route, because a 27B open-weight model matched the most expensive one. And name the model a saving is measured against. The router cost 80% less than Claude Sonnet 5.5, and about nine times as much as Gemma 3 27B.
Three steps come next: a real workload in place of synthetic questions, a router trained on real models in the deployed arm, and cost as part of the router's training.
The most useful
test is your own questions. Build the response matrix for your pool, check the oracle gap, and see whether one model already answers well enough. If you do, I'm keen to hear what you find.Resources
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article