The Fine-Tuning Race: A Multi-Model Evaluation Platform for Right-Sizing SLMs on AWS
Fine-tuning one model tells you how that model performs, not which of a dozen candidates wins on your data or whether any beats the frontier model you already pay for. A fine-tuning race trains candidates in parallel on identical data, scores them on one shared held-out set, and ranks them against a frontier baseline on quality, cost, and latency; in our first race, the candidate that looked strongest on paper scored near zero.
Series: Prototype (3 articles)
- 1The Fine-Tuning Race: A Multi-Model Evaluation Platform for Right-Sizing SLMs on AWS This article
Introduction:
Teams running high-volume generative AI workloads keep arriving at the same question: could a fine-tuned small language model (SLM) do this job for less? For narrow, repetitive tasks such as classification, extraction, routing, and tool calling, the answer is often yes. Getting to that answer is the hard part.
Fine-tuning itself is no longer the obstacle. Serverless model customization in Amazon SageMaker AI handles training, evaluation, and deployment without you provisioning a single GPU. What’s still manual is the comparison. Fine-tuning one model tells you how that model performs. It doesn’t tell you which of a dozen candidates performs best on your data, or whether any of them beat the frontier model you’re already paying for. Answering that means running each candidate yourself, on the same data, under the same conditions, and putting the numbers side by side.
In our own delivery work, each new candidate model cost roughly two weeks of setup: a custom container, plus training, checkpoint, and merge scripts. Almost none of it carried over to the next model. So model choice tended to fall back on published benchmarks and parameter counts, which is a poor guide to how a model behaves on one specific task.
In this post, we describe a pattern we call a fine-tuning race, and the platform we built to run it on AWS. You bring a dataset, and the platform fine-tunes several candidate models in parallel on identical data, scores every one of them on the same held-out set with the same decoding settings, and ranks them against a frontier baseline on a single leaderboard covering quality, cost, and latency. We cover the architecture, the two training paths (self-hosted LLaMA-Factory on SageMaker training jobs and SageMaker AI serverless model customization) we orchestrate behind one comparison layer. The platform is available as an AWS sample at https://github.com/aws-samples/sample-slm-finetuning-and-evaluation-platform .
Why model selection is the bottleneck
The market has moved toward task-specific models. Gartner predicts that by 2027 , organizations will use small, task-specific models at least three times more than general-purpose large language models (LLMs). NVIDIA Research makes a related argument in Small Language Models are the Future of Agentic AI : agents make many narrow, repetitive calls that SLMs handle well, and a heterogeneous system that pairs a frontier model for open-ended reasoning with SLMs for the narrow steps is a sensible target architecture.
Published AWS results support the pattern. Reinforcement fine-tuning on Amazon Bedrock delivers average accuracy gains of around 66% over base models. Reinforcement Learning with Verifiable Rewards (RLVR) on SageMaker AI improved a Qwen 2.5 7B agent’s tool-calling reward by 57% on scenarios it hadn’t seen during training. A Qwen3-1.7B model fine-tuned with supervised fine-tuning (SFT) followed by Direct Preference Optimization (DPO) gained roughly 30 percentage points of accuracy and finished about 9 points ahead of Llama-3.2-3B, a model with nearly twice the parameters.
Even so, adoption lags the economics, and recent CIO surveys point to why: limited specialist talent, and unclear return on an effort of uncertain length. Both of those are costs of finding out, not costs of fine-tuning.
Producing a trustworthy answer requires three skills at once. You need fine-tuning craft to choose an objective (SFT, preference losses such as DPO, ORPO, or SimPO, KTO, or verifiable-reward reinforcement learning) and a method (LoRA and its variants, freeze tuning, or full fine-tuning), and to set learning rates that differ by roughly 30 times across those objectives. You need enough AWS fluency to handle training containers, job orchestration, instance sizing, and checkpoint and merge mechanics. And you need evaluation rigor: disjoint held-out sets, identical decoding across candidates, task-appropriate metrics, and honest cost accounting. Teams that have all three are rare. Teams that have all three available for a two-week detour are rarer.
What serverless model customization gives you today
Before describing what we built, it’s worth being precise about what SageMaker AI already does, because the surface has expanded quickly and the gap is narrower than it was a year ago.
Serverless model customization launched in December 2025 . The following table summarizes how it has grown since.
Date | Capability |
|---|---|
Dec 2025 | General availability with Amazon Nova, Llama, Qwen,
DeepSeek, and gpt-oss; SFT, DPO, and reinforcement learning |
Feb 2026 | Amazon Bedrock reinforcement fine-tuning extends to
open-weight models through OpenAI-compatible APIs |
Mar 2026 | 12 more open-weight models with RLVR and Reinforcement Learning from AI Feedback (RLAIF), down
to Qwen3 1.7B |
Apr 2026 | Qwen3.5 in 4B, 9B, and 27B sizes |
May 2026 | Agent-guided customization through agent skills in SageMaker Studio and other IDEs |
Jun 2026 | Multi-turn reinforcement learning for agentic tasks; Gemma 4 and the NVIDIA Nemotron 3 family |
Aug 2026 | Full fine-tuning on serverless for more than 25 open source models |
The built-in evaluation is capable as well. You can score a customized model with LLM-as-a-judge backed by an Amazon Bedrock model, with a custom scorer that applies your own reward function or standard metrics such as F1 and ROUGE, or against academic benchmarks including MMLU and IFEval. A Compare with base model option measures the customized model against the checkpoint it started from.
What none of this does yet is compare candidates to each other. Each evaluation job scores one model, the built-in comparison is candidate-versus-its-own-base, and a frontier model can act as a judge but never appears as a scored row. You can launch several jobs and point similar evaluator configurations at each, but assembling the result into one ranked table on one shared held-out set, with a frontier baseline and confidence intervals, is still your work.
That assembly step is what our platform automates.
Prerequisites
To follow the pattern described here, you need:
- An AWS account with permissions to create SageMaker AI training jobs, endpoints, Amazon Simple Storage Service (Amazon S3) buckets, and AWS Identity and Access Management (IAM) roles
- Access to Amazon Bedrock in your Region for the frontier baseline and optional LLM-as-a-judge scoring
- A labeled dataset for your task, typically 1,000 examples or more, or a task with a verifiable reward such as structured extraction or tool calling
- Sufficient service quota for the GPU instance types your candidate models require
Solution overview
A race is one experiment with one deliverable: a decision.
You supply a dataset, a description of the task, and an appetite for how much to spend, expressed as roughly 4, 8, or 16 candidates. A rule-based planner proposes a set of candidate models and methods, along with an estimated cost and runtime. Nothing bills until you approve the plan.
After approval, the platform starts a training job per candidate, plus the evaluation jobs that score it. Jobs run in parallel and are isolated from each other, so one model failing to converge or hitting a quota limit doesn’t take down the rest of the race. Each job runs the same three stages inside the same frozen container image: train, merge, then evaluate under vLLM. Metrics land in Amazon S3. Nothing is compared until every arm has produced metrics through the same path.

The following diagram illustrates the solution architecture.

The workflow includes the following steps:
- You sign in through Amazon Cognito and upload a dataset, which is stored in Amazon S3 along with all race state and artifacts.
- The planner proposes a race and estimates its cost from a maintained per-instance rate table, versioned with the code and checked against the AWS Price List API. You review and approve before anything runs.
- The orchestration core starts one training job per candidate, routing each to the appropriate engine.
- Each job trains, merges adapter weights where applicable, and evaluates the result under vLLM against the shared held-out set.
- A scheduled Amazon EventBridge rule invokes a reconcile loop that polls job status, advances each run’s state machine, and writes progress back to Amazon S3.
- When all arms finish, the comparison layer assembles the leaderboard, including a row for the frontier baseline invoked through Amazon Bedrock.
- You export a deploy bundle for the winning model and stand it up on a SageMaker AI endpoint in your own account.
Two decisions here are worth explaining:
The orchestration core is deterministic Python on the SageMaker Python SDK rather than an agent. Races are long-running and expensive enough that reproducibility matters more than flexibility, and when something misbehaves you want to read the code path that caused it. We reserve model calls for genuinely fuzzy judgment: LLM-as-a-judge scoring on open-ended tasks, and turning a failed job’s logs into a readable explanation. The model narrates. It never chooses configuration or authorizes spend.
The reconcile loop is the reason races are practical to operate. Because state lives in Amazon S3 and an EventBridge schedule drives progress, a race advances with no browser open and survives a restart of anything in the control plane. A multi-hour, 16-arm race that depends on someone keeping a tab open is not something you can hand to a colleague.
Orchestrating two training engines
We deliberately run two engines behind one comparison layer, because neither on its own produces the decision we’re after.
The self-hosted engine is LLaMA-Factory running in a SageMaker AI training job from an image in Amazon Elastic Container Registry (Amazon ECR). It brings breadth: LLaMA-Factory reaches more than 100 open models, of which the platform ships a curated catalog of 30 ready to race, full and freeze fine-tuning for small models, the LoRA variant family (DoRA, rsLoRA, PiSSA, and LoRA+), the preference losses (DPO, ORPO, SimPO, and KTO) against a single dataset, and efficiency options such as NEFTune, the Liger kernel, and sequence packing. If your candidate sits in the long tail, or you want to race methods against each other rather than models, this is the engine that reaches.
Serverless model customization brings managed infrastructure and verifiable-reward reinforcement learning. Under RLVR, the model generates several candidate responses per prompt, a reward function scores each one, and Group Relative Policy Optimization (GRPO) updates the policy toward the higher-scoring responses. RLAIF replaces the hand-written reward function with a judge model, which is what you want when correctness is difficult to express in code, such as tone or instruction adherence. SageMaker AI absorbs the parts of self-managed reinforcement learning that consume weeks of engineering time: GPU capacity, memory orchestration between rollout and training phases, reward infrastructure, and checkpointing.
Because both engines emit metrics in the same shape, the same model can appear as two comparable rows, one per engine. That matters more than it sounds. “Which engine should we use?” is a worse question to answer first than “which model wins?”, and this way nobody has to answer it up front.
Designing a fair comparison
Training several models is straightforward. Making their scores mean the same thing is where the work actually goes. Six things corrupt a comparison quietly, without raising an error, and each one produces a plausible-looking number at the end.
Assert that the held-out set is disjoint. Not “we split it 80/20 and trust the shuffle.” Assert it in code, before any billable job starts. Leakage between training and evaluation data is the most common way a race returns a confident wrong answer.
Fix decoding across candidates. Same temperature, top-p, maximum tokens, stop conditions, and prompt scaffold for every fine-tuned arm. The frontier baseline is called through a different API and cannot share every setting, so its row is labeled rather than presented as decoding-identical. A candidate evaluated under greedy decoding while another samples isn’t being compared to it.
Validate chat templates before spending anything. Every instruction-tuned model was trained against a specific template. Apply a different one at fine-tuning or evaluation time and the model misreads roles and structure, and quality drops with no warning. Hugging Face’s Matthew Carrigan called this the silent performance killer, and the ChatBug paper later showed template mismatch is severe enough to work as an attack. We derive the template per model and check it against the engine’s own registry before the job starts.
Strip reasoning tokens before scoring. Reasoning-distilled models emit chain-of-thought inside <think> blocks. If your scorer sees that text, exact match and F1 collapse for reasons that have nothing to do with model quality.
Check engine configuration keys against engine source. Fine-tuning frameworks move fast and configuration keys get renamed. A silently ignored key produces a job that succeeds and trains something other than what you asked for, which is worse than a job that fails.
Keep a verification store. Every combination of model, engine, method, and variant gets smoke-tested by a real run, and the outcome badges that combination in the model picker. This is the part that compounds across projects. After a dozen races you aren’t guessing whether DoRA works on a given model in a given engine. You know, because it ran.
Accounting for cost and latency
Quality is the axis everyone measures. Cost and latency are where the decision is usually made, and where comparisons most often mislead.
Training cost is estimated before the run against the AWS Price List API and gated behind approval. Our runtime heuristic is reliable for models in the 7–8B range and drifts to roughly ±40% on very small ones, where fixed overheads dominate and a token-count estimate models them badly. We report a range rather than implying precision we don’t have.
Latency is reported as a per-request mean measured during the evaluation run. Tail latency is what decides whether a model is viable in a synchronous path, so measure p90 and p99 against your own serving stack before committing.
The cost comparison between an SLM and the frontier baseline is not like-for-like, and the leaderboard says so. The baseline’s figure is real API spend for the evaluation run. The SLM’s figure is a projected self-hosted cost at an assumed utilization. Collapsing those into one “cost” column is the single most common way SLM comparisons overstate the case, so we show both, labeled.
Serving economics have also shifted. AWS and the vLLM community optimized multi-LoRA serving for Mixture of Experts models , reporting a 454% improvement in output tokens per second and 87% lower time to first token for gpt-oss-20b between vLLM 0.11.1rc3 and 0.15.0, with further Amazon-specific gains on SageMaker AI and Bedrock. Dozens of fine-tuned variants can share one GPU with adapters swapped per request, which weakens the objection that adopting SLMs means running an endpoint per model.
Reporting quality with confidence intervals
A leaderboard is a ranking, and rankings invite over-reading. Two guardrails keep that in check.
Report a confidence interval on every quality score. Ours is a binomial (Wilson) interval, which fits accuracy and exact match cleanly; for continuous metrics such as F1 or a judge score, read it as indicative rather than exact. Evan Miller’s Adding Error Bars to Evals sets out the approach: treat evaluation questions as a sample from a larger population, report the standard error, and use paired analysis between candidates to narrow it at no extra cost.
Then know when the interval itself is untrustworthy. Below a few hundred data points, the Central Limit Theorem approximation degrades and normal-approximation intervals mislead, as Bowyer et al. showed at ICML 2025 . The leaderboard raises a small-test-set warning rather than letting a 199-row evaluation imply three significant figures.
Results
Here is the race that convinced us the harness earns its cost.
The task was topic classification on AG News, with a 199-row held-out set. Among several candidates, two are worth putting side by side.
Candidate | Accuracy | Output tokens per
answer | Relative latency |
|---|---|---|---|
Fine-tuned Qwen3-0.6B | 80.4% | ~7 | 1x |
Reasoning-distilled 1.5B model, 3.5x the raw
GPU throughput | ~0% | ~247 | ~11x |
On specifications alone, the 1.5B model is the obvious pick. It’s larger, newer, reasoning-distilled, and moves 3.5 times the tokens per second on the same hardware. Under a real harness on a real task, it lost badly, and not because it’s a poor model. It overthinks a short-label task. Asked to emit one of four topic labels, it produced around 247 chain-of-thought tokens per answer, blew the latency budget by an order of magnitude, and frequently never emitted a parseable label at all.
A team choosing on specifications, public benchmarks, or intuition takes the 1.5B model. A team running a fair race takes a model 2.5 times smaller that is both more accurate and eleven times faster.
An earlier engagement produced the finding that motivated the platform in the first place: a fine-tuned Qwen3-4B met production requirements at a fraction of the cost of the foundation model it replaced. Reaching that conclusion took about two weeks of bespoke work for that one model. The same conclusion is now a configuration change.
It’s worth saying plainly that a race ending in “keep the frontier model” is also a good outcome. The deliverable is a decision backed by measurements. Sometimes the measurements say don’t migrate, and that saves more than a migration would have.
Deploy the winning model
The platform exports a deploy bundle that is version-matched to the exact training image that produced the weights, and a single deploy.sh stands the model up on a SageMaker AI endpoint in the same account. We tested this for both ungated and license-gated models. Gated families such as Llama require the account owner to accept the model license before weights are pulled, which the bundle handles at the start rather than failing partway through.
If you’d rather have serverless inference than a managed endpoint, Amazon Bedrock Custom Model Import is a second path for supported architectures including Llama, Mistral, Mixtral, Qwen, and gpt-oss, with scale-to-zero and access to Amazon Bedrock Guardrails, Knowledge Bases, and Agents.
Datasets, trained weights, and evaluation artifacts stay in the account that created them.
Clean up
Training jobs stop billing when they finish, but several resources in this pattern do not. To avoid ongoing charges:
- Delete any SageMaker AI endpoints created from deploy bundles, along with their endpoint configurations and models.
- Delete or empty the Amazon S3 bucket holding datasets, checkpoints, merged weights, and metrics. Merged weights for a 16-arm race are the largest item here by a wide margin.
- Delete unused images from the Amazon ECR repository.
- If you deployed the platform with the AWS CDK and no longer need it, run cdk destroy to remove the application stack.
Limitations and considerations
A few constraints are worth knowing before you adopt this pattern:
- Cost estimation is weakest where races are cheapest. The ±40% band on sub-1B models is a real limitation, not a rounding note.
- Single-instance training sets a ceiling. Practically, that means models up to around 20B parameters, and full fine-tunes up to around 2–3B on a single 48 GB GPU. Going beyond that requires multi-instance FSDP or ZeRO-3 on EFA-capable instances.
- Upstream drift is an ongoing maintenance cost. Pinning a frozen engine image buys reproducibility and defers the cost. It doesn’t remove it. Engines change, model catalogs change, chat templates change, and someone has to own that upkeep or the verification store slowly turns into a liability.
- The guided flow is untested at scale with operators new to fine-tuning. A rule-based planner proposes the race and a human approves it. Whether that’s enough scaffolding for someone who has never tuned a model is still an open question for us.
Conclusion
In this post, we described a pattern for choosing among fine-tuned small language models by racing them: training several candidates in parallel on identical data, scoring them on one shared held-out set under identical decoding, and ranking them against a frontier baseline on quality, cost, and latency together. We covered the architecture on SageMaker AI, why we orchestrate a self-hosted engine alongside serverless model customization, the six details that corrupt this kind of comparison without raising an error, and a result where the candidate that looked strongest on paper scored near zero.
If you’re running a narrow, high-volume generative AI task on a frontier model and haven’t measured what a fine-tuned SLM would do on it, you’re making a pricing decision without data. The measurement is now a week of work rather than a quarter of it.
To deploy the platform described here, see the sample repository. To get started with the managed pieces of this pattern, see the serverless model customization documentation and the hands-on workshop in the aws-samples repository . For reward function design, the RLVR tool-calling walkthrough and reinforcement fine-tuning best practices are the most useful starting points.
Authors:
Series: Prototype (3 articles)
- 1The Fine-Tuning Race: A Multi-Model Evaluation Platform for Right-Sizing SLMs on AWS This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article