AWS Builder Center

A 3B Model That Beats Frontier LLMs at Your SQL

From a production query log to a small SQL assistant that out-scores models 10 to 100 times its size on your schema.

The claim

A Llama 3.2 3B model beat Claude Haiku 4.5 and Claude Sonnet 4.6 at writing SQL for one database schema. We fine-tuned it on about 200 examples. It did not beat them at SQL in general. The frontier models are far stronger there. It beat them at the one thing that matters in production. Writing queries that return the right rows from your tables.
That result is easy to misread. So let me be clear before we get to the how. This is not "small models beat big models." It is this. A model that has seen your schema, your column names, and your join patterns beats a model that has not. Even when the second model is far more capable in the abstract. Scale does not replace domain knowledge. That is the whole story. And it is a story about process far more than code.
The code lives in the hands-on workshop this post comes from. There is a link at the bottom, and more along the way. This post is the narrative. What we did. Why each step is there. How to read the results without fooling yourself.



Why customize at all?

Every text-to-SQL demo hits the same wall. You ask a strong model a question. It writes clean SQL. The query returns the wrong answer. Maybe it guessed your revenue column is called  revenue  when it is  price * quantity . Maybe it joined on the wrong key. Maybe it invented a table that sounds right but does not exist.
There is a ladder of fixes. Climb it from the bottom. Each rung costs more than the one below it.
  1. Prompt engineering. Free, fast, and always worth trying first. But the schema has to go in the prompt every time. Nothing the model learns sticks between calls.
  1. RAG. Puts the right schema in front of the model. It fixes retrieval, not writing. The model still writes SQL the way it always did. It only has better notes. If it does not know your join conventions, handing it the table definitions will not teach them.
  1. Supervised fine-tuning (SFT). Changes what the model does by default. Format, schema, conventions. This is the big lever for text-to-SQL. Most of the gain comes from here. But SFT rewards looking like your examples, not being right.
  1. Reinforcement learning from verifiable rewards (RLVR). Trains for correctness itself. The model writes SQL. You run it. You reward it on whether the result matches. This is the only rung that trains for the thing you care about.
This post climbs the top two rungs, in order. Along the way it makes a point that surprised some of the engineers who first saw it. Almost all of the interesting work is in the data and the evaluation, not the training. You run cells. You do not write training loops. Your judgment goes elsewhere. Is the data real. Is the reward honest. Is the comparison fair.



Step 1: The data is the product

Here is the design choice that makes everything after it trustworthy. We do not use a synthetic text-to-SQL dataset. We build the training set from the query log of a real database.
The environment gives you an Aurora PostgreSQL cluster. About 250 analytical queries have already run against it. PostgreSQL's  pg_stat_statements  extension recorded every one, with real call counts and real run times. Those queries are the raw material. They ran against this schema for real. So they carry the awkward six-way joins. The queries people run forty times a day. All the mess that synthetic datasets smooth away. A model trained on tidy synthetic data is confident and wrong on the messy parts. A model trained on your real traffic is not.
There is a catch, and it teaches something.  pg_stat_statements  groups statements together, so it strips the literal values out.  WHERE country_code = 'FR'  comes back as  WHERE country_code = $1 . So the first real job of the pipeline is to make these queries whole again. And to add the other half of each training pair. The plain-English question. Nobody ever wrote that down.
The pipeline runs in seven steps. Two of them carry the weight.
  • Repair and describe (the slow step, about 10 minutes). For each query, Amazon Bedrock puts realistic values back in place of the placeholders. Then comes the part that matters. We run the query against the live database to confirm it works. If it fails, we hand the SQL and the Postgres error back to Bedrock, ask for one fix, and run it again. Only queries that survive execution get in. Last, Bedrock writes the plain-English question the query answers. That question becomes the  user_query  the model learns to translate.
  • Extract the schema. The table and column definitions get written out as text and added to every training prompt. So the model learns this schema by name instead of guessing.
Two rules fall out of this design. They are the ones worth taking to your own project.
Every training pair is checked by running it. We never train on SQL we have not seen run. An unchecked pair would teach the model to write queries that do not work.
Checking costs more than writing. The slow part of the pipeline is not the LLM calls. It is the database round-trip after each one. On a real project, plan for that.
The dataset comes out in two shapes. A simple  {prompt, completion}  form for SFT. And a richer chat form for RLVR, where the right answer is hidden from the model and shown only to the reward function. That difference in shape is the difference between the two training methods. It shows up as data layout. Which brings us to the training.



Step 2: SFT, showing the model your SQL

Supervised fine-tuning is the step most people picture first. It is continued training on labeled examples. It uses the same goal the model was pre-trained with. Predict the next token. But now on your prompt and completion pairs instead of the open internet.
Here is how it works. The model reads a prompt and predicts the reference SQL one token at a time. At each spot, the loss measures how surprised the model was by the right token. Training nudges the weights to make the reference more likely. Do that over every example for a handful of passes. The model's odds start to pile up on the kind of SQL your data holds. Bare queries. No markdown fences. Your column names. Your join patterns.
Two practical notes matter more than the math.
  • LoRA makes this cheap. Instead of updating all 3B weights, we freeze the base model and train small adapter matrices. About 1% of the parameters. The frozen base keeps everything the model already knew. The adapters hold the new skill. That is why this runs on modest hardware, and why the original model stays whole.
  • The prompt template is load-bearing. The wrapper you train with is the wrapper you must serve with. If the serving prompt drifts from the training prompt, the model is answering a question it was never asked. This is the most common reason a fine-tuned model "worked in training" and fails in production.
And here is the limit that sets up the next step. SFT rewards looking like the reference, not being right. A query that looks like the reference but returns the wrong rows still scores well. The loss only compares tokens. It never runs the query.
Even so, SFT does most of the work. In our run it moved the base model's aggregate reward from about 0.43 to about 0.58 on held-out queries. The question is whether you can close the rest of the gap by training for the thing you care about.



Step 3: RLVR, training for right, not plausible

This is the rich part. It is also why text-to-SQL fits reinforcement learning so well.
Most RL post-training needs a reward model. That is a second network trained on thousands of human preference votes. You need it because "is this response good?" has no answer in code. It is slow, costly, and noisy. SQL has no such problem. A query is either right or wrong, and you find out by running it. So the reward function is written in code. Run the model's SQL. Run the reference SQL. Compare the result sets. No human labels. No learned reward model. That is the "VR" in RLVR. Verifiable rewards.
The algorithm is GRPO (Group Relative Policy Optimization), made popular by DeepSeek. SFT says "make the reference more likely." GRPO asks a sharper question. "Of the answers the model wrote itself, which ones beat its own average, and can we make those more likely?" It works in four moves.
  1. Write a group of rollouts. For each prompt, the model writes many full SQL answers on its own. 32 in our run. Sampled with enough temperature that they differ. No reference goes in. The model is exploring.
  1. Score each one. Every rollout runs against the database. It is scored on whether it ran and on result-set F1.
  1. Find the relative advantage. Here is the GRPO trick. Instead of a separate value network, it uses the group's own mean reward as the baseline. A rollout that beats the group average gets a positive advantage. One that does worse gets a negative one. There is no critic model to train. That makes it much cheaper than PPO.
  1. Update the policy. Push the model toward the tokens of its better attempts and away from its worse ones. A KL penalty keeps it from drifting too far from the SFT checkpoint.
Why you cannot skip SFT and start here. GRPO only has a signal when the group of rollouts has some spread in reward. Some right, some wrong. Start from the raw base model and almost every rollout fails to run. Every advantage is near zero. There is nothing to push toward. SFT gives the warm start. The model already writes valid SQL often enough that the group spreads out and the comparison means something.
We did not ask you to take that on faith. The workshop runs it as an ablation. The same RLVR setup on the SFT checkpoint and on the raw base model, side by side. The SFT arm starts higher and stays higher. The starting reward is measured before any gradient step, which makes it the cleanest evidence in the study. Nothing about the training setup can explain it.



The results, and how to read them

Here is the headline comparison on the held-out validation set. Treat these as reference values, not promises. Every run differs, and I will get to why.
Model
Exec Success
Exact Match
Result F1
Aggregate Reward
Llama 3.2 3B (SFT + RLVR)
~1.00
~0.38
~0.42
~0.59
Llama 3.2 3B (SFT)
~0.99
~0.37
~0.41
~0.58
Llama 3.2 3B (RLVR only)
~0.99
~0.22
~0.25
~0.47
Llama 3.2 3B (base)
~0.96
~0.19
~0.20
~0.43
Claude Haiku 4.5
~0.98
~0.10
~0.12
~0.38
Claude Sonnet 4.6
~0.94
~0.11
~0.12
~0.37
Four things in this table are the findings. They hold up run to run.
  1. The fine-tuned 3B beats both frontier models by a wide margin on Exact Match and Result F1. Those are the metrics that measure whether the query returned the right answer. The gap is large and steady.
  1. Everyone can write valid SQL. Exec Success is high for all of them. Syntax was never the problem. The gap is all in correctness against the schema.
  1. SFT does most of the work. It took the base model from about 0.43 to about 0.58.
  1. RLVR adds a real but small gain on top. A point or two of aggregate reward here.
That last point needs honesty, not hype. On about 200 examples and 60 gradient steps, the RLVR gain is small. That is a real result, not a broken lab. RLVR pays off more as your dataset grows and your reward function gets better. At this scale you are seeing the mechanism work. You are also learning the judgment you would need on a real project. Is the extra stage paying for itself yet? Sometimes the honest answer is no, not at this data volume. Knowing how to tell is the point.
One note on the metrics, because they trip people up.  Aggregate Reward  is  0.3 × execution_success + 0.7 × result_set_f1 . So anything that even parses floors near 0.30. A base model at 0.43 is not "43% correct." And  Exact Match  compares result rows as ordered lists. A correct query with a different  ORDER BY  scores zero. Some of that gap is a measurement quirk, not a skill gap. That is why you read several metrics, not one.



"So why not use a frontier model with a better harness?"

This is the sharpest question the workshop gets. It deserves a straight answer. The easy framing, "the frontier model has never seen your schema," is not quite right. A frontier model with the right harness can see your schema. It can read it from the prompt, the way we do. Or it can make tool calls to read  information_schema  at run time. Schema access is not the difference. A better harness closes that gap.
So the case for a small, specialized model is not about capability. It is about cost at volume.
  • Cost and speed. A fine-tuned 3B to 7B model can match frontier accuracy on a narrow task. The table above shows it. When accuracy ties, the tie-breaker is cost per call, latency, and the hardware you need to serve it. All three favor the small model by a wide margin at volume.
  • The schema stops being per-call overhead. With a frontier model you pay to send the schema on every request, forever. Or you pay for tool calls that read it. Fine-tuning bakes your conventions into the weights. So the prompt gets shorter and the behavior gets steadier.
  • Control and portability. The small model is yours. No per-token API bill. No outside endpoint to depend on. It can run in your own VPC.
To keep the comparison fair, note one thing. Both the fine-tuned model and the frontier baselines get the schema the same way. Schema access never separates them. What separates them is simple. One model has your conventions in its weights. The other reasons about them fresh on every call.



What this teaches

Strip away the specifics. The lessons are about method, not text-to-SQL.
  • Ground your data in reality. Real query logs beat synthetic datasets. They carry the shape, and the mess, of what you will face.
  • If correctness can be checked by machine, make your reward function check it. RLVR works in any domain where you can verify an answer in code. Code that must pass tests. Math with a known answer. Structured extraction against a schema. It works far less well on open-ended tasks like summary quality, where you are back to needing human preferences.
  • Instrument everything. Every run in the workshop logs to MLflow on its own. Eleven experiments by the end. The final numbers tell you which model won. The curves tell you why, and whether to trust it. A model that flatlined at step 10 and one still climbing at step 60 look the same in a table and nothing alike in a reward curve.
  • Right-sizing the model is an efficiency call. A small specialized model means lower latency, lower cost per call, modest hardware, and less energy per inference. For a query you run at volume, the smallest model that clears the bar is both the efficient choice and the lower-impact one.



Try it yourself

Everything above is a hands-on workshop. The notebooks. The CloudFormation stack. The Aurora database seeded with a synthetic cosmetics-sales dataset. The evaluator that scores generated SQL by running it. You run the cells and watch each step build your own training pairs, your own model, and your own comparison table.
(The workshop runs through AWS Workshop Studio.)
It is built for data scientists, ML engineers, and solutions architects who want to see what LLM customization involves, end to end. No prior fine-tuning experience needed. Every concept, LoRA, GRPO, reward functions, ablations, is explained where it first shows up.

Go deeper



The dataset in the workshop is fully synthetic. It holds no real customer, business, or personal data. The reward values quoted here come from a single run. Your numbers will differ, and the workshop explains why in detail.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article