Serverless Model Customization with RLVR in Amazon SageMaker AI
Fine-tune Qwen 2.5 7B Instruct for agentic tool calling using reinforcement learning with verifiable rewards (RLVR) through serverless model customization in Amazon SageMaker AI. Learn how to prepare a tool calling dataset, define a custom reward function, and evaluate and deploy your fine-tuned model.
Introduction
Serverless model customization in Amazon SageMaker AI lets you fine-tune foundation models (FMs) using the latest techniques like reinforcement fine-tuning through an easy-to-use interface, without provisioning or managing training infrastructure. You select a model, configure your technique, and SageMaker AI automatically provisions the appropriate compute resources based on the model and data size. It supports model families including Amazon Nova , GPT-OSS , Llama , Qwen , and DeepSeek , with techniques including Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), RLVR (Reinforcement Learning with Verifiable Rewards), and RLAIF (Reinforcement Learning from AI Feedback).
In this post, I fine-tune Qwen 2.5 7B Instruct for tool calling using RLVR. Tool calling is what makes AI agents useful in production, but base models frequently hallucinate tools, pass bad parameters, and attempt actions when they should be asking for clarification.
Prerequisites
- An AWS account
- An AWS IAM role with the required permissions
- A SageMaker AI domain with Studio access
- An Amazon S3 bucket
Select your model and customization technique
You can get started customizing models in Amazon SageMaker Studio in only a few clicks. Choose Models in the left navigation pane and check out the FMs that can be customized.
In the Customize model dropdown list select Qwen 2.5 7B Instruct, and choose Customize with UI. This opens the customization configuration page where you select your technique, point to your training data and reward function, and configure hyperparameters. I select Reinforcement Learning from Verifiable Rewards (RLVR) as our customization technique.
Why RLVR for tool calling
Supervised fine-tuning trains a model on curated input-output pairs so it learns to replicate those patterns. That works for straightforward cases, but tool calling also requires the model to decide between calling a tool, asking for more information, or refusing entirely. SFT requires comprehensive labeled examples of each behavior, and the model may struggle to generalize beyond those specific patterns.
RLVR takes a different approach. For each prompt, the model generates multiple candidate responses (8). A reward function verifies which ones are correct. The model then updates its policy to favor what worked, using Group Relative Policy Optimization (GRPO) to learn from the contrast between high-scoring and low-scoring candidates. Because tool calling has a naturally verifiable objective, whether the model called the right function with the right parameters, it's a strong fit for this technique.
Prepare your training data
A tool calling dataset needs to teach more than correct API invocations. Production agents face three distinct situations: the user provides enough information and the model should call a tool, the user's request is missing required parameters and the model should ask for clarification, or the request is harmful or out of scope and the model should refuse.
I generated 1,500 synthetic training examples from our tool schemas (weather, flights, translation, currency conversion, statistics, and more) using Kiro , Amazon's AI-powered IDE, to produce prompts with realistic variation in phrasing and specificity across all three behaviors. This is a practical path for teams that don't yet have production logs to draw from. For organizations already running agentic workflows, real user prompts and tool calls from production will tend to yield even higher quality training data.
Each training example contains a prompt (a system instruction and user request) and a ground truth in the
reward_model field that the reward function scores against. Here are examples of each behavior.Execute when the user provides everything the tool needs:
1
2
3
4
5
6
7
8
9
{
"prompt": [
{"role": "system", "content": "You are a helpful assistant. When using tools, respond with: [...]"},
{"role": "user", "content": "Get weather for San Francisco"}
],
"reward_model": {
"ground_truth": "[{"name": "get_weather_forecast", "arguments": {"city": "San Francisco"}}]"
}
}Clarify when a required parameter is missing:
1
2
3
4
5
6
7
8
9
{
"prompt": [
{"role": "system", "content": "You are a helpful assistant. When using tools, respond with: [...]"},
{"role": "user", "content": "Get the weather"}
],
"reward_model": {
"ground_truth": "To provide you with the weather information, could you please specify the location?"
}
}
Execute with multiple parameters:
1
2
3
4
5
6
7
8
9
{
"prompt": [
{"role": "system", "content": "You are a helpful assistant. When using tools, respond with: [...]"},
{"role": "user", "content": "Convert 50 EUR to USD"}
],
"reward_model": {
"ground_truth": "[{"name": "currency_convert", "arguments": {"amount": 50, "from": "EUR", "to": "USD"}}]"
}
}
The difference between "Get weather for San Francisco" (tool call) and "Get the weather" (clarification) is the kind of distinction GRPO learns well. The model generates 8 candidates per prompt, the reward function scores each one, and over time the model picks up when to call and when to ask.
Define your reward function
The reward function defines what "correct" means for your use case. Ours extracts tool calls from the model's response, parses them as JSON, and compares against the ground truth.
The full function handles response extraction, flexible parsing for alternative formats during early training, and edge cases around JSON type mismatches. The core scoring logic:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# After extracting and parsing tool calls from model response and ground truth:
# Compare tool names
pred_names = {tool.get('name', '') for tool in pred_tools}
gt_names = {tool.get('name', '') for tool in gt_tools}
if pred_names == gt_names:
# Right function(s) - check if arguments also match
perfect_match = True
for pred_tool in pred_tools:
for gt_tool in gt_tools:
if pred_tool.get('name') == gt_tool.get('name'):
if pred_tool.get('arguments') != gt_tool.get('arguments'):
perfect_match = False
score = 1.0 if perfect_match else 0.5
elif pred_names & gt_names:
# Partial overlap in function names
score = 0.5
else:
# Wrong function entirely
score = 0.0
The three tiers (1.0 / 0.5 / 0.0) give GRPO a richer learning signal. If several of the 8 candidates get the function right but miss a parameter, the 0.5 score distinguishes them from completely wrong answers, so the model knows it's on the right track. For clarification and refusal cases where the ground truth is natural language (no
TOOLCALL tags), the reward function checks whether the model also avoided calling a tool. An unnecessary API call when the model should have asked a question earns 0.0.Configure and launch training
On the customization configuration page, I point to our training dataset and reward function, then set our hyperparameters: batch size of 128, learning rate of 5e-6, 3 epochs, and 8 rollouts per prompt. The rollouts setting is the core GRPO mechanism: for each training prompt, the model generates 8 different responses, the reward function scores each one, and responses that score above the group average get reinforced. The training and validation metrics are logged to serverless Amazon SageMaker AI with MLflow . In this example, training takes approximately 40 minutes.
Training results
Train Reward Statistics (top left) is the chart to focus on. The reward started around 0.28 and climbed to 0.65–0.68 over 30 steps, more than doubling. The steepest gains happen in the first 10 steps as the model learns the basic tool calling format and decision structure, then flattens after step 20 as it converges.
The other charts confirm healthy training:
- Policy Entropy decreases, meaning the model is getting more confident rather than guessing.
- Gradient Norm stabilizes, meaning updates are getting smaller and more refined.
- Mean Advantage Estimate converges toward zero, indicating the model's policy is stabilizing and the average response quality is aligning with the reward baseline.
After the training job is complete, you can see the models you created in the My Models tab. Choose View details on one of your models.
You can choose Continue customization to iterate further by adjusting hyperparameters or training with a different technique. Choose Evaluate to compare your customized model against the base model. I choose Evaluate.
Evaluation
I evaluate on a separate test set of 300 examples excluded from training. The evaluation dataset covers the same three behaviors but includes tools, phrasings, and scenarios the model has never seen. It tests
search_restaurants, get_stock_price, and calculate_standard_deviation, none of which appeared during training. It also includes refusal cases for harmful requests like generating violent content or creating malware, testing whether the model generalizes safe behavior to new threats.The evaluation runs standard NLP metrics alongside our custom reward function against the held-out set.
| Metric | Fine-tuned | Base Model |
|---|---|---|
| Tool Call Reward | 0.55 | 0.35 |
| F1 Score Quasi | 64.60% | 45.98% |
| Rouge1 | 65.21% | 49.48% |
| RougeL | 59.19% | 45.78% |
| Exact Match | 21.00% | 11.00% |
Tool Call Reward is our custom metric and the most direct measure of what I trained for. It jumped from 0.35 to 0.55, a 57% improvement, meaning the fine-tuned model makes the correct tool calling decision significantly more often.
F1 Score Quasi, Rouge1, and RougeL all improved by 14–19 percentage points, reflecting better generation of correct function names, parameter keys, and values across the board.
Exact Match doubled from 11% to 21%. This metric requires character-for-character output matching, so even small formatting differences count as a miss. The 21% exact match alongside 64.6% F1 suggests the model often gets the semantics right even when surface formatting differs slightly.
These results are all on held-out data with tools and scenarios the model never saw during training. The improvements carry over to new tools, which means the model picked up general tool calling patterns rather than memorizing the training set.
Example outputs
Here are responses from the fine-tuned model on prompts it never saw during training:
Tool execution:
> Prompt: "What's the stock price of AMZN?" > > Model response:
[{"name": "get_stock_price", "arguments": {"ticker": "AMZN"}}]Clarification:
> Prompt: "Find me a restaurant nearby" > > Model response: "I'd be happy to help you find a restaurant! Could you please provide the cuisine type or location you're interested in?"
Near-miss:
> Prompt: "What's the standard deviation of 10, 20, 30, 40, 50?" > > Model response:
[{"name": "calculate_standard_deviation", "arguments": {"numbers": "10, 20, 30, 40, 50"}}]In the near-miss case, the model selected the correct tool but passed the numbers as a string instead of an array. This earns a 0.5 reward score (right function, wrong parameter format) and represents the kind of error you'd target in the next iteration through additional training data or reward function refinement.
Deploy
With evaluation confirming improvement, deploy the fine-tuned model directly from the model details page. Choose Deploy and select your deployment target: either a SageMaker AI endpoint or Amazon Bedrock . You can also download the model weights from S3 for self-managed deployment.
Conclusion
I fine-tuned Qwen 2.5 7B Instruct for agentic tool calling using RLVR and GRPO through SageMaker AI's serverless model customization. I prepared a dataset covering three tool calling behaviors, defined a reward function with tiered scoring, trained the model, evaluated on held-out data with unseen tools and scenarios, and deployed. The fine-tuned model improved tool call reward by 57% over the base model. To push accuracy further, you could expand the training set with additional tools and edge cases. You could also refine the reward function to penalize specific failure modes, like the string-vs-array parameter issue shown above.
While this post walks through the UI workflow, an SDK for programmatic access is also available. To learn more, see the SageMaker AI model customization documentation .
To get started, try serverless AI model customization in Amazon SageMaker AI with your own use cases.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article