What I Learned Customizing a Model for the AWS AI League
Learn how I fine-tuned Amazon Nova Micro for the AWS AI League, improved training data, tuned hyperparameters, and reached the top 10.
I joined the AWS AI League model customization challenge to fine-tune a model, deploy it, and climb the leaderboard. I finished in the top 10.
I trained Amazon Nova Micro from Amazon SageMaker Studio with Customize with UI, deployed to Amazon Bedrock, and submitted for scoring. My first run proved the pipeline worked — and also showed that generic starter data produces a verbose, off-topic assistant. What moved me up the board was better training data, Playground testing, then careful hyperparameter changes.
This article walks through that path in the order I actually used it.
1. Which model I used
When people talk about building with AI, there is an important distinction between pre-training and fine-tuning:
- Pre-training from scratch means feeding billions or trillions of tokens of raw web text into an uninitialized neural network across massive GPU clusters for weeks or months. This is where a foundation model learns language syntax, grammar, general reasoning, and broad world knowledge. It costs millions of dollars and massive compute power.
- Fine-tuning, on the other hand, takes that pre-trained base model as a finished foundation. Because the model already knows how to read, reason, and generate language, you do not need to re-teach it the basics. Instead, you train it on a small, high-quality dataset of curated examples to specialize its behavior: teaching it your target tone, enforcing specific output formats (like strict sentence limits or valid JSON), aligning it with domain constraints, and conditioning it to admit when it does not know an answer rather than hallucinating.
By starting with an established base model and applying fine-tuning, you can achieve specialized domain performance in minutes on serverless compute with just a few hundred training rows.
The challenge model: Amazon Nova Micro
For the AWS AI League competition, the designated base model was Amazon Nova Micro.
Nova Micro is a lightweight, cost-effective text model in the Amazon Nova family. Because it is small and fast, training runs complete quickly (around 20–25 minutes), making it ideal for the rapid experiment-and-submit cycle of a hackathon or challenge.
In SageMaker Studio, getting to it took three clicks:
- Open Models in the left menu.
- Select JumpStart models.
- Search for Nova Micro and open its card.
Other supported models in SageMaker AI
While the League required Nova Micro so everyone competed on the same starting line, SageMaker AI serverless customization is not limited to one model. Depending on your AWS Region and account, the catalog includes options from multiple providers:
- Amazon: Nova Micro, Nova Lite, Nova Pro, and Nova 2.0 variants
- Meta: Llama 3.1, 3.2, and 3.3 Instruct models
- Qwen: Qwen 2.5 and Qwen 3 series
- DeepSeek AI: DeepSeek-R1 distilled variants
- OpenAI: gpt-oss models
Different models support different customization techniques and context lengths. But for this competition, every step in my run was built on Nova Micro.
2. Customize with UI: fine-tuning without writing a training script
In Amazon SageMaker Studio, after you open a model and choose Customize model, you get two paths:
| Path | How it works | Best when |
|---|---|---|
| Customize with UI | A form: pick technique, attach a dataset, set hyperparameters, click Submit. No training script. SageMaker AI provisions compute, runs the job, logs metrics (including to MLflow), and tears training resources down when it finishes. | You want to iterate on data and behavior quickly |
| Customize with code | Sample notebooks in JupyterLab. Same customization idea, but you launch and adjust the job in code. | You need automation, repeatable pipelines, or deeper recipe edits |
Both paths can customize a foundation model. They differ in how you drive the job — form versus notebooks — not in the idea of fine-tuning itself.
The AWS AI League model customization challenge uses Customize with UI. That is the path I followed: open Nova Micro, fill the form, train, deploy to Bedrock, submit to the leaderboard. I still mention Customize with code so the Studio choice is clear; the rest of this post stays on the UI path the challenge required.
3. SFT, LoRA and Sequence Length
In Amazon SageMaker Studio, the Model and technique section of the Customize with UI form lets you configure how the model will be customized.
For the AWS AI League challenge, The required settings:
- Customization technique: Supervised Fine-Tuning (SFT)
- Training type: Low-Rank Adaptation (LoRA)
- Sequence length: 8K (8,192 tokens)
These settings answer three different questions: What does the model learn from? How are its parameters updated? How much text can each training example contain?
Supervised Fine-Tuning (SFT): Learning from Examples
SFT teaches a model by showing it examples of questions and the answers it should produce.
For example, imagine I want the model to answer AWS questions clearly and directly. My training data might contain:
1
2
3
Training example
User: What is Amazon S3 used for?
Ideal assistant response: Amazon S3 is an object storage service used to store and retrieve files, backups, images, and other data.During training, the model compares its generated responses with the desired answers and adjusts its trainable parameters to better match the examples.
By repeating this process across many examples, the model can learn patterns in how to answer, including the expected level of detail, tone, and format.
The important point is that SFT teaches the model through examples of the behavior you want. If your examples contain vague questions or incorrect answers, the model may learn those undesirable patterns too.
LoRA: Updating Fewer Parameters
SFT describes what the model learns from. LoRA describes how the model is adapted during training.
A foundation model already contains a large number of learned parameters, often called weights. Updating all of them during fine-tuning can require substantial computing resources.
There are two common approaches:
- Full fine-tuning: Updates the model's original weights across the network.
- LoRA (Low-Rank Adaptation): Keeps the original weights frozen and trains a smaller set of additional parameters, called adapters.
Think of the original model as a skilled assistant who already understands language. Instead of changing everything it knows, LoRA adds a smaller set of trainable adjustments to help it perform the target task.
For the challenge requirement , I used LoRA because it provided a parameter-efficient way to adapt Nova Micro without updating all of its original weights.
Sequence Length: 8K Tokens
The third setting was sequence length, which I left at 8,192 tokens (8K) as the challenge requirement.
A token is a piece of text that the model processes. A short word might be one token, while a longer word or sentence may be split into several tokens.
Sequence length sets the maximum number of tokens available for a training example, including its input and target response, subject to the training format.
For example, consider a training example containing a long AWS question followed by a detailed answer. A larger sequence length gives the example more room to fit within the configured limit. A short question with a two-sentence answer needs much less space.
it for those examples.
How the Settings Work Together
These settings serve different purposes:
| Setting | What it controls |
|---|---|
| Nova Micro | The base model being customized |
| SFT | The learning objective: learning from example inputs and target responses |
| LoRA | The parameter-efficient method used to adapt the model |
| 8K sequence length | The maximum sequence length configured for training |
Together, they defined my training setup.
4. What Goes Into the Training Data?
The challenge gives a ready-made sample
.jsonl so you can train quickly. It is already in the Nova conversation format. I used that file for my first job.In Customize with UI, I followed these steps with the sample file:
- Under Model and technique, keep the challenge requirements — SFT, LoRA, and 8K.
- Under Dataset and output, choose Upload files and select the sample
.jsonlthey provided (or its S3 path if you use that).
- Keep the default output location and Run on SageMaker Training jobs.
- Leave hyperparameters at defaults for this first run, then Submit.
The sample JSONL is for a baseline, not for climbing the board. Next I had to update the data.
How I updated the dataset after the sample
Training examples should reflect the domain, question types, and response style the leaderboard rewards. I kept the same JSONL structure as the sample, but replaced the content with challenge-aligned examples.
Before writing new rows, I asked:
- What must the model answer? Subject areas and recurring question patterns.
- What does a good answer look like? Accurate, relevant, clear, and appropriately concise.
- What mistakes hurt the score? Off-topic detail, inconsistent formatting, unsupported claims, inventing facts when it should say it does not know.
Structure the dataset correctly
Amazon Nova customization expects a JSONL file: one complete JSON object per line (no wrapping array, no commas between lines). The sample file already follows this; my updated file had to as well.
For conversational SFT, each object follows Nova’s conversation schema. In general it contains:
| Element | Purpose |
|---|---|
| Format version | Identifies the conversation schema the workflow expects |
| System message (optional) | Standing instructions for tone or behavior |
| Messages | Conversation turns: the user’s question and the ideal assistant reply |
Conceptually, one example is:
- User: a question that looks like the challenge
- Assistant: the accurate, well-structured answer you want the model to produce
Validate the format before upload, and avoid reserved strings that the training job rejects.
How many examples?
| Size | Guidance |
|---|---|
| Under ~100 | Training can fail — treat ~100 as a practical floor |
| ~200 per task | AWS’s recommended starting point |
| A few hundred diverse rows | Often enough if coverage and quality are strong |
| Many near-duplicates | Feels large, but teaches repetition more than generalization |
Challenge detail: if evaluation sends only the question, do not rely on a system message alone. Demonstrate the behavior in the user–assistant pairs.
What makes a strong training example
I used five checks on every new row I added after the sample:
| Check | Question to ask |
|---|---|
| Clarity | Is the question understandable? Is the desired answer unambiguous? |
| Relevance | Does this example belong to the challenge task? |
| Consistency | Same standards for length, tone, and structure as the rest of the file? |
| Accuracy | Are facts and recommendations correct? |
| Completeness | Does the answer address the question without leaving out what matters? |
I aimed for about two to four sentences when the question allowed it. Longer was not automatically better.
Build diversity into the examples
I varied examples across question types (facts, comparisons, recommendations, planning, logistics, honest refusals), difficulty, user situations, and response patterns. I grew by category in small batches and fact-checked before upload.
5. Hyperparameters I Used (and What AWS Suggests)
Hyperparameters determine how the model learns from the training data. I started with the default settings to establish a baseline, then experimented with different values after improving my examples.
First Run: Default Settings
For my first run, I kept the default values to see how the model performed before making changes.
- Batch size — 64: Imagine your dataset contains 320 question-and-answer examples. With a batch size of 64, the model processes 64 examples before making a training update. It takes 5 batches to process all 320 examples in one epoch.
- Epochs — 2: An epoch is one complete pass through the training dataset. With 2 epochs, the model goes through all 320 examples twice, giving it two opportunities to learn from them.
- Learning rate — 0.00001: This controls how large the model's learning updates are. Think of it as the size of each step toward learning the desired answers. Steps that are too large can make learning unstable, while steps that are too small can make learning slower.
| Setting | Value |
|---|---|
| Batch size | 64 |
| Epochs | 2 |
| Learning rate | 0.00001 |
After Improving the Data: Adjusting the Settings
After improving my training examples, I experimented with additional settings to see whether they could improve the model's performance.
- Warmup steps — 1: At the beginning of training, the learning rate can start low and gradually increase. Warmup steps determine how many initial update steps this takes. For a small dataset, a long warmup can use up a significant part of a short training run before the learning rate reaches its intended level.
- Learning rate — 0.00005: I tested a value five times higher than my initial learning rate of 0.00001. This allows larger updates to the model's weights. The goal was to see whether the model could learn the patterns in my improved examples more effectively, although a higher value does not guarantee better results.
- Epochs — 3 (up to 5): I increased the number of passes through the dataset from 2 to 3, with up to 5 as an option to test. This gives the model more opportunities to learn from the examples. However, too many passes can cause it to memorize the training data instead of generalizing to new questions.
- Batch size — 64: I kept the batch size unchanged so I could compare results without changing how many examples the model processes per update. With 320 examples, for instance, this means 5 batches per epoch.
- LoRA alpha — 128: LoRA lets the model learn through a smaller set of trainable parameters rather than updating all its original weights. Alpha controls the scaling of those learned updates. I kept the default value while experimenting with the other settings.
- Maximum context length — 8,192 tokens: Tokens are small pieces of text that the model processes. This setting limits the amount of text an example can contain. For a question about an AWS service followed by a detailed answer, the context length determines how much of that question-and-answer text can fit into the training example.
| Setting | Value |
|---|---|
| Warmup steps | 1 |
| Learning rate | 0.00005 |
| Epochs | 3 (up to 5) |
| Batch size | 64 |
| LoRA alpha | 128 |
| Maximum context length | 8,192 tokens |
What I Changed First
I followed this order to make my experiments easier to compare:
- Improve the training data so the examples provide clearer, more consistent answers.
- Adjust warmup steps to suit the length of the training run.
- Test a higher learning rate to see how it affects learning.
- Increase epochs to give the model more opportunities to learn from the data.
I kept the batch size and LoRA alpha unchanged . The goal was to understand how each adjustment affected evaluation results rather than assume that changing more settings would automatically improve the model.
6. Deploy the Model to Amazon Bedrock
After training, you have a customized model artifact. However, the AWS AI League leaderboard does not evaluate this artifact directly. You first need to deploy the model to Amazon Bedrock and then register the deployment for evaluation.
Why Deploy with Amazon Bedrock?
| Benefit | What It Means |
|---|---|
| Serverless inference | AWS manages the underlying infrastructure, so you do not need to choose or manage GPU instances. |
| On-demand inference | You can run prompts against the deployed model when needed. Charges depend on the applicable pricing for your deployment and usage. |
| Managed security | You can use AWS access controls and supported encryption features to protect access to your model. |
| Leaderboard compatibility | The challenge requires the Bedrock Deployment ARN to register your model for evaluation. |
Important: Choose Deploy with Bedrock, not Deploy with SageMaker. The challenge registration expects a Bedrock Deployment ARN, so a SageMaker endpoint ARN is not a substitute.
Step 1: Deploy your trained model
Once the training job completes:
- Open View custom model for the completed job.
- Select Actions → Deploy with Bedrock.
- In the dialog:
- Keep On-Demand selected.
- Enter a unique deployment name.
- Keep the default execution role unless the challenge instructions say otherwise.
- Select Deploy.
- Open the Deployments tab and wait for status to change from InProgress to In Service. In my experience, this took about 10–15 minutes.
- On the deployment card, select View details.
Step 2: Test the deployed model in the Playground
Before submitting to the leaderboard, I tested responses in the Playground on the deployment details page.
- Enter a question similar to those expected in the challenge.
- Select Run.
- If needed, adjust inference settings (maximum output tokens, temperature, Top P, Top K).
- Review the response for:
- incorrect facts
- instructions the model ignored
- invented details
- answers that are too long or break the expected format
Use those observations to decide what to fix in the next dataset pass.
Playground testing is optional unless the challenge says otherwise, but it was the cheapest way for me to see what the model actually learned before the board scored it.
Improve from Playground results, then upload again
After the sample baseline, my update loop was:
- Test the deployed sample model with challenge-style questions in the Playground.
- Note recurring failures (length, facts, constraints, inventions).
- Add targeted examples that show the correct behavior.
- Validate the new
.jsonl. - In Customize with UI, select SFT / LoRA / 8K again, upload the updated dataset (new unique name, for example
dataset-02), submit a new job. - Deploy, test, and compare.
What I recommend checking every run
These are general checkpoints I now use whenever I customize a model with Customize with UI on Amazon SageMaker AI and deploy to Amazon Bedrock. They came from AWS workshop guidance, the Nova data preparation docs , and what broke or wasted time in my own runs.
Dataset checkpoints
- File is
.jsonland every line is valid JSON - No blank lines and no wrapping array
- First turn is
user, last turn isassistant - Only
userandassistantroles - No reserved strings in any text:
User:,Bot:,Assistant:,System:,<image>,<video>,[EOS] - At least 100 rows (I treat ~200+ as a better start; ~300–500 as a practical sweet spot)
- Examples match the real task the model will be asked to do
- Examples pass clarity, relevance, consistency, accuracy, and completeness
- Variety across question types and situations — not many near-duplicates
- Desired tone, length, and refusals appear in the user–assistant pairs (do not rely only on a system prompt if inference may send only the user question)
- Dataset and model names use letters, numbers, and hyphens only
- Each new run gets a unique dataset and model name
Training checkpoints
- Confirm technique and sequence length for your model and use case before you submit
- MLflow tracking server is InService before you start the job
- Improve the dataset before you chase hyperparameters
- On a small dataset, keep warmup steps short (I use 1; a long default can burn the whole run)
- Keep learning rate in a safe range (about
1e-6to1e-4; I often start near0.00005after data is solid) - Keep epochs modest (I stay at 5 or fewer)
- Change one setting at a time so you can tell what helped
- Avoid stacking too many concurrent training jobs (I treat 2 as the practical ceiling)
Deploy and test checkpoints
- Choose the deployment path your evaluation or product needs (Deploy with Bedrock when you need a Bedrock Deployment ARN)
- Wait until the deployment is In Ser
- Test representative prompts in the Bedrock Playground before you trust the run
- Avoid starting another deploy while too many models are still
Creating
Quotas, cleanup, and time
- Watch account deployment limits — in my environment it was 10 deployments per account and could not be raised
- After each round, delete the old Bedrock deployment, then delete the old custom model
- Budget about 40–50 minutes per full loop (train ~20–25 min, deploy ~10–15 min, test and review the rest)
- Prefer a few deliberate experiments over many rushed ones
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article