Agentic Model Customization: Simplifying Fine-Tuning with AI-Guided Workflows in Amazon SageMaker AI
Amazon SageMaker AI now offers agent-guided model customization that handles this orchestration through a conversational interface. Purpose-built Agent Skills guide developers through the full customization lifecycle — from technique selection to training to evaluation. Rather than making silent assumptions, the agent asks clarifying questions and validates your assumptions. Read more to learn how conversational AI agents are making model customization accessible to every builder in SageMaker AI .
How conversational AI agents are making model customization accessible to every builder in SageMaker AI
Fine-tuning a foundation model requires a series of coordinated decisions: selecting a base model, choosing a training technique, formatting datasets, configuring hyperparameters, launching training jobs, evaluating outputs, and deploying endpoints. Mistakes at any stage can produce models that underperform or behave unpredictably in production.
Amazon SageMaker AI now offers agent-guided model customization that handles this orchestration through a conversational interface. Purpose-built agent skills guide developers through the full customization lifecycle from technique selection to training and evaluation according to best practices. Rather than making silent assumptions, the agent asks clarifying questions and validates your decisions along the way. The output is an executable jupyter notebook you can inspect, modify, and run.
These model customization agent skills work with any coding assistant that supports the agent skills open standard, including Kiro IDE and Claude Code. When combined with the AWS MCP (Model Context Protocol) server, the agent can access AWS services directly — validating S3 buckets, checking permissions, and executing training jobs without leaving the IDE. In this post, we'll walk through the full workflow by fine-tuning a model to simplify clinical medication instructions into plain language that patients can understand using Kiro IDE with the agent skills.
Agent Skills
The Amazon SageMaker AI agent skills are built conforming with the Agent Skills open format bringing in model customization capabilities and expertise. The agent-guided model customization workflows are powered by nine modular skills that cover the full customization lifecycle:
| Skill Name | Phase | Description |
|---|---|---|
| Use Case Specification | Configuration | Structured discovery to define business problem, users, and success criteria |
| Planning | Discovery | Generates a dynamic, multi-step customization plan tailored to your use case |
| Fine-tuning Setup | Configuration, Training | Selects base model from SageMaker Hub and recommends technique (SFT, DPO, or RLVR) |
| Dataset Evaluation | Evaluation, Training | Validates dataset format and schema before training |
| Dataset Transformation | Data Engineering | Converts between ML data formats (OpenAI chat, SageMaker, HuggingFace, Bedrock Nova) |
| Fine-tuning | Training | Generates training notebooks for SageMaker serverless fine-tuning |
| Model Evaluation | Evaluation | Configures LLM-as-Judge evaluation with built-in and custom metrics |
| Model Deployment | Deployment | Determines deployment pathway (SageMaker endpoint or Bedrock) and generates code |
| Directory Management | Configuration | Sets up project structure and organizes artifacts |
In a Kiro workspace, the skills live under the
.kiro/ directory allowing the agent to discover and activate the skills automatically: 1
2
3
4
5
6
7
8
9
10
11
12
.kiro/
├── agents/sagemaker-agent.json # Agent configuration
└── skills/ # 9 reusable skills
├── use-case-specification/
├── planning/
├── finetuning-setup/
├── dataset-evaluation/
├── dataset-transformation/
├── finetuning/
├── model-evaluation/
├── model-deployment/
└── directory-management/ IDE integration with AWS MCP
The integration has three moving parts: Kiro IDE (or other IDE of choice like Claude Code) provides the conversational interface, the AWS MCP server connects the IDE to your AWS account, and the SageMaker AI agent skills orchestrate the workflow. When you describe your fine-tuning goal in Kiro, the agent activates the relevant skills, which call SageMaker APIs, access S3 data sources, and interact with model registries through the MCP server — all without leaving the IDE. The generated notebooks and artifacts stay in your project directory, so you can version-control them and plug them into existing ML pipelines. For more on AWS MCP, see the AWS MCP documentation .
Supported Fine-Tuning Techniques
The agent currently supports three fine-tuning techniques and recommends the right one during the planning phase based on your use case.
| Technique | Description | Best For |
|---|---|---|
| SFT (Supervised Fine-Tuning) | Trains on input/output pairs | Task-specific behavior: instruction following, format compliance, domain-adapted responses |
| DPO (Direct Preference Optimization) | Trains on preferred vs. rejected outputs | Aligning tone, style, and subjective preferences to match human judgement |
| RLVR (Reinforcement Learning with Verifiable Rewards) | Trains using code-based reward functions | Tasks where correctness can be programmatically verified |
The Use Case: Medication Instruction Simplification
Medication instructions from pharmacies and healthcare providers are often written in clinical terminology that patients struggle to understand. Complex dosing schedules, contraindication notices, and side-effect descriptions in medical jargon contribute to poor medication adherence and real patient safety risks.
We wanted to adapt Qwen3-0.6B from Amazon SageMaker JumpStart to rewrite clinical medication instructions in plain language. The fine-tuned model is intended for a patient portal where it surfaces simplified instructions alongside the original clinical text. Here's the kind of transformation we're targeting:
Clinical input: "Administer 500mg amoxicillin PO TID x 10d. Contraindicated with concurrent allopurinol administration. Monitor for hypersensitivity reactions including urticaria and angioedema."
Simplified output: "Take one 500mg amoxicillin capsule by mouth 3 times a day for 10 days. Do not take this with allopurinol. Call your doctor right away if you get a rash, hives, or swelling of your face, lips, or tongue."
How It Was Built: A Step-by-Step Walkthrough
The workflow starts with a single natural language prompt. We submitted the following to the Kiro agent:
| 🗨️ Sample User Prompt - "I want to fine-tune a model to simplify medication instructions for patients. I have a dataset called med-instruction-simplification-v1 in my S3 bucket at s3://west-2-[redacted]-test-inputs/. Can you help me set up and run the fine-tuning job?" |
|---|
The below screenshot shows the initial interaction where Kiro acknowledges the request and immediately activates the AWS MCP server to investigate the S3 bucket contents.
Rather than jumping straight to code generation, Kiro first calls the AWS MCP server to check whether the S3 bucket was accessible and inspect the dataset structure. That's the agent validating inputs before consuming any compute — catching format mismatches or permission issues before they waste a training run.
Kiro's Planning Phase: Skill Activation and Context Gathering
Upon receiving the user prompt, the agent doesn't immediately begin executing tasks. It enters a planning phase where it identifies and activates the skills necessary to complete the job. In the process, the agent generates the plan which the users review the plan can can prompt the agent to modify the plan if necessary. From the initial prompt, the agent recognizes two relevant skill domains and activated both: the planning skill for structuring the overall workflow, and the finetuning-setup skill for configuring the training job. Before generating any code, the agent asks targeted questions about dataset readiness and use case details to inform its technique recommendation.
The agentic experience embeds ML best practices into the workflow, making expert-level model customization accessible to developers without deep ML expertise. This is particularly evident in how the agent handles the transition from training to evaluation. After the fine-tuning job completes, the agent autonomously activates the model-evaluation skill without being prompted. Rather than applying generic evaluation metrics, it uses the conversation history to recommend four metrics tailored to the medication simplification use case: correctness, completeness, coherence, and instruction following.
This workflow shows how agentic model customization encodes the best practices, guiding users through steps they might not know to take and generating deployment-ready evaluation code while maintaining transparency and developer control. The Kiro IDE with Amazon SageMaker AI Agentic Model Customization agent skills creates a jupyter notebook for the medical instruction use-case in the kiro workspace project.
The generated notebooks executed in SageMaker AI Studio, results in a completed training job for Qwen3-0.6B with SFT as shown below.
This walkthrough demonstrated a customized medical scenario explore more use-cases in the SageMakerAI Model Customization blog .
Conclusion
In this post, we walked through the full model customization lifecycle using SageMaker AI agent skills in Kiro IDE. Starting from a single natural language prompt, the agent planned the workflow, configured and ran an SFT fine-tuning job on Qwen3-0.6B, and evaluated the results with metrics tailored to the medication simplification use case.
Amazon SageMaker AI serverless model customization handles the infrastructure complexity of model customization. Agent skills take that a step further by encoding ML best practices into the workflow itself. They capture the decisions an experienced practitioner would make, such as which technique to use, what to validate before training, and which metrics matter for your task, then surface them through conversation in whatever IDE you're already working in. Because these skills follow the agent skills open standard, they aren't locked to a single tool.
Get started today by visiting the awslabs GitHub repository for the SageMakerAI agent skills plugin and step-by-step guid. Furthermode, explore our curated sample datasets and pre-built evaluators to experience firsthand how SageMaker AI serverless model customization with agent skills can accelerate your path from idea to production models.
This post was co-authored with Lauren Mullennex and Bobby Lindsey.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article