AWS Builder Center
Fine-tuning 16 Hugging Face LeRobot policies on Amazon SageMaker: what it actually takes

Fine-tuning 16 Hugging Face LeRobot policies on Amazon SageMaker: what it actually takes

One command fine-tunes any of 16 Hugging Face LeRobot policy types on your own robot recordings. Before a GPU is billed for hours, a license check and a 200-step smoke test on the same instance type must both pass, and the smoke test includes reloading the checkpoint it saved. Here is what we measured and what broke. One bug left the part of a model that outputs actions untrained for nearly five hours while the loss curve looked fine.

Series: Prototype (3 articles)

  1. 3
    Fine-tuning 16 Hugging Face LeRobot policies on Amazon SageMaker: what it actually takes This article
Robot learning has a long tail of policies. Hugging Face's  LeRobot  0.6.1 ships 19 of them. At one end is ACT, the Action Chunking Transformer, a 52-million-parameter model you train from scratch in under two hours. At the other end are vision-language-action models that start from a 3-billion-parameter base; some of them need an 80 GB GPU. Picking one for your dataset is an experiment, and not one you can run on a laptop GPU.

So we built a sample that turns "which policy should I try on my data" into a question SageMaker answers for a few dollars per smoke test and tens of dollars per full fine-tune. Then we ran all 16 policy types the sample offers on public datasets, so you do not have to guess. The other three of LeRobot's 19 are left out on purpose. One is a reinforcement-learning actor that  lerobot-train  cannot train. One was pretrained on data whose license limits any result to research use. One fine-tunes weights that declare no license at all. This article is about what that took. The numbers are in the middle, the repository is at the end.

What we built


One run of ./finetune.sh follows the circled numbers 1 to 5 ; set-up happens once per Region. Orange

One run of  ./finetune.sh  follows the circled numbers ① to ⑤; set-up happens once per Region. Orange arrows reach outside your AWS account; dashed boxes and arrows are optional. The full-size diagram is  docs/architecture.png  in the repository.

Set-up, once per Region.  cdk deploy  (the AWS Cloud Development Kit) creates an Amazon ECR repository, an Amazon S3 bucket and an AWS CodeBuild project. It also creates a scoped SageMaker execution role that every job runs under, and the AWS Systems Manager (SSM) parameters that record those stack facts. Then it starts the image build. CodeBuild builds the training image in your account, in about 15 minutes and with no Docker on your laptop. It pushes the image to ECR and records the digest-pinned image address (its content hash, so a re-tagged image cannot swap in) in SSM as well. Everything after this reads those parameters. The SageMaker jobs and the token secret live in your account too, but the stack does not create them.  ./finetune.sh  submits the jobs at run time, and the secret is created in AWS Secrets Manager the first time you store a token.  cdk destroy  removes the secret again when it removes the stack, unless you deployed with  -c retainData=true  or named the secret yourself.

Data flow of one run:
  • ① Laptop to S3.  ./finetune.sh  checks the dataset against the policies you enabled, reads the stack facts, and waits for a running image build if there is one. It prints one plan with the instance type, step budget and estimated cost, and asks once. Then it uploads the dataset to the bucket with  aws s3 sync .
  • ② Submit the jobs. For each policy: a CPU staging job first, if the base weights live on the Hugging Face Hub. The staging job is the only job with network access and the only one that can read the token secret; it writes the weights into the bucket. Then a 200-step smoke job, then the training job. The smoke job and the training job pull the image from ECR by digest, read the dataset and any staged weights from the bucket, and run with network isolation on (no network access at all). For gated weights (weights the Hub releases only after you accept the publisher's terms),  ./finetune.sh  checks your stored Hugging Face token against the Hub before the staging job. It asks for a token only when none is stored or the Hub rejects it, and keeps it in Secrets Manager. You can also store it at deploy time with  cdk deploy -c hfTokenPrompt=true .
  • ③ The gate. The smoke job writes its checkpoints and a  SUMMARY.json  back to the bucket. The laptop reads that summary and submits the training job only when the gate passes. The smoke job's step rate also sets the training job's time limit and the hours estimate printed before it is submitted.
  • ④ Inside the training job. The container downscales the videos if needed, checks the dataset again, renders the  lerobot-train  flags from the policy's recipe, and trains. Then it evaluates on held-out episodes (recordings the model never trained on) against the hold-still baseline, a policy that predicts the arm stays where it is. Checkpoints are mirrored to the bucket as they are written. At the end, the newest checkpoint, the evaluation, a license notice and  SUMMARY.json  are published as  model.tar.gz .
  • ⑤ Back on the laptop. The report stage downloads  model.tar.gz  into  runs/<run>-<policy>/model/ , compares  SUMMARY.json  with the reference numbers, and writes a manifest. A small inference script shows how to load the checkpoint offline and predict one action chunk (a short sequence of future actions).
The repository has three parts. The first is the stack above, deployed once per Region. The second is  ./finetune.sh , which runs steps ① to ⑤ for one policy, or several in parallel, with one question in between. The third is a trainer container with a registry of recipes, one YAML file per policy. A recipe holds the training flags, the base checkpoint, the GPU memory the policy needs, the instance types it accepts, the license class and the evaluation settings. Adding a policy means adding a file. Every Python package in the image is pinned by a lock file.

Three decisions did most of the work.
  1. Check the data before you pay for a GPU. The laptop compares the dataset with the policy's needs. How many cameras, and what shape. Does the policy need a language instruction? How wide are the actions? Does it need reward labels? A policy that pads actions to 32 dimensions refuses a 48-dimension dataset at plan time, not inside a paid job.
  1. Put a smoke test in front of every training job. The smoke test is that 200-step run: same instance type, same image, same flags as the real job. It costs under a dollar for ACT and a few dollars for policies that load a large base checkpoint. It checks five things. The job exited cleanly. The weights actually changed. No parameter is accidentally trained in bfloat16, where small updates round away to nothing. The evaluation produced a finite number. And a fresh process, running offline and with nothing cached beyond the staged base weights, can reload the published checkpoint and predict an action of the right shape. That proves the checkpoint needs nothing from the network. For policies that produce one deterministic answer per frame, the reload must also reproduce the evaluation's predictions. Policies that sample their output only have to produce an action of the right shape; the four vision-language-action models in the table below are in this group. For them the difference from the evaluation is recorded but not judged.
  1. Download base weights once, then train offline. Policies that start from a published checkpoint get a small CPU job first. It downloads the weights from the Hugging Face Hub into your bucket, pinned to a commit. Training jobs then run with network isolation on and read the staged copy. If a gated model needs a Hugging Face token, only that CPU job reads it from Secrets Manager. No training job ever sees it.
Each of those gates exists because we broke something first.

Five ways a training run goes wrong before you notice

Training logs are optimistic. A loss curve goes down whether or not the model learned anything useful. On robot data, several things go wrong with no error message at all, or with one that arrives only after the instance has started billing. We hit each of these ourselves:
  1. Cameras that are not 16:9 get squashed. A converter squashed every camera to 640 pixels wide by 360 high, whatever its real shape, and the policy trained happily on distorted frames. Now the dataset check reports each camera's aspect ratio and warns when it is not 16:9. The downscale inside the job keeps the shape: a 4:3 camera recorded at 640 by 480 becomes 480 by 360, not 640 by 360. Policies that need a square camera, such as TD-MPC, are refused at plan time.
  1. A dataset stored as PNG frames used to pass every local check. It crashed only when the paid job started loading data, because the loader expects video. The local check now reads each camera's storage type and refuses PNG datasets before anything is submitted.
  1. A finished run that loses to hold-still can hide behind a falling loss curve. Our baseline is a "hold-still" policy: it predicts that the arm stays where it is. After 200 steps most policies are still worse than it, so the smoke gate does not ask them to beat it. What the gate checks is the five things listed above, not the score. What matters for the baseline is that every evaluation prints the error next to it, so a finished run that still loses cannot hide. That habit is what caught the bug in the next section.
  1. Extending a finished run continues at the learning-rate floor. By the end of a run the scheduler has decayed the rate to its floor, the lowest rate the schedule reaches. If you extend a finished run (resume it with a larger step budget,  --extend-steps ), training stays at that floor. For recipes that declare a decaying scheduler, extending now stretches the decay to the new total, so the rate ramps down again. The other recipes only get the new step count: ACT trains at a constant rate anyway, and a few policies use LeRobot schedules that cannot be stretched.
  1. A LoRA recipe silently skipped part of what it was told to train. LoRA freezes the model and trains small extra matrices, called adapters, next to the frozen weights. Hugging Face's PEFT library (parameter-efficient fine-tuning) does the wiring. Our FastWAM recipe told PEFT to train the whole action expert. The model reaches that expert through an alias PEFT cannot see. The run went for nearly five hours, published a checkpoint and passed the reload check. It never updated its action encoder or action head, the layer that turns the model's internal state into joint commands.
The repository's TRAPS document lists more. Most of these five cost us GPU time, and the last one deserves its own section.

The trap that made everything else worth it

FastWAM is a world-action model: it predicts future video frames and robot actions together, built on the Wan2.2 video model. A video expert and an action expert share one backbone, about 24 GB of weights in total. Our LoRA recipe put adapters on the video expert and trained the action expert and the proprioception encoder in full. A 20k-step run on an NVIDIA H100, on the four-task SO-101 dataset, finished with a smooth loss curve. The run published an adapter and a merged checkpoint, which is the adapter folded back into full model weights, and passed the reload check. We had seen a reload disagree with its evaluation before. We blamed sampling noise and let the recipe run the reload check in a lenient mode. The output still had to have the right shape, but its values no longer had to match. That was our mistake, and it is what let this bug through.

The held-out error was 8.85 times the hold-still baseline. The policy was worse than a robot that does not move.

Two clues pointed at the cause. First, the action encoder and head in the merged checkpoint still looked freshly initialized. Their largest weights sat just inside the bounds PyTorch's default initialization draws from (0.408 for the encoder, 0.031 for the head). After 200 steps of real training, the weights already push past those bounds. They had never been updated. Second, two reloads of the same checkpoint disagreed with each other by a wide margin. A deterministic policy cannot do that.

The cause was that the two libraries reached the action expert by different routes. PEFT's  modules_to_save  wraps a named module in a trainable copy and routes calls through the copy. FastWAM reaches its action expert through an alias set with  object.__setattr__ , which PEFT's module walk does not see. So the forward pass kept using the original frozen expert. The trainable copy received no gradients. The evaluation and each reload built the base model afresh and loaded the adapter onto it. The base checkpoint's action encoder and head were trained for a different robot, with seven action values instead of the SO-101's six. Their shapes do not match, so LeRobot drops them at load time and keeps freshly initialized ones. Every process therefore got its own random encoder and head through the alias, and that is why they disagreed with one another. The "merged" checkpoint carried PEFT's trainable copy, still at its random initialization and never updated. Nothing raised an error, because nothing was wrong from either library's point of view.

The fix had two parts. The recipe now names the leaf modules the forward pass actually uses, such as  mixtures.action.action_encoder  and  mixtures.action.head , instead of the whole expert. And the checkpoint loader that the evaluation and the reload check share now rebuilds a fresh policy from the merged weights. Before, it used the merged model in place, where the alias still pointed at the untouched original. The same 20k-step budget then landed at 2.01 times the hold-still baseline, and 60k steps at 1.26 times. That is still not better than holding still on the four-task SO-101 dataset it trained on, so the policy stays "smoke-tested" rather than "trained", and the recipe says so. But the number now measures the model, not a bug.

We changed two things because of this. The reload check is strict again for recipes that produce one deterministic answer: they must reproduce their evaluation to within 0.0001, and the FastWAM recipes now do, with a difference of exactly zero. And a registry test now rejects the FastWAM LoRA recipe if it names the whole expert instead of its leaf modules. One honest caveat: the smoke test's "weights changed" check did not catch this bug and still would not. For an adapter checkpoint it looks at the LoRA matrices, and those had trained all along. The reload check saw the untrained head twice. First in strict mode, when it failed and we blamed sampling noise. Then in lenient mode, where the spread it still recorded, next to a plain FastWAM run that reloaded exactly, pointed at the LoRA recipe. That is why we will not relax it again.

So, with the gates in place, did any of it work?

Did it work? The numbers

All five training runs below ran on one  ml.g6e.4xlarge  (an NVIDIA L40S with 48 GB) with network isolation on. The dataset is  lerobot/svla_so101_pickplace : 50 episodes of an SO-101 arm picking and placing an object, two cameras, six joints. Episodes 0, 10, 20, 30 and 40 were held out. The metric is the mean absolute joint error on the held-out frames (every frame for ACT, every tenth frame with four draws averaged for the four vision-language-action models), as a fraction of the hold-still baseline. Below 1.0 means the policy beats a robot that does not move. The chunk is the number of future actions a policy predicts at once.

Lower is better. The baseline grows with the chunk length, so ratios compare only within one chunk l

Lower is better. The baseline grows with the chunk length, so ratios compare only within one chunk length: SmolVLA, pi0 and pi0.5 predict 50 actions at once, ACT 30, X-VLA 24. The order is not a ranking.

policy
chunk
steps × batch
wall clock
cost
error vs hold-still
ACT
30
100k × 8
1 h 54 min
~$10 (about $5 on Spot)
0.42
SmolVLA
50
20k × 16
1 h 23 min
~$7
0.38
X-VLA
24
30k × 16
3 h 51 min
~$20
0.46
pi0
50
20k × 8
4 h 51 min
~$26
0.37
pi0.5
50
20k × 8
6 h 22 min
~$34
0.35
ACT trains from scratch; X-VLA fine-tunes a published base; SmolVLA, pi0 and pi0.5 train an action expert on a frozen vision-language backbone (SmolVLM2 for SmolVLA, PaliGemma for pi0 and pi0.5).

Costs are the on-demand list prices of the Region we ran in, Asia Pacific (Tokyo), in September 2026; us-east-1 prices differ. The ACT run repeated on SageMaker managed Spot training finished without interruption and was billed 51 percent less.

Read the table with care. The hold-still baseline grows with the chunk length, so the ratios compare only within one chunk length. The rows sit side by side; they are not a ranking. Within the chunk-50 trio, pi0.5 scored best at 0.35 for $34. pi0 came next at 0.37 for $26. SmolVLA reached 0.38 for $7: within 0.03 of both, for a fifth of pi0.5's cost and a quarter of pi0's. ACT's 0.42 and X-VLA's 0.46 each stand alone. What the table does show is that every model, from 52 million to 3.6 billion parameters, beat the hold-still baseline on all six joints of a 50-episode single-task dataset. The from-scratch ACT run did so in under two hours for about $10.

ACT is the shipped default for a different reason. We have one like-for-like comparison, recorded before this repository on a private 80-episode single-task dataset. ACT and X-VLA used the same chunk length and the same number of training samples. ACT reached a 16 percent lower held-out joint error with 17 times fewer parameters, at a fraction of the cost, and it carries no upstream weights to license. So the vision-language-action policies are opt-in. If your dataset has several tasks or a language instruction matters, SmolVLA is the license-clean first step. pi0.5 scored best among the chunk-50 models but inherits the Gemma Terms of Use.

The other 11 policy types the sample offers (Diffusion Policy, VQ-BeT, TD-MPC, Multi-Task DiT, EVO-1, EO-1, pi0-FAST, GR00T, MolmoAct2, FastWAM and LingBot-VA) passed their smoke test and are in the repository as "smoke-tested". The pipeline runs them end to end, but we have not spent the hours to say whether they learn well here. They ran on the same pick-and-place dataset or one of two other public ones: a four-task SO-101 set ( hjkso1406/so101-4tasks-100eps ) for most of the language-conditioned policies, and  lerobot/xarm_lift_medium  for TD-MPC, a public v3.0 dataset with reward labels. Three of them (pi0-FAST, EO-1 and FastWAM) plus the LoRA recipe for FastWAM need an 80 GB GPU. Their smoke tests ran on Amazon EC2 hosts with NVIDIA H100 GPUs, with the same container image, input channels and training flags as a SageMaker job. The other 13 policy types ran end to end as SageMaker jobs.

The runs taught us things no benchmark would.

Lessons from running it for real

Let the pipeline pick the instance. Quotas are per instance type, per Region and per account, and new accounts start at zero.  --instance-type auto  walks the policy's registry list of instance types in order, skips multi-GPU types unless you pass  --allow-multigpu , and takes the first one with a non-zero quota in your account and Region. So a run does not stop on one quota you forgot to request. If no listed type has quota, it stops before submitting anything and names the quotas to request.

The smoke test's step rate is faster than what you are billed for. Billable time includes start-up, the held-out evaluation and publishing, so steps per billable second is lower than the training loop's own rate. ACT's 100k steps took 6,822 billable seconds, or 14.7 steps per billable second, below the rate its smoke test had measured. The hours and dollars you confirm in the plan come from the registry's step rate for that GPU, and that estimate errs long. After the smoke test, the  train:  line re-estimates the hours from the smoke's rate, and that number comes in a little short. The four vision-language-action runs held 96 to 99 percent of their smoke rate in steady state. Per billable second they delivered only 85 to 95 percent of it; start-up, the evaluation and checkpoint writes took the rest.

Checkpoints are the real storage bill. A pi0-class checkpoint is about 19 GB: 16 GB of 32-bit (fp32) weights plus 2.5 GB of optimizer state. The pipeline keeps the newest three by default and prunes the rest during training. Set  keep_checkpoints: 0  to keep everything.

Multi-GPU instances buy nothing here. The trainer is one process on one GPU. A four-GPU box costs 2.5 to 4 times more for the same step rate, so the run refuses one at plan time unless you ask for it explicitly with  --allow-multigpu .

Spot works for this workload. Checkpoints are mirrored to S3 as they are written, and the job resumes from the last one. The ACT run saved 51 percent. The maximum wait time is set to twice the runtime cap, which bounds how long a Spot job can take end to end.

Licenses are part of the pipeline, not a footnote. Several base models come with their publishers' conditions: PaliGemma under the Gemma Terms of Use, GR00T under the NVIDIA Open Model License, Qwen2.5-VL under a research license, MolmoAct2 for research only. The registry records the license class and the terms for each. The plan refuses that policy until  accept_licence: true  is set for it, and the job checks again before training. Where the registry asks for it, you also add an  approval_ref , your own sign-off note such as a ticket number. That applies to the research-only entries EO-1 and MolmoAct2, the Wan2.2-based world-action models FastWAM and LingBot-VA, and GR00T. The job also writes a  LICENCE_NOTICE.md  next to every output. The sample code is released under the MIT-0 license. The repository distributes no model weights and no datasets; your job downloads the weights into your account under the publisher's terms. Production or commercial use of those models needs your own legal review.

Write the hold-still baseline next to every number. It is the single habit that turned a beautiful loss curve into a bug report.

Those lessons are about the jobs. The next section is about the account they run in.

Security posture, briefly

Training jobs run with network isolation on and an image pinned by digest. The SageMaker execution role has no managed policy. It reads one bucket and writes only under the job and hub prefixes, and its trust policy admits SageMaker only for this account's resources. The bucket blocks public access and refuses non-TLS requests. cdk-nag, a rule checker for CDK stacks, runs every time the stack is synthesized, and every suppressed rule has a written reason. CI scans the tree for account ids, keys, tokens, private addresses and internal hostnames. Before a release we scan the git history too.

What is left to you is listed in the repository's SECURITY.md and in a STRIDE threat model, a table of threats sorted into six standard classes. The training container runs as root because the SageMaker runtime starts it that way. ECR tags stay mutable because the CodeBuild layer cache needs one; jobs pin the image by digest from SSM, so a moved tag changes nothing they run. The bucket has no versioning. Checkpoint pruning deletes old checkpoints during training, 16 to 30 GB each for the large models such as pi0, and versions would keep paying for all of them. If you share the bucket, enable versioning with a short expiry for old versions. By default  cdk destroy  also empties and deletes the bucket and the repository; deploy with  -c retainData=true  to keep them, and the token secret with them.

That is the posture you get out of the box. Here is the first run.

Try it

You need an AWS account where you can deploy a CDK stack, Python 3.10 or newer, the AWS CLI v2, and Node.js 22 with  npm i -g aws-cdk . No GPU and no Docker on your laptop; the training image is built in CodeBuild.
git clone https://github.com/aws-samples/sample-lerobot-policy-finetuning-on-sagemaker
cd sample-lerobot-policy-finetuning-on-sagemaker
python3 -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt -r cdk/requirements.txt
export AWS_REGION=us-east-1 AWS_DEFAULT_REGION=us-east-1
cd cdk
cdk bootstrap
cdk deploy
cd ..
./finetune.sh --dry-run # checks the dataset, waits for the image build if needed, prints the plan, submits nothing
./finetune.sh # asks once, then runs the smoke test and the ACT training job
The default configuration downloads the public SO-101 dataset and trains ACT. The measured quick start took about 2 h 20 min and about $11 in total, from  cdk deploy  through the smoke test and the training job. That includes the one-time image build (about 15 minutes, about $0.50 of CodeBuild). To train on your own recordings, copy your LeRobot v3.0 dataset folder to  datasets/<name>/ , then set  dataset.name  to that folder name and  dataset.source  to  local  in  project_config.json .  ./finetune.sh --compat  then tells you which of the 16 policies accept it before anything is submitted. To try a vision-language-action model, flip its  enabled  flag, read the license terms the README links, and set  accept_licence  where the README's table asks for it (SmolVLA needs no acceptance). pi0, pi0.5, pi0-FAST and GR00T also need a Hugging Face token from an account that has accepted the gated repositories' terms.  ./finetune.sh  asks for it before the staging job and stores it in Secrets Manager.

Three practical notes before you start. Request your  ml.g6e.4xlarge  training-job quota early; new accounts default to zero. Policies that start from Hub weights also need  ml.m5.4xlarge for training job usage  for their CPU staging job. Next, read what the smoke test reports. Losing to hold-still after 200 steps is normal for most policies and does not fail the gate. A checkpoint that fails to reload, weights that did not move, or a parameter stored in bfloat16 does fail it. The training job is never submitted, and finding that out cost under a dollar for ACT. And when you are done, run  cdk destroy lerobot-finetune-infra  from the  cdk  folder. It removes the stack with the bucket, the checkpoints in it and the image repository. Copy out what you want to keep first, or redeploy with  cdk deploy -c retainData=true  before you destroy; that setting is read at deploy time.
If a policy you care about is missing from the registry, open an issue. Adding one is adding a YAML file, and the smoke test, the reload check and the hold-still baseline come with it.


Authors

 Dinesh Mane   is an Applied Scientist at AWS, specializing in generative AI, ML and physical AI.
 Ozan Cihangir  is a Prototyping Architect at AWS, specializing in generative AI, robotics and IoT.

Series: Prototype (3 articles)

  1. 3
    Fine-tuning 16 Hugging Face LeRobot policies on Amazon SageMaker: what it actually takes This article
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article