
Fine-Tuning NVIDIA's DreamZero, a 14B World Action Model, on Amazon SageMaker: What It Actually Takes!
Fine-tuning NVIDIA's DreamZero, a 14B video-diffusion World Action Model for robot manipulation, on Amazon SageMaker, end to end. We turn a research repo into a one- command pipeline: bring a LeRobot dataset, get servable merged weights in S3 for ~$110. Along the way: a silent LoRA-loading trap that made checkpoints 9.9× worse, five ways to train a broken model without an error, and what a spot interruption really costs.
Series: Prototype (3 articles)
- 2Fine-Tuning NVIDIA's DreamZero, a 14B World Action Model, on Amazon SageMaker: What It Actually Takes! This article
How we turned a research-grade robot foundation model into a one-command SageMaker pipeline, and the five ways we found to silently train a broken model along the way.

Robot foundation models are having their moment. NVIDIA's DreamZero is a 14-billion-parameter video-diffusion World Action Model: show it camera views and a robot's joint state, and it predicts both the next seconds of video and the action chunk that gets the robot there. Fine-tuned on a few hundred demonstrations of your own robot, it becomes a manipulation policy for your task.
The gap between "here is the research repo" and "here is a servable checkpoint trained on my data" turned out to be wider than we expected. This post is the story of closing that gap on Amazon SageMaker: what we built, what it costs, what we measured, and the failure modes that only showed up when real money was on the line. The result is an open sample you can run with three commands.
Five ways to train a broken model without an error
Every one of these produces a training job that ends
Completed , a loss curve that looks healthy, and a policy that learned the wrong thing. They shaped the pipeline more than any feature did.- Camera order. The embodiment config maps your cameras to top, left and right, and the order is semantic. Swap two and the model trains happily on the wrong views.
- State and action slices. Mis-slice the packed joint vector and the loss curve still looks beautiful.
- Action labels that copy the state. A common data-collection bug. A model that echoes its input scores perfectly on open-loop metrics, which is exactly why you never notice.
- Video frames misaligned with parquet rows. A re-encode that drops frames breaks the frame-to-row correspondence the data loader depends on.
- Serving the raw LoRA directory. The upstream loader composes the adapters onto the wrong base. We measured it: 9.9× worse than the correct composition, and worse than not fine-tuning at all.
The first four are caught by a validation stage that runs before any GPU is billed. The fifth gets its own section below.
What we built
One command takes a LeRobot -format dataset (a HuggingFace repo id or a local folder) and produces servable, merged model weights in your S3 bucket:
python3 pipeline/run_pipeline.py --name my-robot
Under the hood that is nine stages:
fetch → detect → convert → validate → prep → stage → smoke → train → merge└─────────── local (stage uploads to S3) ──────────┘ └─ one SageMaker job each ─┘

The durable infrastructure (an ECR repo, an S3 bucket, the SageMaker execution role) is one CDK deploy. The training image is a SageMaker PyTorch Deep Learning Container plus the pinned DreamZero source and eight environment fixes we needed to make it run reliably as a bring-your-own-container training job. It builds in CodeBuild, so you never need a local Docker daemon.
Three design decisions did most of the heavy lifting.
A validation stage runs before any compute. It checks your embodiment config against the actual dataset: dimension slices, camera names, fps, file completeness, and a scan for action labels that are a copy of the measured state. It exists because of the first four failure modes above, none of which raises an error on its own.
A smoke gate comes before the full run. The pipeline runs the identical container, dataset and hyperparameters for a handful of steps, about $10 and 25 minutes. Read the seconds per step off that job and you know the full run's cost before you spend it, because compute is linear in step count.
An automatic merge stage closes the fifth failure mode, and it deserves its own section.
The trap that made everything else worth it
A LoRA fine-tune produces adapter weights, not a servable model. The upstream repo's loading path composes those adapters onto the raw video-diffusion base rather than the robot-pretrained base they were actually trained against. Nothing errors: the model loads, runs, and produces actions.
We measured what that mistake costs: 9.9× worse overall MSE than the correct composition, which is worse than not fine-tuning at all, and up to 42× worse on the single most affected action dimension. A team without a reference number would ship it and conclude that fine-tuning doesn't work.

So the pipeline's final stage merges the adapters onto the correct base as a SageMaker job and rewrites the checkpoint's config so a loader cannot walk back into the trap. It refuses to overwrite existing merged weights, prefers the training job's verified output, and is cheap to re-run (~$5, ~15 minutes) if it's ever in doubt. The README's rule is blunt: serve from
models/<name>-merged/ , never the raw LoRA directory.Did it work? The numbers
We validated the whole path on the public lerobot/aloha_static_screw_driver dataset (a bimanual ALOHA robot). The acceptance test for the port was parity with a reference run of the same recipe (same hyperparameters, same seed, same four GPUs) on a plain EC2 host. That is not a formality, because the SageMaker run does not execute identical code: the container applies three patches to upstream (one changes the data shuffle stream), pins its own DeepSpeed and flash-attn builds, runs offline, and reads and writes through SageMaker's input channels and checkpoint mirroring. Any of those could have moved the numbers. None did:
Measurement | SageMaker pipeline | EC2 reference |
|---|---|---|
Final training loss (1000 steps) | 0.0957 | ~0.096 |
Open-loop MSE, merged checkpoint | 0.00106 | 0.00100 |
Fine-tuned vs base model on its own data | ~7× lower MSE | — |

Same convergence and the same accuracy, which is what a port has to prove. The table is not a claim that SageMaker trains better than EC2. The next figure shows what that accuracy looks like in the raw evaluation output: the fine-tuned policy's predicted trajectory tracking the ground truth through a full motion.

And the cost of a first servable checkpoint, everything included:
Phase | Wall clock | Cost |
|---|---|---|
CDK deploy + CodeBuild image build (once per region) | ~10–15 min, plus CodeBuild's queue for a host | ~$3 |
One-time base-weight staging (~128 GB) | ~1–2 h | ~$3/month S3 |
Smoke gate (10 steps) | ~25 min | ~$10 |
1000-step LoRA fine-tune, ml.g7e.24xlarge | 4 h 11 m | ~$93 |
LoRA → base merge | ~15 min | ~$5 |
First servable checkpoint | ~6–7 h | ~$110 |
One number we want to flag because it's the kind that misleads: a fine-tune's headline "N× better than base" is usually dominated by a single action dimension with the largest scale. We've seen one dimension carry 94–97% of the base model's total error. Read the per-key table, and always compare against a constant-hold baseline (predict "no movement" for every step). One of our experimental runs scored 21× against the base while losing to constant-hold. Open-loop MSE finds broken checkpoints; it does not crown good ones.
Lessons from running it for real
These are the things no amount of code review found. Each one surfaced only when a real job ran long enough to hit it.
Quota is not capacity, and nothing tells you capacity in advance. An approved SageMaker quota for a GPU instance type says nothing about whether the region currently has the hardware, and there is no API that answers that question ahead of time: dry-run calls ignore capacity, spot placement scores coexist happily with
InsufficientInstanceCapacity , and a job that seats now says nothing about the one you submit an hour later. So stage your weights, submit, and treat a long Pending as normal: it bills nothing until hardware arrives, and cancelling it to retry only loses your place in the queue. Capacity is region-local, so if one region stays dry for a day, another may not be.A LoRA checkpoint should not weigh 95 GB. Ours did: the adapter is under 0.5 GB, but DeepSpeed's resume state carried the entire frozen 14B backbone in every save. The cause is a one-line gate in the Hugging Face Trainer: it only excludes frozen parameters from DeepSpeed checkpoints for PEFT models, and DreamZero injects LoRA itself. A 50-line patch flips that gate on save and loads non-strictly on resume. A checkpoint is now 3.6 GB, resume from it is verified, and a 5000-step run's S3 footprint drops from ~890 GB to ~40 GB.
Let SageMaker carry the checkpoints, but check them before resuming. With saves that size,
CheckpointConfig does exactly what it is for: it mirrors each save to S3 as it lands and restores the prefix on a spot relaunch, and the trainer resumes from the highest checkpoint it finds. The one thing it does not do is check that a checkpoint is whole. A reclaim during a save leaves a torn directory that the trainer would pick over the good one below it and die on ten minutes later. A thirty-line guard in the entrypoint deletes incomplete checkpoints before the trainer looks, and it never fails open: if it cannot judge what was restored, it stops the job rather than train from step 0 over a recovery point it cannot get back.A spot interruption is a data-loss event, not just a delay. When a managed spot job is reclaimed and relaunched, the new attempt reuses the same job name, and therefore the same S3 checkpoint prefix. If the fresh attempt starts from step 0, it overwrites the better-trained checkpoints it should have resumed from, in a bucket with no versioning to recover them from. We learned this the hard way, and the design now has two layers: SageMaker restores the prefix on relaunch and the trainer resumes from the highest checkpoint it finds, while the entrypoint's guard stands between the two, deleting any checkpoint that is incomplete and refusing to guess if it cannot tell. Measured on a 20-step job that was reclaimed three times (before its first save, after
checkpoint-10 , and seconds after its final save), every relaunch resumed from the newest complete checkpoint, the last one skipped straight to staging the finished weights, and the job completed with its artifacts intact. The pipeline exposes this as an opt-in --spot flag.Two region variables feed two different consumers. The CDK CLI reads
AWS_REGION ; boto3 reads only AWS_DEFAULT_REGION . Export just one of them and your infrastructure deploys to one region while your jobs submit to another, with no error anywhere. Export both, every time.The local driver is the fragile half. The SageMaker jobs are server-side and don't care if your laptop sleeps, but the process that carries a finished training run into the merge stage is local. It survives credential rotation (the poller rebuilds its session every poll), and if it dies anyway, one flag re-attaches to a running job instead of paying for a duplicate:
--start-at train --train-job <name> .Security posture, briefly
This is a sample, not a production-certified workload, and the repo says so rather than hoping you won't ask. The CDK stack is linted with cdk-nag on every synth (an unsuppressed finding fails the build), all upstream weights and code are pinned to commit SHAs, the training image is consumed by digest so a re-pushed tag can't change what a queued job runs, and the built image ships with a pip-audit CVE inventory including a per-advisory reachability assessment. The known gaps (root containers, no VPC, S3 versioning off, and friends) are enumerated in
docs/SECURITY.md with a concrete remedy each, because a sample that hides its gaps just relocates them into your account.Try it
Three commands, on the shipped demo dataset:
export AWS_REGION=us-east-1 AWS_DEFAULT_REGION=us-east-1 # both!
cd cdk && pip install -r requirements.txt && cdk bootstrap && cdk deploy./setup.sh --stage-assetspython3 pipeline/run_pipeline.py --name aloha-demo
About six hours later there are servable merged weights in your bucket at
s3://<your-bucket>/sagemaker/models/aloha-demo-merged/ . Bringing your own robot's data is an embodiment config away, and the README walks through it, including the LeRobot v3 → v2.1 conversion most public datasets now need.Two practical notes before you start: request your GPU quota early (
ml.g7e.24xlarge and friends default to zero), and treat the smoke gate as a measurement tool, not a formality. It's the difference between learning your real cost per step for $10 and learning it from a day-old bill.
Authors:
Series: Prototype (3 articles)
- 2Fine-Tuning NVIDIA's DreamZero, a 14B World Action Model, on Amazon SageMaker: What It Actually Takes! This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article