Why robot fleet benchmarks can be gamed
FaultForge: generate the rare, fleet-wide failures robot benchmarks miss, with known causes, then audit the data so no shortcut can pass.
FaultForge: synthetic failure data for robot fleets · Part 1 of 6
Robot fleets fail in ways single-robot datasets never show, and the benchmarks meant to catch those failures can be passed without any reasoning. This series introduces FaultForge: it generates rare fleet-wide failures with known causes, then audits the data so no shortcut can pass.
Key terms in this article
AMR: autonomous mobile robot, the wheeled robots that move goods around warehouses.
AP: access point, a Wi-Fi radio the robots use to talk to the fleet manager.
RCA: root cause analysis, working back from symptoms to the fault that started them.
SOTA: state of the art, the best reported result on a benchmark.
A warehouse runs 400 autonomous mobile robots (AMRs). At 02:10, a firmware update reaches six Wi-Fi access points (APs) in zone B, starting with AP-17. By 02:16, about thirty robots in two aisles are slowing down, re-localizing and queueing for the same charger bank. The fleet dashboard shows a flood of alerts: low battery, path blocked, localization lost, task timeout. Which one was the cause?
An experienced operator might get there in an hour. An AI agent trained on single-robot failure clips has never seen anything like it. A failure like this might happen only a few times a year at a site, so you can't wait for enough examples to train on. And if you build a benchmark for this kind of diagnosis, there's a good chance a model can score well on it without doing any diagnosis at all.
Those are the two gaps.
Gap 1: the failures you need are the ones you don't have
Fleet operators collect telemetry from every robot all day, but almost all of it describes normal operation. The incidents that matter most, the ones that take down a whole zone during peak hours, are rare and expensive. One maintenance-software vendor estimates that a single unplanned sorting-robot breakdown during peak fulfillment can cost $15,000 to $40,000 in delayed shipments and missed service level agreements (SLAs) . More broadly, Aberdeen Research's widely cited figure puts the average cost of unplanned downtime in manufacturing at about $260,000 an hour .
What makes fleet failures hard is that they spread through things the robots share:
- Floor layout. A robot stopped in a narrow aisle blocks everyone behind it.
- Task dependencies. A stalled pick holds up the pack station that's waiting for it.
- Shared infrastructure. Wi-Fi APs, charger banks and power circuits serve many robots at once.
- Firmware and hardware cohorts. A bad release hits every robot running it, wherever it is on the floor.
Single-robot failure datasets don't capture any of this. They show one arm dropping one object, or one robot missing one waypoint.
Gap 2: benchmarks a shortcut can pass
The second gap is less obvious and, in my view, more dangerous. On LIBERO, a widely used robot-manipulation benchmark, independent 2026 audits found that models score well without doing the task:
- A probe with no language encoder, about 90 million parameters, scored at or near reported state of the art (SOTA) . It never read the instruction.
- A policy of about 540,000 parameters hit 95.1% average success across the four standard suites. A model that small almost certainly isn't reasoning about the task.
- Move the camera or start position a little and top scores fall from 95% to below 30% .
- Across six systems, clean scores differ by just 1.58 points, while robustness and paraphrase scores span 14.62 and 31.08 . The clean number barely separates them.
The benchmarks are still useful. But a headline score can reflect a shortcut in the data, not the skill the benchmark claims to measure.
I ran into the same problem in telecoms. In my Autonomous network operations center (NOC) series , a log-blind classifier that never read the incident logs scored an F1 score of 1.0 on three reproduced public root cause analysis (RCA) benchmarks. It predicted the root cause from the mix of alarm codes alone. That result came from the internal research behind that series. It hasn't been peer reviewed, but the pattern matches what the robotics papers now report.
Fleet diagnosis has plenty of tempting shortcuts:
- Blame the first robot to raise an alarm.
- Blame the robot with the most severe alert.
- Blame whichever robot just got new firmware.
If a synthetic dataset lets those shortcuts score well, a model trained or tested on it learns the shortcut.
Generate, then audit
FaultForge closes both gaps together. It has two halves:
- A generator that produces rare fleet-wide failure scenarios with a known cause for every symptom, shaped by the real layout and infrastructure of a site.
- An auditor that blocks any generated dataset a no-reasoning shortcut can pass, before anyone trains or evaluates on it.
A generator without an auditor scales up whatever shortcuts it happens to create. An auditor without a generator has nothing hard enough to check.
As Figure 1 shows, the pipeline runs from the fleet graph through the generator and the audit gate to export.
FaultForge works at the level of fleet operations and diagnostics: it produces events and telemetry, not camera images or point clouds. It's designed to sit alongside simulators such as NVIDIA Isaac Sim. The simulator renders the world, and FaultForge decides which failures to put in it and records why each one happened.
What's proven, and what isn't
The methods come from work I've already published:
- Multi-kernel graph generation of synthetic telco faults (Autonomous NOC series ).
- The Degeneracy Auditor from the same series.
- Temporal graph networks for real-time fraud detection (part 1 ).
Those methods were built and tested in telecoms and payments. Robot fleets are the next domain, and I haven't yet validated FaultForge on a real fleet. Each post says what has been measured and what is still design.
What's in this series
- Why robot fleet benchmarks can be gamed (this post)
- Mapping how a fault spreads across a robot fleet: a multi-kernel graph model for each fault's footprint
- Why failure data needs time order, not just snapshots: requirements for temporal fault generation, and how to evaluate it
- Auditing synthetic failure data before you train on it: shortcut caps, planted distractors and validity checks
- A reference architecture on AWS: Amazon Neptune, Amazon SageMaker AI, an audit gate in AWS Step Functions, and exporters to simulation formats
- What's still open: validation limits, the gap between telemetry and pixels, and where this goes next
Do you run a robot fleet and have failure stories that a benchmark would never capture? Tell me in the comments, because those stories make the best test cases.
References
- Oxmaint, autonomous sorting robot maintenance guide: https://oxmaint.com/industries/delivery-operations-management/autonomous-sorting-robot-maintenance-guide-warehouse-automation
- The real cost of unplanned downtime in manufacturing, citing Aberdeen Research (Reliability Magazine, 2026): https://reliamag.com/articles/cost-unplanned-downtime-manufacturing/
- LIBERO shortcut audit (arXiv 2606.04233): https://arxiv.org/pdf/2606.04233
- Small-policy LIBERO result (arXiv 2609.03715): https://arxiv.org/abs/2609.03715
- Perturbation sensitivity (arXiv 2510.13626): https://arxiv.org/abs/2510.13626v1
- Robustness and paraphrase spread (arXiv 2609.25562): https://arxiv.org/html/2609.25562
- The Autonomous NOC, a 12-article series
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article