Robot Fleet Orchestration with Cloud-Assisted VLA
Cube robots push boxes without bumping into each other. The secret? A VLA loop across edge and cloud, plus MAPF for collision-free paths.
Introduction
What comes to mind when you hear "Physical AI"?
Robotic arms autonomously grasping objects, VLA (Vision-Language-Action) models, imitation learning, reinforcement learning — these are probably the keywords most people associate with the term.
At AWS, we take a broader view of Physical AI. In short, it encompasses any system that integrates perception, understanding, reasoning, and learning to interact with the physical world .
In this post, we explore this broader notion of Physical AI using AWS services and small robots. We built a demo in which an AI agent orchestrates multiple robots to transport packages. Take a look at the demo video first:
The VLA Loop via the Cloud
Most Physical AI articles and hands-on tutorials deploy a VLA model directly on the robot or edge device. However, not every robot has the GPU or memory to run a VLA model. Small robots like the ones we use here certainly don't — and in practice, edge resources are often limited.
Even on a robot powerful enough to run a VLA model, each robot's egocentric camera has a limited field of view. Obstacles behind shelves, the positions and headings of other robots, the overall spatial layout — these cannot always be captured by a single onboard camera.
Now consider what becomes possible with an overhead camera looking down at the entire field. The AI can integrate information that no individual robot could perceive on its own.
- Vision: An overhead camera captures the full state of the field.
- Language: An LLM in the cloud interprets the situation and plans high-level tasks.
- Action: Commands are sent to the robots to manipulate the physical world.
If you look closely, this is the VLA loop itself — realized not on each robot, but through the cloud. This approach can complement an onboard VLA model where one exists, and enable intelligent behavior even on robots that cannot host a VLA model at all.
Prerequisites
To reproduce this demo you will need:
- toio — 2 or more units + toio mat
- USB camera
- Edge PC (we used an M1 Mac)
- AWS account
- Source code (Download here » )
toio robots report their coordinates and can be commanded to move to specific positions via API on a dedicated mat. Placing a dark border or background sheet under the mat improves detection accuracy. Mount the USB camera above the toio mat looking straight down. Refer to the README.md in the download package for detailed setup instructions.
Architecture
Here is the overall architecture. We use AWS IoT Core as a message bus — robot commands, AI reasoning traces streamed in real time, and other coordination are all handled through MQTT topics.
End-to-End Flow
The system works as follows:
- map_parser captures the field with the camera, runs YOLOE (described below) to detect objects, divides the toio mat into a 5×7 grid map, and publishes it via MQTT to AWS IoT Core (topic:
toio/map). - An IoT Rule triggers AWS Lambda . The AI agent (built with the Strands Agents SDK) uses Amazon Bedrock (Claude Sonnet) to analyze the grid map and decide task assignments. It then invokes the CBS Solver — exposed as a tool — to compute collision-free paths for all robots.
- The computed paths (waypoints) are published via MQTT (topic:
toio/waypoints). - Bedrock's reasoning trace is streamed in real time over MQTT (topic:
toio/agent/thinking). - walker receives the waypoints and issues BLE calls to the robot API. The robots push packages to their goal positions.
- After movement completes, map_parser publishes an updated map. The loop continues until no packages remain.
At a high level, you can see a two-layer architecture: edge-side object detection and cloud-side route planning. Let's look at each in detail.
Object Detection on the Edge
On the edge, we need to detect robots, packages, and obstacles from the camera feed.
Typically, training an ML model for object detection requires a large amount of labeled data per object class — a significant effort.
We used YOLOE's Visual Prompts feature. With Visual Prompts, you simply provide a single reference image and draw bounding boxes around the objects you want to detect. One-shot labeling is complete, and the model can detect objects without any training. This workflow is handled by the included scripts (
tools/capture_reference.py and tools/draw_bbox.py).Running on an M1 Mac at the edge, we achieve approximately 4–5 FPS — enough for near-real-time field awareness.
Converting to a Grid Map
Detection results are converted into a 5×7 grid map. Each cell is labeled as "empty," "robot," "cargo," or "obstacle."
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{
"grid": [
["__", "__", "__", "GT", "__", "__", "__"],
["BT", "__", "__", "__", "PA", "__", "__"],
["__", "__", "BL", "__", "BL", "__", "__"],
["__", "__", "PA", "__", "PA", "__", "__"],
["__", "RT", "__", "__", "__", "__", "__"]
],
"legend": {
"RT": {"name": "Red Toio", "type": "robot"},
"BT": {"name": "Blue Toio", "type": "robot"},
"GT": {"name": "Green Toio", "type": "robot"},
"PA": {"name": "Package", "type": "cargo"},
"BL": {"name": "Block", "type": "obstacle"}
}
}This grid map serves as the input to the cloud-side AI agent. The role of map_parser is to transform unstructured camera data into structured data the AI can work with. By sending a compact JSON representation instead of raw images, we also reduce cloud-side processing overhead and response latency.
Route Planning in the Cloud
On the cloud side, an AI agent built with the Strands Agents SDK running on Lambda receives the grid map and makes decisions.
The LLM handles route planning (task planning). Specifically, it:
- Interprets a directive like "clear packages to the edge" into concrete grid coordinates.
- Decides which robot should handle which package (e.g., assigning the nearest one).
- Considers obstacle placement to select reachable goals.
- Decides to move blocking packages out of the way first.
- Falls back to moving one robot at a time when no collision-free solution is found for concurrent operation.
- Splits the work into multiple rounds when there are more packages than robots.
The actual path computation is delegated to the CBS Solver, implemented as an @tool in the Strands Agents SDK. Once the LLM determines the tasks, it invokes the CBS Solver via tool calling to compute collision-free paths.
Multi-Agent Path Finding
The problem of finding collision-free paths for multiple robots moving simultaneously is known as MAPF (Multi-Agent Path Finding). In this demo, we use CBS (Conflict-Based Search), one of the most well-known MAPF algorithms.
CBS operates as a two-level search:
- Low-level search: Compute the shortest path for each robot independently (standard A* search).
- High-level search: Check whether the computed paths conflict with each other. If a conflict exists, add a constraint and re-run the low-level search.
Let's make this more concrete. First, each robot computes its shortest path ignoring all other robots. Naturally, paths may intersect. For example, suppose we find the conflict: "at time step 3, Robot A and Robot B occupy the same cell."
CBS adds a time-indexed constraint — "Robot A may not use that cell at step 3" — and re-computes only Robot A's path. The new path might conflict elsewhere. If so, another constraint is added and the process repeats until all conflicts are resolved.
Rather than computing all robot paths jointly in a single high-dimensional search, CBS surgically fixes only the points of conflict. This is what makes it efficient.
In standard MAPF/CBS, robots simply navigate to their goals. In our demo, however, the robots push packages — much like Sokoban (a classic box-pushing puzzle game). This means the state space is no longer just "robot position" but the pair "(robot position, package position)."
Multiple robots each pushing a package, avoiding each other and obstacles — essentially multi-agent Sokoban. This is more complex than standard MAPF, but solvable by extending the CBS framework.
For more on MAPF, the paper Conflict-Based Search For Optimal Multi-Agent Path Finding is an excellent reference.
AI Agent Reasoning Trace
Let's look at how the AI agent reasons, as streamed via MQTT:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
Analyzing the map.
0 1 2 3 4 5 6
0 __ __ __ GT __ __ __
1 BT __ __ __ PA __ __
2 __ __ BL __ BL __ __
3 __ __ PA __ PA __ __
4 __ RT __ __ __ __ __
- Robots: GT(0,3), BT(1,0), RT(4,1)
- Packages: PA(1,4), PA(3,2), PA(3,4)
- Obstacles: BL(2,2), BL(2,4)
- Goal: Pack 3 packages into the top-left corner
Strategy (attempt 1)
- 🔵 BT → PA(3,2) → (0,0)
- 🔴 RT → PA(3,4) → (0,1)
- 🟢 GT → PA(1,4) → (0,2)
CBS Solver call: No solution. Goal assignments conflict with each other.
Strategy (attempt 2) — reassign and retry
- 🟢 GT → PA(1,4) → (0,0)
- 🔵 BT → PA(3,2) → (0,1), route around BL(2,2)
- 🔴 RT → PA(3,4) → (0,2)
CBS Solver call: Solution found (GT: 11 steps, BT: 12 steps, RT: 15 steps)
All 3 robots move in parallel, routing around BL to converge packages in the top-left.
Analyzing the map.
0 1 2 3 4 5 6
0 PA PA PA RT __ __ __
1 GT BT __ __ __ __ __
2 __ __ BL __ BL __ __
3 __ __ __ __ __ __ __
4 __ __ __ __ __ __ __
Packages are already at (0,0), (0,1), (0,2). No movement required.
ALL_DONEOn the first attempt, the CBS Solver could not find a solution. The agent autonomously adjusted the goal assignments and retried — finding a valid solution on the second attempt.
After the robots physically move and deliver all three packages, Round 1 is complete. map_parser then publishes an updated map to verify the state. If more packages remain (when there are more packages than robots), the system continues looping through additional rounds until everything is cleared.
This "think, try, adapt" process is exactly what makes AI agents powerful. A rigid rule-based system would struggle with this kind of flexible problem-solving.
Conclusion
In this post, we demonstrated robot fleet orchestration as a cloud-assisted VLA loop. Rather than replacing onboard VLA models, this approach complements them — making capable robots smarter and enabling even resource-constrained robots to operate autonomously.
We used toio — a small, affordable robot — with a minimal architecture to keep the demo accessible, but the orchestration approach itself is robot-agnostic. The possibilities are wide open depending on your use case and creativity.
We plan to showcase an enhanced version of this demo at AWS Summit Japan in June. If you get the chance, stop by our booth — we'd love to meet you!
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article