Reachy Mini x Strands Agent on Jetson Thor - Part 5: The Voice-Assistant Loop — Wake, Look, Speak
Part 5 of the series: the centerpiece loop — \"Hey Reachy\" -> transcribe offline -> a brand-new per-wake Strands Agent that picks one of twelve tools -> speak one short sentence -> tear down. Idle is $0; the agent exists only between wake and reply.
Series: Reachy Mini x Strands Agent on Jetson Thor (14 articles)
- …
- 5Reachy Mini x Strands Agent on Jetson Thor - Part 5: The Voice-Assistant Loop — Wake, Look, Speak This article
What would I like to build?
This is Part 5 of the series — and the centerpiece. Part 3 gave the robot ears and Part 4 gave it eyes. This post is the loop that ties them — and everything still to come — together: "Hey Reachy" → transcribe → a fresh per-wake Strands Agent that picks a tool → speak one short sentence → tear down.
The design goal is a robot you can leave switched on without it costing anything to sit idle. So the loop is cost-minimal by construction: while waiting, only the offline Vosk wake listener runs — no agent, no model loaded into the loop, no backend calls, $0. The Strands Agent exists only between a wake and a reply. On wake it's built fresh around the one request, it reasons with its model, calls exactly the tool it needs, speaks a single sentence, and is then deleted and garbage-collected. Nothing LLM-related survives to the next wake.
Two trigger sources feed one worker — the voice wake word and an optional AWS IoT Core MQTT subscription — so the robot has a single owner and is never driven by two sources at once.
Goals
- Tie the wake layer, vision, motion, emotions, and data-lake tools into one wake → think → act → speak loop
- Keep idle at $0 — only the offline Vosk listener runs until a wake
- Build a brand-new agent per wake and destroy it afterwards, so no LLM process persists between interactions
- Give the agent its full twelve-tool surface and let it pick the right one for the request
- Bound every wake with a hard
ModelCallBudgetso cost and latency can't run away - Reply with one short spoken sentence via local TTS (Piper → espeak-ng → print), never raw JSON
- Accept voice and MQTT triggers through one worker that owns the robot
- Print a per-wake cost/latency summary so every interaction is accountable
The Overall System
The loop is a pipeline with a single robot owner at its center. A trigger (voice or MQTT) enqueues a request; one worker thread drains the queue;
handle_wake builds a fresh agent; the agent reasons and calls a tool; the reply is spoken; the agent is torn down; and the loop returns to idle.
System Components:
- Triggers — the offline Vosk wake word and an optional AWS IoT Core MQTT subscription, both enqueuing to one queue
- Worker — a single thread that drains the queue so the motors and agent are never driven by two sources at once
handle_wake— builds the fresh per-wakeAgent, runs it under the budget, speaks the reply, tears it down- Model — local Nemotron via Ollama ($0) by default, or Amazon Nova 2 Lite on Bedrock
- Twelve tools — vision, six motion primitives, emotions (×2), and three data-lake tools
- Speaker — local TTS via Piper, falling back to espeak-ng, then print
Sequence Diagram
Follow one full wake — trigger, route, build, reason, tool, speak, tear down — and back to idle.
Architecture
handle_wake(request) is the per-wake core. Every wake it constructs an Agent with the full toolset and a cost guard, runs it, and destroys it:1
2
3
4
5
6
7
8
9
10
budget = ModelCallBudget(MAX_MODEL_CALLS)
agent = Agent(
model=_build_model(),
system_prompt=sys_prompt,
tools=[look_and_describe, play_emotion, list_emotion_moves,
nod, shake_head, look_around, wiggle_antennas, spin_body, move_head,
list_iot_tables, get_table_schema, query_iot_data],
hooks=[budget],
)
result = agent(request)The twelve tools span every capability in the series:
| Group | Tools | Covered in |
|---|---|---|
| Vision | look_and_describe | Part 4 |
| Motion | nod, shake_head, look_around, wiggle_antennas, spin_body, move_head | Part 1 |
| Emotion | play_emotion, list_emotion_moves | Part 8 |
| Data lake (S3 Tables) | list_iot_tables, get_table_schema, query_iot_data | Part 6 |
The system prompt steers the agent to discover-before-guess and to always reply with one short spoken sentence — never raw JSON, table dumps, or column lists.
How it works
Idle costs nothing
While waiting, only an offline Vosk wake-word recognizer runs against the mic stream. No Strands Agent exists, no model is loaded into the loop, and nothing is sent to a backend — the idle path is
$0. A second trigger, an AWS IoT Core MQTT subscription, enqueues to the same worker, so a wake can come from voice or a published message. Both feed one worker that owns the robot.On wake: head up, then transcribe offline
When a wake token matches and the worker is not already busy,
main enables the motors and raises the head to neutral as the "I'm listening" cue, then transcribes the request — still fully offline, still $0, before any backend is touched:1
2
3
mini.enable_motors()
mini.goto_target(INIT_HEAD_POSE, antennas=INIT_ANTENNAS_JOINT_POSITIONS, duration=1.0)
request = listen_for_command(ar, vmodel) # offline, $0listen_for_command reuses the running arecord stream but builds a fresh KaldiRecognizer so the wake word itself isn't carried into the request. It returns when Vosk reports end-of-utterance with non-empty text, or after LISTEN_SECONDS (default 8) as a backstop.A brand-new Strands Agent per wake
The model comes from
_build_model(): by default LLM_BACKEND=ollama runs a local Nemotron model through Ollama on the Thor GPU ($0); LLM_BACKEND=bedrock runs Amazon Nova 2 Lite on Bedrock instead. ModelCallBudget is a HookProvider registered on BeforeModelCallEvent; it counts every model call and raises once the count passes MAX_MODEL_CALLS (default 12), hard-capping a single wake. The cap is set high enough for a data-lake question, whose discover → schema → query → answer cycle needs several round-trips.The request reaching the agent is routed first: a spoken request beginning with the emotion prefix becomes a
play_emotion instruction; otherwise it passes through unchanged so the agent stays generic.One short spoken sentence, then teardown
The agent's final string is cleaned, spoken, and accounted for:
1
2
3
4
result = agent(request)
_print_summary(result, budget, t_wall)
reply = _clean_reply(str(result))
speak(reply)_print_summary prints the backend and model, model calls : <n> / MAX_MODEL_CALLS, token in/out/total counts, an estimated LLM cost ($0 for local Nemotron), latency, cycle count, and wall time — a per-task accounting of exactly what the one wake spent. speak does local TTS through the Reachy speaker: a Piper voice if one loaded at startup (synth a WAV, play via aplay), falling back to espeak-ng, then to printing the reply.Teardown runs in a
finally so it always happens:1
2
del agent # destroy the agent instance
gc.collect()Nothing LLM-related survives between wakes. The head deliberately stays up (
_head_up) rather than dropping to sleep, and the loop returns to idle Vosk-only listening at $0.Technical Challenges & Solutions
Challenge 1: Keeping idle free
Problem: A always-on robot assistant that holds a model resident or polls a backend would cost money and power to do nothing.
Solution: The agent is built only inside
handle_wake and destroyed in a finally. Between wakes, only the CPU-side Vosk listener runs — no agent, no loaded model in the loop, no backend traffic. Idle is genuinely $0, verified by the per-wake summary that reports $0 for local-backend runs.Challenge 2: Two trigger sources, one robot
Problem: Voice and MQTT can both arrive at any time. If both drove the robot directly, two tasks could move the motors or call the agent at once.
Solution: Both triggers enqueue onto one queue drained by a single worker thread. The robot has exactly one owner; a wake heard mid-task is ignored, and the MQTT callback is fire-and-forget so its keep-alive never stalls.
Challenge 3: A runaway agent loop
Problem: A tool-calling agent can loop — reason, call, reason, call — and on a cloud backend that's unbounded cost; on the robot it's unbounded motion.
Solution:
ModelCallBudget caps model calls per wake (default 12) and aborts cleanly when exceeded. The cap is generous enough for the longest legitimate flow (a data-lake question's discover → schema → query → answer) but still bounds the worst case.Challenge 4: Speaking, not dumping data
Problem: Tools return JSON, table rows, and column lists. Read aloud, that's unusable.
Solution: The system prompt requires the agent to answer with one short, natural spoken sentence and never read raw JSON or tables, and
_clean_reply strips any <think> reasoning a reasoning model emits before it reaches speak.Getting Started
Run it
1
./reachy_assistant.sh # say "Hey Reachy"; Ctrl-C to stopreachy_assistant.sh creates the venv; installs reachy-mini, vosk, strands-agents, and piper-tts; downloads the Vosk model and Piper voice once; starts the daemon with --no-media --no-wake-up-on-start (so the controller owns the head and keeps it up for the session); and exports MAX_MODEL_CALLS, LLM_BACKEND, and the model ids before launching reachy_assistant.py. Say "Hey Reachy" and give it a request — a look, a gesture, an emotion, or a data-lake question.What's Next
In Part 6 - IoT Data-Lake Q&A on S3 Tables , I wire the agent to an AWS data lake (S3 Tables / Apache Iceberg) through Lambda and Athena, so "Hey Reachy, has the kitchen water sensor tripped today?" becomes a discover → schema → query → answer flow spoken back in one sentence.
Summary
This post covered the centerpiece loop that turns the parts into an assistant:
- Wake → think → act → speak — offline wake, offline transcription, a fresh per-wake agent that picks one of twelve tools, then one spoken sentence
- $0 idle — only the Vosk listener runs between wakes; the agent is built in
handle_wakeand destroyed in afinally(del agent; gc.collect()) - Pluggable brain — local Nemotron via Ollama by default, Amazon Nova 2 Lite on Bedrock as an opt-in swap
- One robot owner — voice and MQTT both enqueue to a single worker, so two sources never drive the motors at once
- Bounded cost —
ModelCallBudgetcaps model calls per wake (default 12), generous enough for a data-lake question but never unbounded - Accountable — a per-wake summary prints backend, model calls, tokens, estimated cost, latency, and wall time; local TTS speaks the reply via Piper → espeak-ng → printMore in this series
- Previous - Part 4: Local Vision with NVIDIA Cosmos Reason 2
- Next - Part 6: IoT Data Q&A with S3 Tables, Athena and Iceberg
Series: Reachy Mini x Strands Agent on Jetson Thor (14 articles)
- …
- 5Reachy Mini x Strands Agent on Jetson Thor - Part 5: The Voice-Assistant Loop — Wake, Look, Speak This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article