AWS Builder Center

Builder Center

Reachy Mini x Strands Agent on Jetson Thor - Part 9: Idle Presence — Noticing Who's Around (Humans vs Cats)

Reachy Mini x Strands Agent on Jetson Thor - Part 9: Idle Presence — Noticing Who's Around (Humans vs Cats)

Part 9 of the series: while resting, the robot periodically asks Cosmos who it sees — fully local, $0 — and routes the observation through species-specific Strands Agent tools so humans and cats can trigger different behaviours, both, either, or neither firing per glance.

Series: Reachy Mini x Strands Agent on Jetson Thor (14 articles)

  1. 1
    Reachy Mini x Strands Agent on Jetson Thor - Part 1: Driving the Reachy Mini Lite with Strands Agent
  2. …
  3. 9
    Reachy Mini x Strands Agent on Jetson Thor - Part 9: Idle Presence — Noticing Who's Around (Humans vs Cats) This article
  4. 10
    Reachy Mini x Strands Agent on Jetson Thor - Part 10: Real-Time Face Tracking with a Single Camera Owner

Chiwai Chan

Principal AWS Consultant at Datacom - Cloud, IoT, Robotics and GenAI

What would I like to build?

Idle presence watcher: a single frame goes to Cosmos Reason 2, and a minimal agent routes the observation to report_human_presence and report_cat_presence
This is Part 9 of the series. So far the robot only acts when triggered — a wake word or an MQTT message. Now I want it to show a little life between interactions: while resting, it should quietly notice who's in the room and have a place to hang future reactions — greet a person, perk up at a cat.
Every IDLE_INTERVAL seconds the robot grabs a frame, asks NVIDIA Cosmos Reason 2 who it sees, and routes the observation through a tiny Strands Agent. The whole thing is fully local — local Cosmos vision, local Nemotron routing — so each glance costs $0, in keeping with the idle-is-free principle that runs through the build.
The design choice worth highlighting is the species split. Instead of one generic report_presence, the observation routes to two tools — report_human_presence and report_cat_presence — so people and cats can drive different behaviours. A single glance can fire both (a person and a cat in frame), either alone, or neither.

Goals

  • Have the robot periodically notice who's around while idle, fully local and $0
  • Use Cosmos Reason 2 to count people and cats and note what each is doing
  • Route the observation through a minimal, disposable Strands Agent to species-specific tools
  • Support both / either / neither firing per glance — humans and cats are independent
  • Never compete with a real task — skip ticks while the robot is busy
  • On a real detection, record a clip (S3) and publish a presence state (IoT Core) for downstream systems

The Overall System

A daemon thread ticks on a fixed cadence. Each tick samples the shared camera frame, asks Cosmos a counts-per-species question, and hands the answer to a throwaway agent that routes to the human and/or cat tool — which, on a real detection, record a clip and publish a presence event.
Presence split — one observation routed to species-specific human and cat tools
System Components:
  1. _idle_watcher — the daemon-thread loop on an IDLE_INTERVAL cadence
  2. Shared camera buffer — _latest_jpeg_b64(), the same frame the face tracker maintains
  3. Cosmos Reason 2 — answers the counts-per-species IDLE_QUESTION, locally
  4. _run_presence_agent — a minimal Strands Agent built per observation and discarded
  5. report_human_presence / report_cat_presence — the two species tools, each logging and (on a detection) recording + publishing

Sequence Diagram

Follow one tick where a person and a cat are in frame — observation, species routing, both tools firing, and the clip/state upload.
Sequence diagram for reachy-mini-idle-presence-humans-vs-cats

Architecture

ToolArgspresence_kindLog colour
report_human_presencepeople, descriptionhumanmagenta
report_cat_presencecats, descriptioncatcyan
Both are @tool-decorated. Each logs a colour-coded line, and only when its count is >= 1 does it record a short clip (record_clip_and_upload() → S3, presigned URL) and publish a presence state message over IoT Core. The presence agent itself is built and discarded per observation, so nothing persists between ticks.

How it works

The idle watcher

_idle_watcher() runs on a daemon thread started at boot. While the robot is resting it loops on a fixed cadence: every IDLE_INTERVAL seconds (default 10) it grabs the latest frame from the shared camera buffer with _latest_jpeg_b64(), sends it plus IDLE_QUESTION to Cosmos Reason 2, prints the observation, and hands it to a Strands Agent. It is fully local, so each tick costs $0. The default IDLE_QUESTION asks for counts and a short note per species:
1
2
How many people and how many cats can you see? Give the count of each (0 if none)
and a short note on what each is doing.

Skipping ticks while a task runs

The watcher checks the shared _busy event at the top of every tick. If a real task is running it logs idle watcher: skip tick (busy with a task) and waits for the next interval, so the presence check never competes with a live look or drives the GPU mid-task:
1
2
3
4
5
6
7
8
while True:
time.sleep(IDLE_INTERVAL)
if _busy.is_set():
vlog("idle watcher: skip tick (busy with a task)")
continue
ans = _look_via_server(IDLE_QUESTION, image=True, image_b64=_latest_jpeg_b64())
...
_run_presence_agent(ans)

Routing one observation to two tools

_run_presence_agent(observation) builds a minimal Strands Agent for that single observation, registers both presence tools, and discards the agent afterward (del + gc.collect()). Its system prompt routes by species:
  • one or more people present → call report_human_presence
  • one or more cats present → call report_cat_presence
  • both present → call both tools
  • neither present → do nothing, emit no prose

The species-specific tools

Each tool logs a line, and only on a real detection does it record and publish — so a fully local run (no S3/MQTT) just logs:
1
2
3
4
5
if people >= 1: # only upload when something is actually detected
video_url = record_clip_and_upload()
publish_state("presence", presence_kind="human", presence_count=people,
presence_description=description,
**({"video_url": video_url} if video_url else {}))
report_cat_presence is identical with presence_kind="cat". Both the clip upload and the state publish are no-ops when S3/MQTT (or the camera owner) are unavailable.

Configuration

Env varDefaultPurpose
IDLE_WATCH1Set to 0/false/no/off to disable the watcher
IDLE_INTERVAL10Seconds between presence checks
IDLE_QUESTIONcounts-per-species promptQuestion sent to Cosmos Reason 2

Technical Challenges & Solutions

Challenge 1: Idle awareness that stays free

Problem: Continuously running vision to watch the room could peg the GPU and, on a cloud VLM, cost money around the clock.
Solution: The watcher ticks on a slow cadence (default every 10s), reuses the already-captured shared frame, and runs local Cosmos + Nemotron — so each glance is $0 and the GPU is touched only briefly, on a timer.

Challenge 2: Presence checks colliding with real tasks

Problem: If a user wakes the robot or an MQTT look arrives mid-tick, two vision workloads would contend for the GPU and camera.
Solution: Every tick first checks the shared _busy event and skips if a task is running. The idle watcher always yields to a real interaction.

Challenge 3: Humans and cats need different reactions

Problem: A single generic report_presence can't let a person and a cat trigger distinct behaviours, and both can be present at once.
Solution: Split into report_human_presence and report_cat_presence, each independently fired by the routing agent. One glance can call both, either, or neither — giving a clean hook for per-species behaviour later, while today each records a clip and publishes a tagged presence event only when its count is real.

Getting Started

GitHub Repository: https://github.com/chiwaichan/chiwaichan-reachy-mini-strands-agent-jetson-thor 

Run it (and tune the cadence)

1
2
3
./reachy_assistant.sh # watcher on by default, every 10s
IDLE_INTERVAL=30 ./reachy_assistant.sh # slower cadence
IDLE_WATCH=0 ./reachy_assistant.sh # disable the watcher entirely
While the robot rests, watch the log: it prints what Cosmos sees each tick and which presence tools fire.

What's Next

In Part 10 - Face Tracking with a Single Camera Owner , the head starts following your face in real time — and I dig into the single camera-owner thread that lets head tracking, the idle watcher, and Cosmos vision all share one camera without V4L2 contention.

Summary

This post covered idle presence awareness:
  • Idle glances, fully local — every IDLE_INTERVAL seconds the robot asks Cosmos who it sees, at $0 per tick
  • Yields to real work — ticks are skipped while _busy is set, so presence never competes with a live task or the GPU
  • Species split — report_human_presence and report_cat_presence route independently; both, either, or neither can fire per glance
  • Disposable router — _run_presence_agent builds a minimal agent per observation and dels it, so nothing persists between ticks
  • Real detections only — clips (S3) and presence state (IoT Core) are recorded only when a count is >= 1, and no-op when the cloud paths are offMore in this series
  • Previous - Part 8: 80 Emotion Moves by Sentiment 
  • Next - Part 10: Real-Time Face Tracking 
# robotics# ai-agents# computer-vision# nvidia# generative-ai

Series: Reachy Mini x Strands Agent on Jetson Thor (14 articles)

  1. 1
    Reachy Mini x Strands Agent on Jetson Thor - Part 1: Driving the Reachy Mini Lite with Strands Agent
  2. …
  3. 9
    Reachy Mini x Strands Agent on Jetson Thor - Part 9: Idle Presence — Noticing Who's Around (Humans vs Cats) This article
  4. 10
    Reachy Mini x Strands Agent on Jetson Thor - Part 10: Real-Time Face Tracking with a Single Camera Owner
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article