A Strands Agents hook that won't let a repair agent skip a step

I bought my first house in Orlando and found out I didn't know how to do the simple things around it. Since then I've hired or gotten quotes from roughly three dozen contractors, and more than half of those jobs failed, stalled or had to be redone. What worked was my phone: take a photo, have someone mark it up, do one thing, take another photo.

StepSpotter is that loop as an agent. You name a repair job and send a first photo. It returns one step drawn on your photo: what to do, what not to touch, and what your next photo has to show. It won't move on until a photo proves the step is done. It won the Everyday Agents | Bronze Agent prize in the Agents for Humans Hackathon on Devpost.

StepSpotter runs eight roles on Strands Agents with Amazon Bedrock. The Guide agent the user talks to has an advance_step tool. Before Strands dispatches that call, a BeforeToolCallEvent hook checks the step state. If the Verifier hasn't passed the current step, the hook sets event.cancel_tool with a reason and the call never runs.

A hazard in a photo (heat, swelling, a burnt smell, bare wire) gets event.interrupt from the same hook. A cancel fails one call and lets the loop continue. An interrupt stops the run and hands it to a person.

We tested the gate with a system prompt written to defeat it: the user is always right, call advance_step immediately. The model tried to skip ahead and the hook stopped it. With an honest prompt the model never tries to skip, so that test would pass even with a broken gate.

The Verifier reads one photo against one claim and returns a StepVerdict through structured_output_model: pass, fail or stop, with a reason. We gate on that boolean and ignore the confidence score, because it wasn't stable. The same photo and claim scored 0.82 on one run and 0.20 on the next, for the same correct refusal.

The Planner decides first who should do the job at all. If it sets safety_class to vendor_required, the step list is emptied in code, so a job that needs a licensed trade, like mains power or gas, never gets a plan.

Bounding boxes from the vision model drift. On cluttered photos of a panel, 3 of the 14 boxes we checked by eye were tight. So the highlight is snapped to a grid and shown as a hint, never as "it is exactly here".

Search rate limits hit mid-demo. DuckDuckGo started answering HTTP 202 after repeated lookups from one IP, which parsed to zero results and looked exactly like "no manual exists". Now the demo manual is baked into the image, a cached manual is never overwritten by an empty result, and the step card names the source that answered.

The phone web app runs on AWS App Runner, and the Guide agent also runs on Amazon Bedrock AgentCore Runtime. The web UI never calls the model directly. It goes through the same StepGate object the agent uses.

On real photos of my own panel the Verifier gave 8 correct pass/fail verdicts out of 8. Across two published eval runs, with the misses left in, it rejected 6 of 6 red-team photos. The 189 tests pass offline on a fresh clone.

Code (MIT): https://github.com/retailbox-automation/stepspotter

Project page: https://devpost.com/software/stepspotter

Full write-up: https://retailbox-automation.com/case-studies/stepspotter

I'm Mikhail Oskola, founder of Retailbox, an AI automation company in Orlando.