AWS Builder Center
Full Stack Challenge: Silent Signal

Full Stack Challenge: Silent Signal

AI agents fail inside HTTP 200 responses. Silent Signal classifies the payload, fingerprints root causes, and flags retry loops — on Amplify Gen 2.

Vision and What It Does

Silent Signal is a full-stack observability tool for AI agents that catches the failures HTTP status codes miss.
Here is the problem in one sentence: when an AI agent calls a tool and that tool fails, the HTTP response is still  200 OK . The failure lives inside the JSON payload — an  isError: true  flag buried in a content block, an empty result array, a malformed response the agent quietly treats as fact. Every dashboard you own says the service is healthy. Your agent is looping on a broken tool and nobody is listening.
I got the word SIGNAL from the PartyRock generator, and this is the interpretation I went with: a signal that exists but that no one is monitoring. The failure is being broadcast. The listener is the missing piece.
Silent Signal is that listener. You paste or POST a tool-call response, and it tells you what actually happened:
  • TOOL_ERROR —  isError: true  appears anywhere in the result, including nested inside content blocks
  • EMPTY_RESULT — the content array is empty, or every text block is blank
  • MALFORMED — the payload will not parse, or the content field is missing entirely
  • SCHEMA_DRIFT — this tool previously returned a different set of top-level keys
  • SUCCESS — none of the above
It groups failures by fingerprint rather than by raw error string, so a hundred instances of the same root cause collapse into one row instead of a hundred unreadable ones. And it flags thrash: the same tool called with near-identical arguments three or more times in a sixty-second window, which is the signature of a model stuck in a retry loop it cannot perceive.
Who it is for: anyone running agents in production — MCP server authors, teams building on LangGraph or Strands, and platform engineers who have been handed an agent and told to keep it up.



Full Stack Breakdown

The challenge asked how the build maps to the five stages of the series — pitch, prototype, MVP, UX, launch. I want to be straight about this: I came to the challenge at the finale rather than following along across the six weeks. But the five-stage arc turned out to be an accurate description of how the 24 hours actually went, and forcing the build through those gates is what kept it shippable.
Pitch. The idea comment on the challenge post. One line — your agent said 200 OK, it was lying — plus three features and a stack. Writing the pitch before writing any code is what stopped this from sprawling into a general-purpose observability platform I had no chance of finishing.
Prototype. Phase 1 was the entire UI running on hardcoded mock data. Three routes, dark mode, a table, a verdict card. No AWS, no database, no classifier. It ran locally in under an hour and it looked exactly like the finished product. That mock data later became the shape of the real schema, which meant zero rework at the data layer.
MVP. The classifier. This is the part that had to be genuinely correct, so I built it as pure functions in  /lib  with a fixture-table test suite before wiring it to any storage. The critical test case: a payload where  isError: true  is nested inside  content[0]  rather than sitting at the top level. That is the real-world shape, and it is the single case the entire product rests on. If that returns SUCCESS, nothing else matters.
UX. Dark mode by default with persistence, and — less glamorous but more important — the empty-state and seeded-data work. A demo dashboard with nothing in it reads as broken. I wrote a seed script producing thirty realistic calls including both thrash windows and drift cases specifically so a first-time visitor lands on something meaningful.
Launch. Git-connected CI/CD on Amplify Hosting, with the backend deploying per-branch through  ampx pipeline-deploy  in the buildspec. Push to  master , get a deploy. Confirmed working twice before I called it done.



How I Built It

I ran the build in six phases with a hard rule: stop at the end of each one, verify, commit. If anything took more than twenty minutes, build the simpler version instead. With a 24-hour clock that constraint did more for the outcome than any technical decision.
Phase 1 — Scaffold. Next.js 15 + TypeScript + Tailwind v4, class-based dark mode persisted to  localStorage , three routes on mock data.
Phase 2 — Classifier. Pure functions, unit tested. The one design decision worth calling out: schema drift is the only classification that needs prior state, so instead of letting the classifier reach into storage, I pass  previousKeys: string[] | null  in as an argument. The function stays stateless and testable; the route handler does the lookup. That decision made Phase 3 trivial.
I also had to define what "drift" means. I settled on removed or type-changed keys count as drift; added keys do not. New optional fields are normal API evolution, and flagging them would fill the dashboard with false positives — which defeats the purpose of a tool whose whole value is separating real failures from noise.
Phase 3 — Backend. Amplify Gen 2  defineData , which provisions AppSync and DynamoDB from a TypeScript schema definition. Route handlers wired to read and write.
Phase 4 — Seed. Thirty sample calls covering every outcome plus both thrash windows.
Phase 5 — Bedrock. Plain-English root-cause summaries via Nova Pro, behind a  USE_BEDROCK  environment flag that defaults off. The deterministic classifier is the primary path and the application is fully functional with zero Bedrock access. Every model call is wrapped in try/catch with a template fallback. This was deliberate: I have had account-level Bedrock access blocked before, and I was not going to let a model availability issue take down a demo on a 24-hour deadline.
Phase 6 — Deploy. Detailed below, because this is where the real problems were.



Challenges

Challenge
Solution
create-next-app@latest pulled Next 16, which Amplify's Gen 2 tooling does not yet track
Pinned to Next 15 in a follow-up commit. Catching this locally saved an hour of confusing build failures later.
Manual CLI deploy does not work for Next.js SSR apps on Amplify
Only Git-connected CI/CD supports SSR. Set up the GitHub repo and connected Amplify Hosting properly rather than fighting it.
Amplify build container's default Node version broke ampx with Cannot find module amplify/data/resource
Explicit nvm use 20 in amplify.yml. Same esbuild incompatibility I had already hit locally.
npm ci failed on lockfile fidelity in the build container
Switched the buildspec to npm install.
A public demo behind a login wall looks broken to a reviewer
Configured defineData for public access so /dashboard renders real data with no sign-in.



AWS Services / Architecture

AWS Service
Purpose
AWS Amplify Hosting
Next.js SSR hosting on the WEB_COMPUTE platform, plus Git-connected CI/CD
AWS Amplify Gen 2 (defineData)
Declares the backend in TypeScript; provisions AppSync and DynamoDB
Amazon DynamoDB
Stores tool-call records, fingerprints, and thrash windows
AWS AppSync
GraphQL API layer generated from the data schema
Amazon Bedrock (Nova Pro)
Optional plain-English root-cause summaries, behind a default-off flag
AWS IAM
Dedicated silent-signal-amplify-service-role for the build pipeline
Amazon CloudFront
CDN distribution, provisioned by Amplify Hosting
Browser (Next.js 15, SSR, dark mode)
|
v
Amplify Hosting (WEB_COMPUTE) --- CloudFront
|
v
Next.js Route Handlers /api/tool-calls
|
+--> classifier (pure fns) --> outcome + fingerprint + argsHash
|
+--> AppSync (GraphQL) --> DynamoDB
|
+--> Bedrock Nova Pro [USE_BEDROCK, default OFF]
|
+--> on failure: template summary fallback

GitHub push to master --> Amplify CI/CD --> ampx pipeline-deploy
(frontend + backend)

The write path: a payload arrives at the route handler, the classifier assigns an outcome and computes a fingerprint (SHA-256 of tool name + outcome + normalized error shape, truncated to 16 hex characters), the record persists to DynamoDB via AppSync, and the dashboard aggregates by fingerprint on read.
Normalization is what makes the fingerprint useful — UUIDs, timestamps, and raw numbers get stripped before hashing, so the same root cause across a hundred calls produces one fingerprint instead of a hundred.



What I Learned

Amplify Gen 2 is genuinely different from Gen 1. Defining the backend as a TypeScript schema and having AppSync plus DynamoDB provisioned from it removes a lot of configuration surface. The trade-off is that the sandbox backend and the pipeline-deployed branch backend are separate environments — which surprised me when the live dashboard came up empty and I had to run the seed script against the deployed backend directly.
SSR changes the deployment story. I assumed manual CLI deploy would work and was wrong; AWS's own docs confirm SSR apps require Git-connected CI/CD. Better to learn that at hour eighteen than hour twenty-three.
Version pinning is a deadline skill. Two of my five real problems were Node or framework version mismatches between my machine and the build container. On a 24-hour clock, pinning aggressively and explicitly is not conservatism, it is speed.
Design the fallback before the AI feature. Building the deterministic classifier first and treating the model as an optional enhancement meant Bedrock availability was never on the critical path. The demo works whether or not the flag is on.
Deciding what is not a failure is the hard part. Getting the classifier to catch nested  isError  was straightforward. Deciding that added schema keys should not count as drift took longer and mattered more. A detection tool that cries wolf gets ignored, which makes it worse than no tool at all.



Link to Repository

The repo includes the full source, the classifier test fixtures, the Amplify Gen 2 backend definition, and the seed script. Clone it, run  npx ampx sandbox  for your own backend, and  npm run dev  to bring it up locally.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article