AWS Builder Center

Builder Center

PaperPal: A Voice Agent That Points at the Confusing Line and Explains It

PaperPal is a real-time voice agent that reads a confusing letter, bill or notice through your phone camera, explains what it wants from you in plain language, highlights the exact line it is talking about, and sets the deadline reminder. Built with Amazon Bedrock, Strands Agents, Textract and Nova Sonic.

gayathri

data science student

🔗 Explore the Project

You can explore the source code and implementation details of PaperPal in the GitHub repository below.
GitHub Repository: https://github.com/Gayathri-S-E/Paperpal

The problem: the paper nobody explains

Most people can handle an app. What trips them up is the paper that arrives with it: an electricity bill with a surprise charge, a bank notice with a deadline, a rental agreement, a hospital invoice. The language is dense, the deadline is buried, and the cost of misreading it is real.
PaperPal is for exactly those people: first-time renters, parents who don't read official English comfortably, small shop owners, anyone who has stared at a notice and thought "what do they actually want from me?"
You hold the document up to your phone and talk to it. PaperPal tells you what it is, what it wants, how much, by when, and what happens if you ignore it. You can interrupt at any moment.

What it does

  • Reads the document (photo in, structured understanding out)
  • Explains it by voice in plain language, and handles interruptions
  • Acts: sets a deadline reminder and drafts a reply letter for you
  • Knows its limits: for legal, medical or tax-liability questions it says so and points you to a professional instead of guessing

How I built it

  1. Capture: the photo goes to Amazon S3 (KMS-encrypted).
  2. Read: Amazon Textract AnalyzeDocument returns text, tables and the bounding box of every line.
  3. Understand: a Strands Agents agent on Amazon Bedrock classifies the document and extracts five things: what it is, what they want, the amount, the deadline, and the consequence of ignoring it.
  4. Talk: Amazon Nova Sonic handles speech-to-speech over bidirectional streaming, so the user can barge in mid-sentence.
  5. Act: the agent has three tools: highlight, set_reminder (EventBridge Scheduler + SNS) and draft_reply (a PDF saved to S3).
  6. Run: Bedrock AgentCore Runtime hosts the agent and AgentCore Memory keeps session context. The frontend is a web app on AWS Amplify.
The tool that matters most is also the smallest:
1
2
3
4
5
6
7
from strands import Agent, tool

@tool
def highlight(block_id: str) -> str:
"""Highlight one line of the document on the user's screen."""
send_to_ui({"type": "highlight", "block_id": block_id}) # UI draws the box
return "highlighted"
Textract already gives every line an ID and a bounding box, so the agent only has to say which line.

Privacy by design

These documents are sensitive, so I treated that as a feature, not an afterthought:
  • Images are encrypted at rest and auto-deleted after {{N hours}}.
  • PII is redacted before anything reaches logs.
  • Bedrock Guardrails keep the agent out of legal and medical advice.
  • Write actions (reminders, drafts) happen only after the user says yes.

The delightful detail: it points at what it's talking about

As PaperPal explains, a box appears on your photo around the exact line it is describing. Interrupt with "wait, what's this line?" and it jumps to that region and explains it.
Why I did this: a voice that says "your payment is due on the 14th" asks you to take its word for it. A voice that draws a box around "Due Date: 14 Oct" lets you check, and trust builds fast when you can see the evidence yourself. It makes the agent feel like a patient friend pointing at the page.
How I know it worked: I gave {{5}} non-technical people (family and neighbours) real documents they found confusing, and asked three questions before and after: What do I owe? By when? What do I do next?
  • Before PaperPal: {{X of 15}} answers correct
  • After PaperPal: {{Y of 15}} answers correct
  • Quote from a tester: "{{real quote}}"

What I learned

  • Barge-in changes everything. Once people could interrupt, they stopped treating it like a tool and started talking to it.
  • Highlighting beats explaining. {{your real observation from testing}}
  • Guardrails are part of the experience. A clear "this needs a professional" is more helpful than a confident wrong answer.

What's next

{{1-2 honest next steps, e.g. more languages, more document types}}
Built for the AWS Builder Center agent challenge.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article