AWS Builder Center

Builder Center

ScreenSense Voice: AI Browser Agent with Amazon Nova

ScreenSense Voice: AI Browser Agent with Amazon Nova

A voice-controlled AI browser agent built with Amazon Nova and Amazon Transcribe that turns natural language commands into autonomous web actions.

Website : Link 

The Problem: Browsers Weren't Built for Everyone

We interact with browsers the same way we did 20 years ago — point, click, scroll, type, repeat. For many people, this works fine. But for millions of others, it doesn't.
People with motor disabilities struggle with precise mouse movements. Elderly users find multi-step web workflows overwhelming. Busy professionals waste hours on repetitive browser tasks. Anyone navigating complex checkout flows or long forms knows the frustration.
What if you could simply tell your browser what you want, and it would do it for you?
That's the idea behind ScreenSense Voice — a hands-free AI browser agent powered by Amazon Nova.

What is ScreenSense Voice?

ScreenSense Voice is a Chrome extension that turns voice commands into browser actions.
You hold a key, speak a command, and the AI agent:
• Understands what is visible on the screen
• Analyzes the web page structure
• Executes actions automatically
For example:
User:
“Add the cheapest USB-C cable to my cart on Amazon.”
ScreenSense Voice:
  1. Opens Amazon
  2. Searches for USB-C cable
  3. Identifies the lowest priced item
  4. Opens the product page
  5. Clicks Add to Cart
The agent then responds:
"Added to cart. Anything else?"
No typing. No clicking. Just voice.

How Amazon Nova Powers the Agent

ScreenSense Voice uses Amazon Nova via AWS Bedrock as the reasoning engine that decides what action to perform on the web page.

Step 1 — Capture Context

When the user gives a command, the extension captures:
• A screenshot of the page
• A structured snapshot of the DOM (buttons, links, inputs)
This provides both visual context and structured page information.

Step 2 — Reason with Amazon Nova

Both inputs are sent to Amazon Nova Lite through the AWS Bedrock Converse API.
Nova analyzes the page and generates a structured action plan.
Example response:
1
2
3
4
5
6
7
8
9
{
"type": "steps",
"reasoning": "The search bar is at the top of the page.",
"actions": [{
"action": "type",
"selector": "#twotabsearchtextbox",
"value": "USB-C cable"
}]
}

Step 3 — Execute the Action

The browser extension executes the action directly on the page.
Then it repeats the process:
Observe → Reason → Act
This loop continues until the task is completed.

Voice Processing with Amazon Transcribe

For real-time voice interaction we use Amazon Transcribe Streaming.
When the user speaks:
  1. Audio is captured in the browser
  2. It is streamed to Transcribe
  3. Text is returned in near real-time
Low latency is critical because users expect instant feedback after speaking.

Architecture Overview

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
User Voice
↓
Chrome Extension
↓
Screenshot + DOM Snapshot
↓
FastAPI Backend
↓
Amazon Transcribe (Speech → Text)
↓
Amazon Nova Lite (Reasoning)
↓
JSON Action Plan
↓
Browser executes action
↓
Observe → Reason → Act Loop

How ScreenSense Voice Helps the Community

Accessibility

Many users cannot easily operate a mouse or keyboard. ScreenSense Voice allows them to interact with any website entirely through voice.

Elderly Users

Older adults often struggle with complicated web interfaces. Natural language commands simplify everything.

Productivity

Professionals perform repetitive browser tasks daily. Automating these workflows saves significant time.

Language Support

Users do not need perfect English commands. Amazon Transcribe and Nova understand natural language variations.

Technical Challenges

Reliable Web Selectors

Websites generate dynamic element IDs. We built a selector prioritization strategy using:
• data attributes
• aria labels
• semantic URLs
• structural CSS patterns
This ensures Nova chooses selectors that actually work.

Observation Loop

Complex tasks require multiple actions.
After every step we capture a new screenshot and DOM snapshot so the agent always has the latest context.

Payload Optimization

Screenshots are compressed to reduce network cost and latency, ensuring fast agent responses.

Real-World Applications

ScreenSense Voice could be used in several domains:
Enterprise automation
Automate browser workflows like expense reporting and CRM updates.
Assistive technology
Enable patients or disabled users to interact with the web independently.
Education
Students can navigate learning platforms and research topics using voice commands.
E-commerce assistance
Users can search and purchase products hands-free.

Future Improvements

Next steps for the project include:
• Multi-tab workflows
• Advanced UI automation with Amazon Nova Act
• Real-time conversational feedback
• Personalized task memory

Conclusion

ScreenSense Voice demonstrates how Amazon Nova can power intelligent agents that interact with the web the same way humans do — by observing, reasoning, and acting.
By combining voice interaction, multimodal reasoning, and autonomous browser control, we move closer to a future where interacting with computers becomes as simple as having a conversation.

Try It Yourself

ScreenSense Voice is open source and available on GitHub: Link 
Clone the repository, configure your AWS credentials, and run the extension locally to start interacting with your browser using voice commands.
The repository includes setup instructions, architecture documentation, and the Chrome extension source code.
Built for the Amazon Nova AI Hackathon 2026.
# ai# amazon-nova# machine-learning# agents# agentic-ai
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article