AWS Builder Center

Builder Center

Building a Multimodal Video RAG System with Amazon Nova on AWS Bedrock

Building a Multimodal Video RAG System with Amazon Nova on AWS Bedrock

A walkthrough of how we built a production-ready video intelligence system using Amazon Nova Lite, Nova Pro, Nova Sonic, and Titan Multimodal Embeddings — enabling natural-language clip search, RAG-based Q&A, and real-time voice interaction over any uploaded video.

The Problem with Video Content

Video is one of the richest formats for storing knowledge, yet it remains almost entirely unsearchable. You cannot grep a video. You cannot run a SQL query over a lecture recording or a product demo. The only way to find something is to watch it — which does not scale.
This is the problem we set out to solve for the Amazon Nova Hackathon.


What We Built

Video RAG is a multimodal search and question-answering system that lets users interact with any video using natural language. After uploading and processing a video, a user can:
  • Ask "What did the speaker say about pricing?" and receive a grounded, cited answer drawn directly from the transcript
  • Say "Show me the moment the logo appears on screen" and receive a trimmed clip of exactly that scene
  • Upload a reference image and find the most visually similar moment in the video
  • Ask "Give me the first 10 seconds" and receive that clip immediately
  • Do all of the above through a real-time voice interface powered by Amazon Nova Sonic


Architecture

The system runs as two separate services:
MCP Tool Server (FastMCP, port 9090) — owns all video processing and retrieval logic, exposed as typed MCP tools. This keeps the heavy media pipeline isolated from the API layer.
FastAPI Server (port 8080) — hosts the NovaAgent, which orchestrates a four-stage pipeline on every user message:
Plain text
User message
    │
    ▼
1. ROUTING  (Nova Lite — fast, cheap binary classification)
   └── Does this message need a video tool?
    │
    ▼
2. TOOL SELECTION  (Nova Pro — structured JSON output)
   └── Which of the four tools? With what query?
    │
    ▼
3. TOOL EXECUTION  (MCP call)
   └── Returns a clip path or a RAG answer
    │
    ▼
4. RESPONSE GENERATION  (Nova Pro)
   └── Wraps the result in natural language
Using Nova Lite for routing and Nova Pro for reasoning is a deliberate cost and latency decision. The routing call is a binary classification and does not need Pro-level capability.


Why a Unified Embedding Space Matters

The reference implementations for video RAG typically use three separate models: CLIP for image/frame embeddings, a text embedding model for transcripts, and another for captions. Searching across all three means running three separate queries and merging results from incompatible vector spaces.
Amazon Titan Multimodal Embeddings V1 eliminates this complexity entirely. Both text queries and images are embedded into the same vector space, which means:
  • A text query can be compared directly against frame embeddings
  • An image query can be compared directly against caption embeddings
  • The entire retrieval system is a single index with a single similarity function
This is the architectural decision we are most satisfied with. It simplifies the codebase, reduces API call overhead, and produces better cross-modal retrieval results.


Batched Scene Captioning

A naive implementation calls Nova Pro once per video frame to generate a scene description. For a 15-frame sample, that is 15 separate API calls.
We batch consecutive frames into groups of 5 and send them to Nova Pro in a single multi-image API call. Nova Pro receives temporal context (it can see what changed between frames) and produces a more coherent scene description. API call count drops from 15 to 3. Latency and cost drop proportionally.


The Ingestion Pipeline

When a video is processed, the following steps run sequentially:
  1. Frame extraction via PyAV — uniform sampling across the full duration
  1. Audio chunking — overlapping segments to avoid cutting sentences at boundaries
  1. Transcription — AWS Transcribe (primary) with OpenAI Whisper as fallback
  1. Scene captioning — batched Nova Pro vision calls (5 frames per call)
  1. Embedding — all frames, transcripts, and captions embedded via Titan Multimodal Embeddings into a single Pixeltable vector store
  1. Registry — index metadata persisted to disk so the server survives restarts without re-processing


Structured Outputs at Every Step

Every LLM call in the pipeline returns a Pydantic-validated structured object, not free-form text. We use Bedrock's  toolConfig  with  toolChoice: forced  to guarantee the model returns a valid JSON object matching our schema.
This means routing returns a  RoutingSchema(tool_use: bool) , tool selection returns a  ToolSelectionSchema(tool_name, user_query) , and clip responses return a  VideoClipResponseSchema(message, clip_path) . There is no string parsing anywhere in the agent pipeline.


Community Impact and Real-World Applications

The inability to search video content is a barrier that affects a wide range of communities:
Education — Recorded lectures, coding tutorials, and training videos are watched linearly or skipped entirely because learners cannot find the specific explanation they need. Video RAG turns hours of recorded content into a queryable knowledge base. A student can ask "explain the part about gradient descent" and receive the exact 45-second clip from the relevant lecture.
Accessibility — Users who are deaf or hard of hearing currently rely on captions, which are not always available or accurate. A speech-plus- caption RAG system surfaces the full content of a video in a form that does not depend on audio at all.
Legal and Compliance — Organizations that record meetings, depositions, or training sessions spend significant time reviewing footage manually. Natural-language search over hours of recorded content reduces review time from days to minutes.
Media and Content Production — Editors and researchers working with large video archives (news footage, documentary interviews, product demos) can locate specific moments by describing them rather than scrubbing through timelines.
Enterprise Knowledge Management — Product demos, onboarding videos, and internal training recordings represent significant institutional knowledge that is currently locked away in unindexed files. Video RAG makes that knowledge accessible to every employee through a simple chat interface.
Our plan for encouraging adoption is to release the project as open source with a one-command setup script, a pre-built Docker image, and detailed documentation for connecting it to existing video libraries via S3 or local file paths.


Try It Yourself

The full source code, setup instructions, and architecture documentation are available in our project repository. The system requires only AWS credentials with Bedrock access enabled — no third-party vector database, no external embedding service, and no GPU.
Built with: Amazon Nova Lite · Nova Pro · Nova Sonic · Titan Multimodal Embeddings · Amazon Bedrock · FastAPI · FastMCP · Pixeltable · AWS Transcribe
# amazon-nova# agentic-ai# amazon-bedrock# retreival-augmented-generation# generative-ai
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article