AWS Builder Center
Expressive Captions: Built captions that shout, and then we argued with them out loud

Expressive Captions: Built captions that shout, and then we argued with them out loud

Four days building a Hinglish caption editor you can talk to — the bake-off numbers, the voice bugs that ate half a sentence, and what's still broken.

We Built an AI Video Editor You Can Talk To

Four days, three people, and one Hinglish reel that broke almost everything we threw at it.

When we signed up for the AWS First Commit Hackathon (Ship It track), our pitch fit in one line: make captions that understand how you speak, not just what you say.
If you make short-form content in India, you probably switch between Hindi and English without thinking about it. You also lean on certain words, drag them out, pause right before the punchline, and change tone halfway through a sentence. Almost every auto-caption tool we tried ignores all of that and gives you the same flat text no matter how you said it. We wanted captions that looked the way the performance sounded.
Then we got a bit greedy. If the editor already knows what you said and how you said it, why should you have to click through five panels to change anything? Why not just tell it?
That's what we spent four days building: Expressive Captions, a short-form video editor that writes Hinglish captions, reads the delivery from the audio, and lets you edit the video by typing or talking to it. Here's how it went, including the parts that didn't go well.

Day one: the transcript came back empty

Our first move was AWS Transcribe with en-IN. Hinglish online is mostly written in Roman script, so an English model seemed like a fair bet.
It wasn't. On our real test reel, about 23 seconds of fast talking, it returned nothing. Not a bad transcript, just an empty one.
Instead of guessing our way through speech engines, we built a small benchmark: four clips (normal, excited, angry, and the real reel), handwritten ground truth, and content word error rate after normalising stretched letters and spelling variants, because Hinglish has no fixed spelling for anything.
Engine
Normal
Excited
Angry
Real reel
Mean
Sarvam translit
0.286
0.308
0.116
0.161
0.218
Transcribe hi-IN + Bedrock romanisation
0.306
0.385
0.116
0.151
0.239
Transcribe en-IN
0.510
0.077
0.326
1.000
0.478
Sarvam gave us the best Roman Hinglish. The catch was that it doesn't return word-level timestamps, and captions without word timings are basically karaoke without the bouncing ball.
So we ended up with a slightly odd setup: two speech engines feeding one caption track. Sarvam gives us the words, Transcribe hi-IN gives us the timings, and we line the two up.
Our first attempt at lining them up was to hand both transcripts to an LLM and ask it to reconcile them. On a 94-word reel it quietly dropped one word, and every timestamp after that slid out of place. That was the end of that experiment. We swapped in Needleman-Wunsch alignment over phonetic keys, which is old, boring, and deterministic, and that was exactly what we needed. Not every problem needs an LLM, and this one definitely didn't.

Reading how something was said

Getting the words right was only half the job. Take "This is actually insane." You can whisper "actually" and drag out "insane." You can shout the whole thing. You can pause before "insane" like you're about to drop news. Same transcript every time, completely different delivery.
So for every word we measure loudness, pitch, duration, and the pauses around it, and turn those into caption properties: emphasis, stretch, and tone.
We did try just asking a model which words sounded important. It found something to highlight in every sentence, because every sentence technically has a most important word, and the captions ended up looking like everything was a big deal. What worked better was scoring each word against the speaker's own baseline. Across our test clips, emphasis landed on roughly 13 to 15% of words, instead of every other word turning into a visual event.
Stretch detection is the weakest part, and we know it. We only had seven genuinely stretched words across all four clips, so the results are noisy. We shipped it anyway and wrote down the next step: get more annotated clips before making the model any more complicated.

Day two: making the editor listen

With captions in decent shape, we moved on to the part we were most excited about. The usual editing loop is click, open a panel, find the caption, change the style, repeat. We wanted to say "make that line angry," "switch to the Dhamaka preset," or "fewer words per line" and have it just happen.
The mic connects to a LiveKit room over WebRTC. A LiveKit Agents worker does speech-to-text and sends the transcript back to the editor, and anything non-trivial goes to an agent built on Amazon Bedrock's Converse API with tool use. The agent has tools for caption words, styles, presets, media layers, timeline operations, and vision.
One thing we decided early: the agent doesn't get to handle everything. "Play," "pause," "undo," and "redo" don't need a language model, and sending "pause" to Bedrock just adds latency and one more thing that can fail. Those get matched in the browser with strict patterns and run immediately. Everything else goes to the agent.

Then someone actually talked to it

In the first real voice test, someone said "hey, increase the size of the white font…" then paused, then said "…from 10 second marker to 12 second."
The speech recogniser treated the pause as the end of a sentence, so we got two separate final transcripts, and the second one replaced the first. The editor heard "from 10 second marker to 12 second" and had no idea what it was supposed to do there.
The fix was to stop treating every recogniser final as a complete turn. We added a small turn state machine: while the agent is still thinking, a new utterance merges into the current turn, and once patches are actually being written, new speech waits.
Digging into that turned up a whole family of related bugs. The input could become available while the previous turn was still running. Cancelling could undo the wrong operation. History could say a turn was cancelled when it wasn't. Two utterances could race each other and leave a history entry stuck on "pending" forever. None of these show up if you only demo the happy path. All of them showed up as soon as someone talked to the editor like a normal person.

The Stop button that didn't stop

We also had a Start/Stop voice button. The button worked. The voice session, sometimes, didn't.
Starting voice is a chain of async steps: mic permission, token request, WebRTC connection, mic activation. If you hit Stop halfway through, there might not be a room to close yet, so the start sequence kept going in the background and eventually opened a live connection anyway. You'd press Start, press Stop, and be left with a session you couldn't see.
We only called this fixed after reproducing the original bug, applying the fix, and checking active connections on the LiveKit server itself instead of trusting the UI. A button saying "stopped" isn't proof that anything stopped, and we had to relearn that more than once.

Day three: "stop" turned out to be a hard word

Remember those local playback commands? They had an edge case.
Say "stop making things red." Our parser correctly understood that this isn't a playback command. But the transport handling ran before the barge-in and merging logic, so what actually happened was: pause the video, then send "making things red" to the agent. Same bug as the split sentence, sneaking in through a different door.
We fixed it by making bare transport commands provisional. A standalone "stop" or "play" still runs immediately, but we hold onto the command and its undo for a moment. If more words follow, we roll back the playback action and send the whole sentence to the agent. Real playback commands stay instant, and normal speech doesn't get chopped in half.

Giving the editor eyes

Next question: what if the edit depends on what's actually in the frame? "Put the captions where my hand is." "Put an angry emoji on my face."
We added vision tools using Amazon Rekognition, so the agent can inspect frames, detect labels and faces, and use that while building an edit. An early version resolved "the face" to the label "Person," which is not the same thing at all. DetectFaces gives you face bounding boxes; DetectLabels is a different API. We switched to the right one.
Caption placement is still approximate, though. The backend doesn't have real font metrics, so it has to estimate how wide the rendered text will be. It's documented as a known limitation.

One project file for everything

As features piled up we needed a single source of truth, and that became a shared Project JSON: captions, word timings, tone, styles, presets, media layers, transforms, AI edits, all of it. The editor uses a TypeScript schema, and the backend keeps a matching Python version.
The agent never touches pixels. It produces structured patches against the project, and every patch is validated against the schema before it's applied. The browser preview and the final export both render from the same project through the same Remotion composition, so preview really does equal export. The AI isn't editing some hidden state that the renderer later interprets differently; it edits the project, and the project gets rendered. That made debugging a lot less painful.

The stack

  • Frontend: React, TypeScript, Vite, Tailwind, shadcn/ui
  • Backend: Python 3.12, FastAPI
  • Video: Remotion, FFmpeg
  • Speech: Sarvam, AWS Transcribe
  • AI: Amazon Bedrock (Converse API with tool use)
  • Vision: Amazon Rekognition
  • Real-time voice: LiveKit Agents over WebRTC
  • Storage: Amazon S3, DynamoDB
  • Infra: Terraform
Large media goes straight to S3 through signed URLs. The renderer and voice worker sit in the private part of the network, and the browser only talks to the public editor and API.

And then App Runner wasn't an option

We'd planned our deployment around AWS App Runner. Halfway through the hackathon, we found out App Runner was closed to new customers, which is not something you want to learn with two days left. We moved the rendering path into a container we controlled, and as a bonus that also fixed an export problem we'd been hitting on Linux. It was the least glamorous part of the project and probably the most realistic one.

We kept an engineering diary

We used Claude as our AI coding assistant during the build, and for every major piece of work we kept an audit doc under  .claude/audits/ . Each one had to spell out what was implemented, what was actually verified, what was mocked, and what we hadn't checked at all.
By the end we had 72 of them. For the first day or two that honestly felt like overkill. By day four, those docs were the reason three people could work on different parts of the system without constantly re-asking why something was built a certain way. They also forced us to keep "we built it" and "we tested it" as two separate statements.

What's still broken

We don't want to pretend this is a finished production editor, so here's what we know about. Stretch detection still throws false positives. Some timing repairs can't be done because the available spans are too small. Caption fitting is an estimate without real font metrics. The deployed voice pipeline has been tested much more with synthesised speech and typed commands than with real microphones and a range of accents. Export handles one video at a time. The API has no authentication yet. And some deployed AWS pieces are running on borrowed credentials while the owning account gets unblocked.
We're listing these on purpose. Knowing exactly where "verified" ends and "still experimental" begins turned out to be one of the most useful things we got out of the whole hackathon.

So, did it work?

The question we started with was whether captions could reflect how someone speaks, not just what they say. After four days of empty transcripts, an LLM aligner that ate a word, race conditions, a Stop button that lied, a mid-hackathon deployment rewrite, and far too many replays of the same four clips: yes, mostly.
The captions shout. They stretch (sometimes when they shouldn't). They react to emphasis. And you can talk to the editor and ask it to change them.
What we're happiest with is the overall split: AI shows up where it actually helps, plain deterministic code handles everything else, and all of it runs through one structured project file.

Try it

Use Chrome or Edge, since the voice interface needs HTTPS mic access. If you'd rather watch first, the demo ( https://youtu.be/D836OKdoY2g ) shows the default captions, the same footage after voice editing, and a side-by-side with another captioning tool.
We built this for the AWS First Commit Hackathon, but the idea goes beyond it: editing a video shouldn't require knowing where every button is. Sometimes you should be able to just say what you want.
And after four days of debugging, the captions really do shout.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article