AWS Builder Center

Builder Center

Building a Bilingual Voice Agent for Waste Pickup with Strands, Twilio, and Amazon Polly and What a Pure-AWS Stack Would Look Like

A technical walkthrough of building a bilingual (English/Hindi) conversational voice agent on AWS Lambda using the Strands Agents SDK, Twilio speech-to-text, and Amazon Polly neural voices with a breakdown of how to replace Twilio entirely with Amazon Connect and native AWS services.

EcoLink's main interface is WhatsApp. But not everyone types. Some users would rather just call.
So we built a second entry point: a phone number you call, speak naturally in English or Hindi, and get the same waste agent same state, same guardrails, same kabadiwala dispatch over voice. This post covers how we built it, what Twilio and Amazon Polly actually do in the stack, and how you could replace Twilio entirely with native AWS services if you wanted to.

How the Voice Turn Loop Works

Every inbound call is a series of HTTP round-trips between Twilio and our Lambda function. The pattern:

No WebSockets, no streaming audio, no media servers. Each turn is a plain HTTP POST from Twilio with SpeechResult in the form body, and a plain XML response back. The Lambda function is stateless per invocation; all conversation state lives in DynamoDB, keyed by the caller's phone number prefixed with  voice:  so it never collides with WhatsApp user IDs.


Amazon Polly via Twilio's  <Say>  Verb

Twilio's  <Say>  verb has a voice attribute that accepts Amazon Polly neural voices directly — no extra integration needed. We use two:
plaintext
_VOICES = {

    "en": {"voice": "Polly.Raveena", "language": "en-IN"},

    "hi": {"voice": "Polly.Aditi", "language": "hi-IN"},

}
Raveena (en-IN) and Aditi (hi-IN) are both Indian-English and Hindi neural voices — they sound natural with Indian names, city names, and rupee amounts, which matters when the agent says things like "estimated Rs 110 to 185 for your paper and cardboard, Indiranagar pickup."
The TwiML builder constructs this entirely with stdlib — no Twilio Python SDK, no XML library:
plaintext
def _say(text: str, lang: str) -> str:

    cfg = _VOICES.get(lang, _VOICES["en"])

    return (

        f'<Say voice={quoteattr(cfg["voice"])} language={quoteattr(cfg["language"])}>'

        f"{escape(text)}"

        f"</Say>"

    )
 xml.sax.saxutils.escape  and  quoteattr  from stdlib handle all the XML escaping. No dependencies.

Language Detection Per Turn

We don't ask the user to pick a language upfront. Every turn, the handler checks whether the transcribed speech is Hindi:
plaintext
def _detect_lang(text: str) -> str:

    """Heuristic: >10% Devanagari characters → Hindi."""

    if not text:

        return "en"

    deva = sum(1 for ch in text if "\u0900" <= ch <= "\u097f")

    return "hi" if deva / len(text) > 0.10 else "en"
The Twilio  <Gather>  is configured with both en-IN and hi-IN as recognised languages. If the user switches mid-call from English to Hindi (common in India), the next turn detects it and the agent switches the Polly voice accordingly.
Romanised Hindi ("haan bhai", "theek hai") is treated as English for language detection — the Devanagari check misses it — but the confirmation gate regex catches it regardless since it explicitly handles both scripts.

Goodbye, Transfer, and Barge-In

Three edge cases worth noting:
Goodbye detection runs before the agent is invoked:
plaintext
_GOODBYE_PATTERN = re.compile(

    r"\b(bye|goodbye|stop|end|quit|hang up|band karo|बंद|अलविदा|बाय)\b",

    re.IGNORECASE | re.UNICODE,

)
If matched, the handler returns a goodbye_twiml() response — no agent call, no DynamoDB write, just <Say> + <Hangup>. Operator transfer is an escape hatch: if the user says "operator", "human", "insaan" etc., and VOICE_TRANSFER_NUMBER is set as a Lambda env var, the call transfers to that number with <Dial>. Useful if a household gets confused and needs a real person. Barge-in is free with Twilio's <Gather>. The agent's reply is spoken inside the <Gather> element, so the user can interrupt mid-sentence and Twilio will cut the TTS and start capturing their speech immediately:
plaintext
def reply_twiml(agent_reply: str, turn_url: str, lang: str = "en") -> str:

    inner = _say(agent_reply, lang)          # spoken inside Gather → barge-in works

    gather = _gather(turn_url, lang, inner)

    nudge = _say(_SILENCE_NUDGE[lang], lang) # fallback if user goes silent

    return _response(gather, nudge)

The Same Agent, The Same State

The voice handler calls  EcoLinkAgent.handle()  — the exact same Strands agent that runs for WhatsApp. All the guardrails apply identically:
  • The confirmation gate still requires an explicit "yes" / "हाँ" before a pickup is created
  • Hazardous items still get routed to drop-off centres, not a kabadiwala call
  • The household address still only reaches the recycler after they accept
  • All price outputs are still labelled "estimated Rs X-Y"
The agent doesn't know or care whether the input came from WhatsApp or a phone call. It gets a  user_id  and a text string.  voice:+919800000001  and  whatsapp:+919800000001  are different user IDs, so their conversation states are separate — but the logic is identical.
One difference: voice callers can't send photos. When the agent's  classify_image  tool would normally be invoked, it gets  image_url=None . The agent handles this gracefully by asking the user to describe the items verbally instead.

What a Pure-AWS Stack Would Look Like

Twilio does two things in this flow: speech-to-text (STT) and text-to-speech (TTS). AWS has native services for both, and for the telephony layer itself. Here's how you'd replace each piece:

STT: Amazon Transcribe (Streaming) instead of Twilio's  <Gather speech> 

Twilio's gather does STT internally and posts the transcript to your webhook. If you want to own the audio:
  • Use Amazon Connect to receive inbound calls. Connect has a built-in Contact Flow that can stream audio to a Lambda via Amazon Lex or directly to Amazon Transcribe Streaming over a WebSocket.
  • Alternatively, use Twilio Media Streams to pipe raw audio to your own WebSocket endpoint on API Gateway, then stream it to  TranscribeStreamingClient  for real-time transcription.
The Amazon Transcribe Streaming API gives you partial results as the user speaks, which lets you detect intent earlier and reduce perceived latency.

TTS: Amazon Polly Directly instead of Polly-via-Twilio

Right now Polly runs inside Twilio's infrastructure — we just name the voice and Twilio calls Polly on our behalf. To own the pipeline:
plaintext
import boto3



polly = boto3.client("polly", region_name="ap-south-1")

response = polly.synthesize_speech(

    Text=agent_reply,

    OutputFormat="mp3",

    VoiceId="Raveena",       # or "Aditi" for Hindi

    LanguageCode="en-IN",

    Engine="neural",

)

audio_bytes = response["AudioStream"].read()
Upload the MP3 to S3, generate a pre-signed URL, and use Twilio's  <Play>  verb instead of  <Say> . Or, with Amazon Connect, feed the audio bytes directly into the contact flow. You get full control over SSML, speech rate, and prosody.

Telephony: Amazon Connect instead of Twilio

Amazon Connect is AWS's cloud contact centre. For a voice agent:
  • Inbound number: claim a Connect number (Indian DID available) — no Twilio account needed
  • Contact Flow: the visual drag-and-drop flow replaces the TwiML state machine
  • Lambda integration: Connect calls your Lambda directly from a "Invoke Lambda" block in the contact flow, passing the transcript and returning the agent reply
  • No signature verification needed: the call comes from within AWS, so you use IAM instead of HMAC webhook signatures
The rough Connect equivalent of our TwiML loop:
plaintext
[Start] → [Get Customer Input (Lex/STT)] → [Invoke Lambda (EcoLinkAgent)] → [Play Prompt (Polly)] → loop
The main tradeoff: Amazon Connect has a per-minute cost ($0.018/min for inbound in India as of writing) vs Twilio's per-minute rate. At hackathon volume, both are negligible. At scale, you'd model them properly.

Why We Kept Twilio

Three reasons:
  1. Speed: Twilio's sandbox gives you a WhatsApp number and a voice number in minutes. Amazon Connect provisioning, especially for Indian DIDs, takes longer to set up and requires a business registration.
  1. Unified provider: EcoLink's  MessagingProvider  and  VoiceProvider  interfaces already abstract Twilio behind a pluggable adapter. Swapping in a Connect adapter is a future  providers/connect.py  file, not a rewrite.
  1. The TwiML loop is simple: For a turn-based conversation agent, plain HTTP webhooks + XML are easier to debug than a stateful WebSocket stream. The simplicity was worth it for a weekend build.

What Fought Back

Twilio's 15-second webhook timeout. The Strands agent sometimes takes 8–12 seconds on a complex multi-tool run (classify → price → find recyclers). A voice call can't wait silently for 12 seconds — the user thinks the line dropped.
Fix: we speak a short acknowledgement while the agent runs. The  <Gather>  starts immediately with "Let me check that for you…" as its inner  <Say> , buying a few seconds. For longer runs, a "thinking" TTS phrase plays and the real reply comes in the next turn.
Cold starts on the agent Lambda. The first call after a period of inactivity would hit a cold start — Strands + boto3 imports take ~3 seconds. The greeting TwiML buys exactly that time. Subsequent turns on a warm Lambda are fast.
Hindi Romanisation. "Haan, book karo" has no Devanagari, so it's detected as English and Raveena reads the reply. But the user spoke Hindi. We accept this for now — the confirmation regex catches "haan" regardless, and the agent mirrors the user's style. A proper fix would use Amazon Comprehend's language detection on the full turn text.

Summary

The voice agent is ~350 lines across two files ( voice_agent.py ,  voice_agent_twiml.py ). It reuses the same DynamoDB state table, the same Strands agent, and the same guardrails as the WhatsApp channel. Twilio handles the telephony and STT; Amazon Polly handles the TTS via Twilio's  <Say voice="Polly.Raveena"> . A pure-AWS alternative exists and is architecturally cleaner, but Twilio won on setup speed for a weekend build.
The full source is at  https://github.com/NanG08/EcoLink_Team_OffBeat 

Built at #EnvironmentalHacks, #BharatBuildsTour · Track: Waste and Energy · Stack: Strands Agents SDK, Amazon Bedrock, Amazon Polly, AWS CDK, Lambda, DynamoDB, S3, Cognito, SNS, Secrets Manager | Written with help of AWS Kiro
# aws-cdk# amazon-cognito# amazon-bedrock# amazon-polly# aws-lambda
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article