Build a voice agent for IoT alarms with Amazon Nova 2 Sonic

Build an AI voice agent that phones on-call staff about IoT alarms, answers questions, and updates incidents with Amazon Nova 2 Sonic.

Godwin Vincent

Published Oct 8, 2026

Last modified Oct 8, 2026

When a refrigeration unit drifts out of range at 2:00 AM, the on-call manager usually gets an automated, one-way robocall. A recorded voice reads a short message and hangs up. To find out what happened, the manager has to sign in to a web portal, look up the asset, check the temperature trend, find the right runbook, and then go back to acknowledge the alarm. Each step adds minutes before anyone acts, and in cold-chain operations such as medication storage or food service, those minutes can mean spoiled inventory and compliance exposure.

What if the alarm call could answer questions instead? The manager could ask "How long has this been climbing?" or "What should I check first?", get an answer grounded in live sensor data and approved runbooks, and say "Acknowledge the alarm and log that I closed the door" without leaving the call.

In this post, you will learn how to build an interactive voice agent for IoT alarm response using Amazon Nova 2 Sonic, Amazon Bedrock Knowledge Bases, Amazon DynamoDB, and AWS Lambda. You will also learn how to route alarms by severity so that only critical incidents use real-time voice, while lower-severity alarms go to a lower-cost text chat channel powered by Amazon Nova Lite.

Solution overview

The solution combines three generative AI patterns:

Real-time speech-to-speech. Amazon Nova 2 Sonic is a speech-to-speech foundation model that you call through the InvokeModelWithBidirectionalStream API. Because one model handles speech understanding, reasoning, and speech generation, you don't need to chain separate speech-to-text, LLM, and text-to-speech services. The model supports natural turn-taking, handles interruptions (barge-in), and calls tools during the conversation.
Agentic tool use. The model decides during the conversation which tools to call. It reads incident, asset, and sensor data, and it writes back acknowledgments, corrective actions, and status changes.
Retrieval Augmented Generation (RAG). Troubleshooting answers come from runbooks that are indexed in Amazon Bedrock Knowledge Bases, not from the model's parametric knowledge.

The following diagram shows the solution architecture.

The workflow is as follows:

A sensor reading crosses a threshold, or an operator triggers a test alarm from the dashboard. An AWS Lambda function creates an incident record in Amazon DynamoDB.
The Lambda function reads the telephony provider credentials from AWS Secrets Manager and checks the incident severity.
For Critical incidents, the function places an outbound call through Twilio Programmable Voice. For High and Medium incidents, it sends a WhatsApp message instead.
When the manager answers, Twilio opens a bidirectional Media Streams WebSocket to an Application Load Balancer (ALB) that terminates TLS with an AWS Certificate Manager (ACM) certificate.
A WebSocket audio bridge on Amazon Elastic Compute Cloud (Amazon EC2) converts the telephony audio (8 kHz μ-law) to 16 kHz linear PCM and streams it to Amazon Nova 2 Sonic. It converts the model's audio back in the other direction.
Amazon Nova 2 Sonic delivers the briefing, answers follow-up questions, and calls tools. Tools read and write DynamoDB and retrieve runbook passages from Amazon Bedrock Knowledge Bases.
For text-channel alarms, a Lambda function receives the WhatsApp reply and runs a tool-use loop with Amazon Nova Lite through the Amazon Bedrock Converse API. It uses the same tools and the same knowledge base.

Severity-based routing

Voice is the most urgent channel, and it also costs the most per interaction. Routing by severity is a UX decision and a cost control.

Severity	Channel	Model	Interaction
Critical	Voice call	Amazon Nova 2 Sonic	Real-time, bidirectional audio with barge-in
High / Medium	WhatsApp or SMS	Amazon Nova Lite	Asynchronous text conversation with tool use
Low	Dashboard only	—	Manual review

Agent tools

The agent uses nine tools. Each tool has a narrow, well-described contract so that the model can choose reliably between similar actions, such as acknowledging an alarm and resolving an incident.

Tool	Type	Purpose
getIncident	Read	Return the full incident record
getAssetDetails	Read	Return asset name, type, location, and sensor configuration
getSensorReading	Read	Return the last 10 readings to show the trend
getAlarmRule	Read	Return the threshold, severity, and re-notification rules
acknowledgeAlarm	Write	Mark the alarm acknowledged and stop repeat notifications
recordCorrectiveAction	Write	Append a timestamped corrective action
updateIncidentStatus	Write	Move the incident to Active, On Hold, or Resolved, with a reason
searchKnowledgeBase	RAG	Retrieve troubleshooting passages from runbooks
endCall	Control	End the call after the agent says goodbye

Prerequisites

To follow this walkthrough, you need the following:

An AWS account with permissions to create IAM roles, Lambda functions, DynamoDB tables, Amazon S3 buckets, Amazon EC2 instances, an ALB, and Amazon Bedrock Knowledge Bases.
Model access in Amazon Bedrock for Amazon Nova 2 Sonic, Amazon Nova Lite, and Amazon Titan Text Embeddings V2 in your chosen AWS Region. This post uses us-east-1.
The AWS Command Line Interface (AWS CLI) and the AWS Cloud Development Kit (AWS CDK) installed and configured.
Python 3.12 and Node.js 18 or later.
A Twilio account with a voice-capable phone number, and the WhatsApp sandbox if you want to test the text channel.
A domain name that you control, so that you can issue a public ACM certificate for the WebSocket endpoint.

Note: Twilio Media Streams requires a publicly trusted TLS certificate for wss:// endpoints. A self-signed certificate on the instance will connect the call but fail silently with no audio. Terminate TLS at the ALB with an ACM certificate instead.

Walkthrough

Step 1: Deploy the data and knowledge layer

The AWS CDK infrastructure stack creates the following resources:

Five DynamoDB tables (on-demand capacity): incidents, assets, sensor readings, alarm rules (with a global secondary index on asset_id), and call sessions.
An S3 bucket, encrypted and with all public access blocked, that holds the runbook documents.
A Secrets Manager secret for Twilio credentials.

The following excerpt shows the sensor readings table. Using asset_id as the partition key and timestamp as the sort key lets the agent fetch the most recent readings with a single query.

1
2
3
4
5
6
7
self.sensor_readings_table = dynamodb.Table(
    self, "SensorReadingsTable",
    partition_key=dynamodb.Attribute(name="asset_id", type=dynamodb.AttributeType.STRING),
    sort_key=dynamodb.Attribute(name="timestamp", type=dynamodb.AttributeType.STRING),
    billing_mode=dynamodb.BillingMode.PAY_PER_REQUEST,
    removal_policy=RemovalPolicy.DESTROY,
)

Deploy the stacks and load sample data:

1
2
3
4
5
6
7
8
cd cdk
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cdk bootstrap   # first time only
cdk deploy --all

cd ../data
python seed_dynamodb.py

Store your Twilio credentials in the secret. Don't put credentials in code or in Lambda environment variables.

1
2
3
aws secretsmanager put-secret-value \
  --secret-id <your-twilio-secret-name> \
  --secret-string '{"account_sid":"<sid>","auth_token":"<token>","from_number":"<+1...>","to_number":"<+1...>"}'

Step 2: Create the troubleshooting knowledge base

The knowledge base contains seven short Markdown runbooks: temperature excursion response, door sensor troubleshooting, compressor failure, power outage procedures, sensor malfunction, alarm acknowledgment policy, and corrective action examples.

Open the Amazon Bedrock console and choose Knowledge Bases, then Create knowledge base.
Select the S3 bucket from Step 1 as the data source.
Choose Titan Text Embeddings V2 as the embedding model.
Choose a vector store. This solution used Amazon OpenSearch Serverless. For small corpora or cost-sensitive workloads, consider Amazon S3 Vectors as the vector store.
Sync the data source and note the knowledge base ID.

Keep runbooks short and focused on one failure mode each. Small, single-topic documents produce cleaner retrieval results and make spoken answers more concise.

Step 3: Build the voice audio bridge

The audio bridge is a Python WebSocket server that runs on Amazon EC2 behind the ALB. It handles the Twilio Media Streams protocol events (connected, start, media, stop) and streams audio to Amazon Nova 2 Sonic by using the bidirectional model in the Strands Agents SDK .

Pre-fetch context into the system prompt. When the stream starts, the server reads the incident and asset records and injects them into the system prompt. The agent can start the briefing immediately, without spending its first turn on tool calls.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
system_prompt = (
    "You are an AI voice assistant for IoT alarm management. "
    "You are on a live phone call with a facility manager about an active alarm.\n\n"
    f"CURRENT INCIDENT: {incident_id}\n"
    f"- Severity: {incident.get('severity')}\n"
    f"- Asset: {asset.get('name')} (ID: {incident.get('asset_id')})\n"
    f"- Reading: {incident.get('reading')}F (threshold: {incident.get('threshold')}F)\n"
    f"- Door state: {incident.get('door_state')}\n\n"
    "Start speaking immediately. Do not call tools before the opening briefing.\n"
    "After the briefing:\n"
    "- Do not call getIncident or getAssetDetails; you already have this data.\n"
    "- Use getSensorReading for history and searchKnowledgeBase for troubleshooting.\n"
    "- Keep responses to 2-3 sentences.\n"
    "- After you say goodbye, call endCall."
)

model = BidiNovaSonicModel(region=REGION)
await model.start(system_prompt=system_prompt, tools=TOOLS)

Convert audio in both directions. Twilio sends 8 kHz μ-law audio. Amazon Nova 2 Sonic consumes and produces 16 kHz, 16-bit mono PCM. Convert each chunk as it arrives so that latency stays low.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# Inbound: Twilio -> Amazon Nova 2 Sonic
mulaw_bytes = base64.b64decode(data["media"]["payload"])
pcm_8k = audioop.ulaw2lin(mulaw_bytes, 2)
pcm_16k, _ = audioop.ratecv(pcm_8k, 2, 1, 8000, 16000, None)
await model.send(BidiAudioInputEvent(
    audio=base64.b64encode(pcm_16k).decode(),
    format="audio/lpcm", sample_rate=16000, channels=1))

# Outbound: Amazon Nova 2 Sonic -> Twilio
pcm_8k, _ = audioop.ratecv(pcm_16k_out, 2, 1, 16000, 8000, None)
mulaw = audioop.lin2ulaw(pcm_8k, 2)
await websocket.send(json.dumps({
    "event": "media", "streamSid": stream_sid,
    "media": {"payload": base64.b64encode(mulaw).decode()}}))

Note: The audioop module was removed from the Python standard library in Python 3.13. If you use Python 3.13 or later, install the audioop-lts package or use another resampling library.

Support barge-in. When the manager talks over the agent, the model emits an interruption event. Send Twilio a clear message so that buffered agent audio stops playing right away.

1
2
elif isinstance(output_event, BidiInterruptionEvent):
    await websocket.send(json.dumps({"event": "clear", "streamSid": stream_sid}))

Handle tool calls during the stream. When the model emits a tool-use event, run the tool and send the result back on the same stream. The conversation continues without dropping the call.

1
2
3
4
5
6
7
elif isinstance(output_event, ToolUseStreamEvent):
    tool_use = output_event.get("current_tool_use", {})
    if tool_use.get("name") and tool_use.get("toolUseId") and tool_use.get("input"):
        result_text = execute_tool(tool_use["name"], tool_use["input"])
        await model.send(ToolResultEvent(tool_result=ToolResult(
            toolUseId=tool_use["toolUseId"],
            content=[{"text": result_text}], status="success")))

Configure the ALB for WebSocket traffic. Use an HTTPS listener on port 443 with your ACM certificate, forward to the instance on port 8080, set the idle timeout high enough for a full call (for example, 3,600 seconds), and turn off HTTP/2 on the load balancer. Point a DNS record, such as an Amazon Route 53 alias record, at the ALB.

Step 4: Implement the tools

Each tool is a small function that calls DynamoDB or the Bedrock knowledge base. For example, searchKnowledgeBase calls the Retrieve API and returns the top passages as text for the model to summarize out loud:

1
2
3
4
5
6
7
resp = bedrock_agent_client.retrieve(
    knowledgeBaseId=KB_ID,
    retrievalQuery={"text": tool_input["query"]},
    retrievalConfiguration={"vectorSearchConfiguration": {"numberOfResults": 2}},
)
results = [r["content"]["text"] for r in resp.get("retrievalResults", [])]
return "\n\n".join(results) if results else "No relevant guidance found."

Write tools such as recordCorrectiveAction append a timestamped entry to the incident, so the dashboard shows what the manager did, in their own words, as soon as they say it.

Return a structured error message, not an exception, when a tool fails. The model can then tell the caller that something went wrong instead of going silent.

Step 5: Route alarms by severity

The alarm Lambda function creates the incident and then chooses the channel. For a critical alarm, it passes inline TwiML that connects the call directly to the WebSocket endpoint and includes the incident ID as a custom stream parameter:

1
2
3
4
5
6
7
8
9
if severity == "Critical":
    twiml = (
        f'<Response><Connect><Stream url="{WS_URL}">'
        f'<Parameter name="incident_id" value="{incident_id}" />'
        '</Stream></Connect></Response>'
    )
    # POST to the Twilio Calls API with To, From, and Twiml
else:
    # Send a WhatsApp alert that summarizes the incident and invites a reply

Passing the TwiML inline removes the need for a separate public webhook to answer the call.

Step 6: Add the text chat channel

For High and Medium alarms, a Lambda function receives WhatsApp replies and runs a standard tool-use loop with the Amazon Bedrock Converse API and Amazon Nova Lite. The function calls converse with the same tool definitions, runs any requested tools, appends the toolResult blocks, and repeats until stopReason is no longer tool_use. Because both channels share the tools and the knowledge base, answers are consistent whether the manager talks or types.

Step 7: Test the solution

Start the dashboard, select an asset, set severity to Critical, and choose Trigger alarm.
Answer the call. The agent briefs you on the asset, the reading, the threshold, and the door state.
Ask, "How long has this been climbing?" The agent calls getSensorReading and describes the trend.
Ask, "What should I do?" The agent calls searchKnowledgeBase and walks you through the runbook steps.
Say, "Acknowledge the alarm and record that I closed and latched the door." The agent calls acknowledgeAlarm and recordCorrectiveAction.
Confirm that the incident status and corrective action appear on the dashboard.
Trigger a Medium alarm and reply to the WhatsApp message to test the text channel.

Evaluating the agent

A voice agent that reads alarm data aloud needs evidence that it is accurate, not only that it sounds good. The project evaluated six areas:

Area	Metrics
Briefing quality	BERTScore, ROUGE-L, and BLEU against reference briefings
Knowledge base retrieval	Precision@3 and NDCG@3 over a labeled query set
Routing and tool selection	Accuracy, weighted F1, and confusion matrices
Voice latency	Average and P95 turn latency, and violations of a 500 ms target
Task completion	End-to-end success: briefing, correct tool calls, and correct write-back
Human evaluation	5-point Likert ratings for fluency, helpfulness, and accuracy

On the test sets used in this project, severity routing (8 cases) and tool selection (12 cases) were classified correctly, KB retrieval reached a Precision@3 of 0.97 across 10 queries, and 11 rated sessions averaged 4.58 out of 5 overall, with no factual errors observed because every fact came from a tool result. Treat these as directional results from a small synthetic dataset. Expand the test sets and use independent raters before you rely on them in production.

The most common failures were not model errors but integration issues: certificate validation on the media stream, sample-rate mismatches, and calls answered by voicemail. Build monitoring for these first.

Production considerations

Security. Validate the X-Twilio-Signature header on every inbound webhook, grant each Lambda function and the EC2 instance role only the DynamoDB tables and Bedrock actions they use, keep credentials in Secrets Manager, and place the EC2 instance in private subnets reachable only from the ALB.
Guardrails. Add Amazon Bedrock Guardrails to the text channel to filter denied topics and sensitive information.
Session length. Amazon Nova 2 Sonic connections have an 8-minute limit. For longer calls, implement the connection renewal and session continuation pattern described in the Amazon Nova documentation.
Scale. The sample runs a single instance. For concurrent calls, run the bridge in an Auto Scaling group or on Amazon ECS behind the ALB. You can also evaluate Amazon Bedrock AgentCore Runtime , which supports WebSocket bidirectional streaming for voice agents, as a managed hosting option.
Languages. Amazon Nova 2 Sonic supports multiple languages, including English, French, Italian, German, Spanish, Portuguese, and Hindi, so you can extend the agent to multilingual teams.
Cost. Voice minutes are the largest variable cost, which is why the solution sends only critical alarms to voice. On the fixed-cost side, review your vector store choice for small knowledge bases.

Cleaning up

To avoid ongoing charges, delete the resources that you created:

Delete the knowledge base and its vector store (for example, the OpenSearch Serverless collection) in the Amazon Bedrock console.
Destroy the AWS CDK stacks
Delete the ALB, target group, EC2 instance, ACM certificate, and DNS records if you created them outside the CDK stacks.
Delete the Secrets Manager secret if it was retained.
Release the Twilio phone number if you no longer need it.

Conclusion

In this post, you learned how to build an interactive voice agent that turns a one-way alarm call into a two-way conversation. Amazon Nova 2 Sonic delivers a grounded briefing, answers follow-up questions, and takes actions by voice. Amazon Bedrock Knowledge Bases supplies runbook guidance, and DynamoDB tools keep the incident record current. Severity-based routing reserves real-time voice for critical alarms and sends everything else to a lower-cost text channel.

You can apply the same pattern to any operational alert that needs a human decision: manufacturing line stoppages, network outages, or facility security events. To get started, see the Amazon Nova 2 Sonic user guide and the Amazon Bedrock Knowledge Bases documentation .

# amazon-nova # agentic-ai # ai # amazon-bedrock # rag

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

Build a voice agent for IoT alarms with Amazon Nova 2 Sonic

Godwin Vincent

Solution overview

Severity-based routing

Agent tools

Prerequisites

Walkthrough

Step 1: Deploy the data and knowledge layer

Step 2: Create the troubleshooting knowledge base

Step 3: Build the voice audio bridge

Step 4: Implement the tools

Step 5: Route alarms by severity

Step 6: Add the text chat channel

Step 7: Test the solution

Evaluating the agent

Production considerations

Cleaning up

Conclusion