AWS Builder Center
Voice-Controlled Humanoid Robots Using Amazon Nova Sonic and AWS IoT

Voice-Controlled Humanoid Robots Using Amazon Nova Sonic and AWS IoT

This project uses Amazon Nova Sonic and AWS IoT for real-time, hands-free voice control of humanoid robots. By integrating AI speech-to-speech streaming with tool calling, developers can create intuitive and responsive robotic systems. The setup includes AWS IoT-enabled robots and AWS cloud infrastructure for robust and scalable voice command execution.

AWS AI Hero, Microsoft MVP - Microsoft Foundry, and Google Developer Expert Cloud(AI)

Seamless Speech-to-Speech Control for Humanoid Robots with Amazon Nova Sonic and AWS IoT

Introduction

Previously, hands-free voice control was complicated, making it difficult to determine the break points for extracting voice commands or user intentions. In the era of large language models (LLMs), we can handle text input commands using tools or function calling. However, enabling voice commands requires developers to use multiple models to convert speech to text, process it through the LLM, and then convert text back to speech. This process makes streaming input challenging, leading to significant delays and errors due to incorrect stopping points.
The ideal solution is an LLM that supports speech-to-speech streaming with integrated tool use, eliminating the need for developers to manage voice commands.  Amazon Nova Sonic  is the model we've been searching for!

Demo






Overview

Amazon Nova Robotic architecture

The system comprises two main components:  AWS IoT -enabled robots and the AWS cloud with real-time AI speech-to-speech streaming and  tool calling .
Amazon Nova Sonic Tool Use

In this blog post, I will focus on the AWS cloud. As for the AWS IoT-enabled robots, they are simply pub/sub Python services that subscribe to an AWS IoT topic. When a new message arrives, it forwards the trigger API call with a buffer queue.
Python
# Create an instance of ActionExecutor
executor = ActionExecutor()

# Callback when any publish is received
def on_publish_received(publish_packet_data):
publish_packet = publish_packet_data.publish_packet
assert isinstance(publish_packet, mqtt5.PublishPacket)
print("Received message from topic'{}':{}".format(publish_packet.topic, publish_packet.payload))
global received_count
received_count += 1

# Add action to queue based on the received payload
try:
payload = json.loads(publish_packet.payload)
action_name = payload.get("toolName")
if action_name:
executor.add_action_to_queue(action_name)
else:
print("No action specified in the payload")
except json.JSONDecodeError:
print("Invalid JSON payload received")

if received_count == input_count:
received_all_event.set()

…………….

# Create MQTT5 client
client = mqtt5_client_builder.mtls_from_path(
endpoint=input_endpoint,
port=input_port,
cert_filepath=input_cert,
pri_key_filepath=input_key,
http_proxy_options=proxy_options,
on_publish_received=on_publish_received,
on_lifecycle_stopped=on_lifecycle_stopped,
on_lifecycle_connection_success=on_lifecycle_connection_success,
on_lifecycle_connection_failure=on_lifecycle_connection_failure,
client_id=input_client_id,
keep_alive_interval_sec=5,
)
print("MQTT5 Client Created")

For the AWS cloud, the setup is built using AWS CDK. It includes the AWS IoT Thing construct ( ThingWithCert  L3 construct) and the  aws-apprunner-alpha  Service. The client certification and robot client code must be manually deployed to each Raspberry Pi.
If the  keep_alive_interval_sec  parameter isn't explicitly set in the sample code, the client might appear to hang up when, in reality, it’s simply taking a long time to reconnect due to the default setting.

How to Use Amazon Nova Sonic to Control a robot?

The web application is built using Express.js and Node.js, incorporating WebSocket for real-time voice input and speech capabilities within the web browser. We have adapted the sample code and updated the prompt as follows:
plaintext
You are a robot Command assistant.
Your primary role is to assist the user by calling available tools to perform actions or physical tasks.
Do not attempt to perform tasks directly; instead, rely on tools to achieve the desired outcomes.
Keep your responses concise and focused on the task at hand.
Don't say anything similar to "can't command the robot to perform physical actions" or "I can't do that".
When the user asks you to perform a task, respond with the name of the tool that can be used to accomplish it.
For example, if the user asks you to "make the robot stand up", you should respond with "stand".

Available tools:
- stand: Command the robot to stand up and maintain a standing position.
- go_forward: Command the robot to move forward in the direction it is currently facing.
- back_fast: Command the robot to move backward quickly.
- left_move_fast: Command the robot to move left quickly.
- right_move_fast: Command the robot to move right quickly.
- sit_ups: Command the robot to perform sit-ups.
- turn_left: Command the robot to turn left.
- turn_right: Command the robot to turn right.
- wave: Command the robot to wave its hand.
- bow: Command the robot to bow.
- squat: Command the robot to squat down.
- chest: Command the robot to perform chest exercises.
- left_shot_fast: Command the robot to perform a fast left punch.
- right_shot_fast: Command the robot to perform a fast right punch.
- wing_chun: Command the robot to perform Wing Chun moves.
- left_uppercut: Command the robot to perform a left uppercut.
- right_uppercut: Command the robot to perform a right uppercut.
- left_kick: Command the robot to perform a left kick.
- right_kick: Command the robot to perform a right kick.
- stand_up_front: Command the robot to stand up from the front.
- stand_up_back: Command the robot to stand up from the back.
- twist: Command the robot to twist its body.
- stand_slow: Command the robot to stand up slowly.
- stepping: Command the robot to perform stepping motions.
With the following tools scheme.
TypeScript
export const DefaultToolSchema = JSON.stringify({
type: "object",
properties: {},
required: [],
});

export const tools = [
{
toolSpec: {
name: "stand",
description:
"Command the robot to stand up and maintain a standing position.",
inputSchema: { json: DefaultToolSchema },
},
},
{
toolSpec: {
name: "go_forward",
description:
"Command the robot to move forward in the direction it is currently facing.",
inputSchema: { json: DefaultToolSchema },
},
},
……………
{
toolSpec: {
name: "stepping",
description: "Command the robot to perform stepping motions.",
inputSchema: { json: DefaultToolSchema },
},
},
];

Code to start the streaming session.
TypeScript
public setupPromptStartEvent(sessionId: string): void {
console.log(`Setting up prompt start event for session ${sessionId}...`);
const session = this.activeSessions.get(sessionId);
if (!session) return;
// Prompt start event
this.addEventToSessionQueue(sessionId, {
event: {
promptStart: {
promptName: session.promptName,
textOutputConfiguration: {
mediaType: "text/plain",
},
audioOutputConfiguration: DefaultAudioOutputConfiguration,
toolUseOutputConfiguration: {
mediaType: "application/json",
},
toolChoice: {
any: {},
},
toolConfiguration: {
tools: tools,
},
},
},
});
session.isPromptStartSent = true;
}

The key is to set the " toolChoice " to "any," ensuring that at least one tool is invoked each time. While the model decides which tool to call, there will always be a tool utilized. If you use the default "Auto" setting, it can be challenging to select a tool for an action.
TypeScript
public async processToolUse(
toolName: string,
toolUseContent: any
): Promise<any> {
// Implement the logic for processing tool use based on the toolName and toolUseContent
console.log(`Processing tool use: ${toolName}`);
console.log(`Tool use content:`, toolUseContent);

if (!Object.keys(Actions).includes(toolName.toLocaleLowerCase())) {
throw new Error(`Invalid tool name: ${toolName}`);
}

console.log("Processing directionTool with toolName:", toolName);
if (this.robot === "all") {
Array.from({ length: 6 }, (_, i) => i + 1).map((i) => {
this.iotPublisher
.publishToRobot(
`robot_${i}/topic`,
JSON.stringify({ toolName: toolName })
)
.catch(console.error);
});
} else {
this.iotPublisher
.publishToRobot(
this.robot + `/topic`,
JSON.stringify({ toolName: toolName })
)
.catch(console.error);
}

return {
success: true,
message: `Tool ${toolName} processed successfully.`,
};
}


Conclusion

The integration of  Amazon Nova Sonic  with AWS IoT for seamless speech-to-speech control of humanoid robots represents a significant advancement in hands-free voice command technology. By leveraging real-time AI speech-to-speech streaming and tool calling, developers can now create more intuitive and responsive robotic systems. The combination of AWS IoT-enabled robots and the AWS cloud infrastructure ensures robust and scalable solutions, making it easier to deploy and manage these systems.
This approach not only simplifies the process of voice command extraction but also enhances the accuracy and efficiency of robotic actions. As we continue to explore the capabilities of large language models and real-time AI, the potential for innovative applications in robotics and beyond is immense. Amazon Nova Sonic is a promising step towards a future where voice commands can seamlessly control complex systems, paving the way for more advanced and user-friendly technologies.

Contributors

AWS Educate Student Ambassador

 
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article