How Vbee scales Speech Inferencing to 10s of millions of requests per day with AWS Lambda

Vbee  is a leading Speech AI company in Vietnam, delivering text-to-speech, speech recognition and conversational AI solutions to enterprises across finance, education, media, and government sectors. As demand for speech synthesis continues to grow, Vbee needed an inference architecture that can scale elastically while maintaining low latency and cost efficiency. 

Vbee started out with hosting their deep learning models on on-premises GPU servers and Amazon EC2 GPU instances. However, this approach was not optimal for Vbee’s requirement of running hundreds of voice models to serve diverse customer needs. The system faced unpredictable workloads due to the popularity gap between models. Top-tier voices, favored by large content creators for their versatility and quality, maintain high usage. Conversely, niche and newly launched voices experience sporadic demand, as they are tailored for fewer use cases and have not yet built a consistent user base. Significant GPU idle time on EC2 instances effectively wasted 35% of the hosting budget. Not to mention that not all model inferencing requires GPU, some can work very well with CPU. Additionally, managing and operating a large fleet of VM-based inference services required substantial operational effort, increasing the burden on the engineering team.

In this blog, you will see how AWS Lambda’s fast on-demand scaling and zero maintenance approach helps Vbee to handle their voice inferencing at scale while reducing the burden on the operations team to manage the infrastructure. You will see how Vbee’s engineers streamlined model packaging, cold start strategies, and orchestration workflows for maximum efficiency within the Lambda eco-system. 

Let’s start with the overall architecture.

With over 10,000 AI voice models and counting in their catalog, Vbee can address diverse speech synthesis use cases across many different languages, regional accents, and speaking styles. Tens of thousands of users are using Vbee’s models on a daily basis, driving to a thousand concurrent requests during peak hours or viral content periods. This number fluctuates significantly during the days, weeks and months, which can be as low as fewer than 10 requests concurrently (figure 1).

The Vbee Text to Speech (TTS) system utilizes a hybrid design, leveraging Amazon EKS for orchestration and AWS Lambda for serverless, distributed processing. The synthesis request received by a Network Load Balancer, then routed to a Coordinator (TTS GATE) within an EKS cluster, which utilizes Apache Kafka to manage asynchronous task queues and metadata (figure 2).

By using EKS, Vbee maintains a persistent, high-speed connection to Kafka and gains a centralized "orchestrator" that can manage complex retries and logic with lower latency and better cost-predictability than a purely serverless workflow.

For each concurrent request, EKS-based TTS Workers orchestrate a four-step serverless pipeline using AWS Lambda. The input text is first split into manageable chunks and then passed to a Text-To-Phoneme function for standardization using a Java core. For the core acoustic synthesis utilizing PyTorch models stored in Amazon S3, the request is routed to the appropriate Lambda function based on the requested voice: it either targets a dedicated function that auto-loads a specific standard voice at runtime, or a flexible voice-cloning function capable of dynamically loading multiple custom models depending on the input voice ID. Next, a Join function merges the synthesized audio chunks into a complete file, which is securely exported and stored in S3. Finally, a presigned URL is generated and provided to the user, allowing them to easily access and download the final audio result.

Let’s start with how Vbee optimizes model runtimes to load and serve the voice models.

Deep learning models for speech are typically very large, in the range from hundreds of MBs to several GBs. The main challenge is fitting these models within Lambda’s memory limits and the 10GB ephemeral storage limit, while maintaining fast loading times.

In Vbee’s case, the team prioritizes Non-Autoregressive (NAR) architectures over Autoregressive (AR), because NAR generates all frames in parallel, which results in faster latency that fits Lambda’s 15-minute limit. Although AR provides better quality for some voices, it is slower due to the fact that AR generates sequentially.

Additionally, by utilizing ONNX format for models, Vbee transforms heavy Deep Learning models into lean, high-performance inference engines. This reduces the runtime dependency footprint by over 80% (replacing heavy PyTorch with ONNX Runtime) and achieves up to 2x faster CPU inference. By migrating Lambda functions to Arm-based AWS Graviton processors, Vbee receives 10% better performance at 10% lower cost compared to x86 functions. Loading large dependencies like PyTorch or TensorFlow, can significantly increase the Lambda’s initialization time, leading to higher costs and latency. Combining NAR and ONNX becomes a natural fit for Lambda.

Vbee also moves the ONNX runtime (ORT) session initialization outside it’s Lambda function handler. Resulting in reducing session-creation cost by loading it at Lambda environment time instead of every single invocation. Below is an example code of an inference function handler.

Currently, Vbee does not use quantization (such as INT8) for two primary reasons. First, because their architecture primarily utilizes small to medium-sized models, the reduction in model size and latency gained from quantization does not provide significant enough value to justify the trade-offs. Second, and most importantly, applying quantization to TTS models noticeably degrades the output voice quality. Because audio synthesis is highly sensitive, listeners can clearly perceive this loss in acoustic fidelity; therefore, Vbee prioritizes maintaining the highest possible naturalness and clarity in our synthesized voices over model compression.

Next is how Vbee reduces the Lambda package size to improve load time.

Lambda supports ZIP packages up to 50 MB compressed and 250 MB uncompressed. For larger models or cases requiring specific libraries that cannot fit into these limits, Vbee uses Docker containers instead (Lambda supports images up to 10 GB). When models exceed ZIP limits but don’t warrant a full container image, Vbee stores model artifacts in S3 and downloads them into memory during function initialization. By prioritizing ZIP deployments and utilizing S3-backed model loading, Vbee avoids the container image pull latency and reduces cold start from over 10 seconds down to only a few seconds. This ensures that even the most specialized, infrequently-used voice models remain responsive and commercially viable.

Custom library-cleaning scripts are used to remove unnecessary files from Python’s site-packages folder. Resulting in deploying heavy Machine Learning workloads to environments with strict storage limits, such as AWS Lambda.

Vbee has built the following script to reduce the footprint of Python site-packages directory. It strips out assets that are not strictly required for runtime. It recursively deletes development caches (pycache), test suites, documentation, package metadata (.dist-info), and pre-compiled bytecode (.pyc). Most importantly, it uses the strip command to remove unnecessary debugging symbols from compiled C/C++ shared libraries (.so files), which can shrink the size of large ML dependencies 30-70% in some cases. In the Vbee case, it reduces the site-packages folder by 50%.

Vbee has built a process called “validation-driven pruning”, which identifies large .so files and performs a “destructive test” by temporarily removing them and validating the service. If the service and its core tests (Torch, ONNX) still pass after Vbee deleted these files, they are marked as redundant, and safely cut significant deployment size.

Below is the snippet Bash script that identifies the largest .so files and test running the service with each of those files deleted.

As a result, a sample ZIP package could look like below, which is a lean project structure after package size optimization:

Vbee also utilizes AWS Lambda Layers to store essential audio processing libraries, including Sox and FFmpeg. This modular strategy serves to minimize deployment ZIP sizes and accelerate deployment speeds, as these layers can be reused globally across various voice model functions. This approach ensures a lean codebase while maintaining high-performance audio manipulation capabilities across the entire TTS pipeline.

To right-size each function’s memory allocation, the Vbee team turned to AWS Lambda Power Tuning  - an open-source tool, available in the AWS Serverless Application Repository, that empirically determines the optimal configuration by running concurrent invocations at different memory settings.

For example, one of Vbee’s 80M-parameter models achieves optimal cost-performance at 3072 MB of Lambda memory, processing 1,000 characters in 10 seconds. While this represents a 15% decrease in performance compared to 4096 MB, it yields a 25% reduction in cost, making it a favorable tradeoff. The team also categorized their functions based on usage, impact and importance, to identify the ones that require right-sizing with Lambda Power Tuning, which is about 70% of all their inference functions. The benchmark results and learned insights are also useful to right-size the remaining similar functions.

After going through the whole exercise, Vbee also found that speech models with 40M parameters or fewer are excellent deployment candidates on Lambda because of the fast processing speed. Models between 40M and 100M parameters are also potentially deployable, but expect higher cost per inference. For models over 100M parameters, while some can still be deployed, it is generally not cost-effective due to high RAM requirements, long cold init time and long processing times.

Using the optimisations presented earlier, Vbee now scales from a few requests per second to thousands of concurrent requests without manual infrastructure intervention. For a company serving hundreds of voice models with wildly different demand patterns, this elasticity means Vbee can onboard new enterprise clients and expand into new markets without capacity planning or over-provisioning.

Lambda's parallel execution model also made request latencies more stable and predictable. It takes approximately 7 seconds on average to process 1,000 characters. While individual inference latency is higher than on GPU (7 seconds vs. 4 seconds per 1,000 characters), the significantly lower variance (±1s vs. ±3s) delivers more predictable quality of service for end users.

"Moving our speech inference to serverless has removed infrastructure as a bottleneck for growth," said Nguyen Thi Thu Trang, CTO of Vbee. "Our engineers now focus on deep learning research and voice quality instead of managing servers - and that shift is what gives us the confidence to scale globally."

As Vbee’s traffic patterns mature, the team is evolving their architecture to a hybrid model - combining Lambda with EKS on GPU-backed EC2 instances to serve models that require a GPU. This hybrid approach routes predictable, steady-state traffic to GPU instances for the lowest possible latency, while letting Lambda absorb the unpredictable, bursty demand that originally motivated the serverless migration. For example, Vbee’s most popular voice models - which receive consistent, high-volume traffic - are strong candidates for dedicated GPU inference, while the long tail of 10,000+ less-frequently-used models remains an ideal fit for Lambda’s pay-per-invocation pricing.

Vbee's journey shows that serverless and GPU-based inference aren't an either-or choice - they're complementary. With Lambda, Vbee could validate their models, understand their traffic patterns, and build operational confidence before investing in dedicated GPU capacity where it matters most.

For more learning resources on serverless and Lambda, visit Serverless Land .