
Deploying a PyTorch AI Inference API on AWS EKS with Docker, Kubernetes, and Terraform
A practical guide to deploying a PyTorch-based AI inference service on AWS EKS using FastAPI, Docker, Kubernetes, and Terraform, covering scalable infrastructure, CI/CD automation, performance testing, and production deployment.
Deploying a Production-Ready PyTorch AI Inference Platform on AWS EKS
Building an AI model is only one part of solving a real-world machine learning problem. The bigger challenge begins when the model needs to be served reliably, scaled according to demand, tested under load, and deployed in a production-like cloud environment.
In this project, I built a cloud-native AI inference platform for unsupervised defect detection using PyTorch, FastAPI, Docker, Kubernetes, Terraform, and AWS. The goal was to take an ML model from experimentation to a scalable inference service with automated deployment and infrastructure management.
1. The Problem
Traditional machine learning projects often focus heavily on model accuracy while giving less attention to deployment, scalability, reliability, and observability.
For industrial defect detection, labeled defect data can also be difficult to obtain. To address this, I built an unsupervised defect detection system using the MVTec AD dataset.
The system used a PatchCore/WideResNet50 CNN-based approach and achieved:
- 0.985 AUROC
- 0.978 F1 score
- Zero labeled defects required for the detection approach
However, achieving strong model performance was only the beginning. The next challenge was turning the model into a service that could handle real requests reliably.
2. Building the Inference API
The trained model was exposed through a REST API using FastAPI.
The API provides an interface between the machine learning model and external applications. Instead of requiring users or applications to directly interact with the PyTorch model, requests can be sent to the API and the service handles inference internally.
The architecture can be summarized as:
Client | vFastAPI REST API | vPyTorch Model | vDefect Detection Result
The inference service was optimized to achieve approximately 200 ms p50 inference latency.
This separation also makes the system easier to containerize, deploy, test, and scale.
3. Containerizing the Application with Docker
To make the inference service portable and reproducible, I packaged the application inside a Docker container.
The container includes the application, dependencies, model-serving environment, and runtime configuration required by the service.
This provides several advantages:
- Consistent execution environments
- Easier deployment
- Dependency isolation
- Simplified scaling
- Reproducible builds
The Docker image was then pushed to Amazon Elastic Container Registry (ECR) so that it could be consumed by the Kubernetes cluster.
The workflow became:
Source Code | vDocker Build | vDocker Image | vAmazon ECR | vAWS EKS
4. Deploying on Kubernetes with Amazon EKS
For orchestration, I deployed the inference service on Amazon Elastic Kubernetes Service (EKS).
Kubernetes allows the application to run as a managed workload rather than depending on a single machine.
The deployment was configured to initially run with multiple pods. This provides a foundation for handling concurrent requests and scaling the inference service when demand increases.
The simplified architecture is:
AWS | EKS Cluster | ----------------------- | | | Pod 1 Pod 2 Pod 3 | | | -----------|---------- | FastAPI Service | PyTorch Inference
Using Kubernetes also makes it possible to introduce features such as automated scaling and rolling deployments.
5. Infrastructure as Code with Terraform
Manually creating cloud infrastructure can become difficult to reproduce and maintain.
To solve this, I used Terraform to define the AWS infrastructure as code.
The infrastructure included components such as:
- VPC
- ECR
- S3
- IAM
- EKS
Using Terraform makes the infrastructure configuration version-controlled and repeatable.
Instead of manually recreating the environment, the infrastructure can be represented through Terraform configuration and deployed consistently.
This also makes it easier to modify infrastructure as the application evolves.
6. Horizontal Pod Autoscaling
AI inference workloads can experience changes in traffic.
Running a large number of pods permanently can waste resources, while running too few pods can create performance bottlenecks.
To address this, I configured Horizontal Pod Autoscaling (HPA).
The deployment was configured to scale between:
2 → 10 pods
This allows Kubernetes to increase the number of inference instances when workload increases and reduce them when additional capacity is no longer required.
The result is a more flexible architecture that can adapt to changing demand.
The deployment cost was approximately $0.25/hour under the tested configuration.
7. Load Testing with Locust
A production system should not only work for a single request. It needs to be tested under concurrent workloads.
I used Locust to perform load testing against the inference service.
During testing, the system initially exposed a significant concurrency problem.
The load test revealed a threading/concurrency bug, resulting in approximately:
52% failure rate
This was an important finding because the service could appear functional during normal testing while failing under concurrent workloads.
After identifying and fixing the concurrency issue, the failure rate was reduced to:
0%
This demonstrates why load testing is an important part of deploying machine learning services.
A model can have excellent ML metrics while the overall application still fails under real-world traffic.
8. Automating Deployment with GitHub Actions
To reduce manual deployment steps, I implemented a GitHub Actions CI/CD pipeline.
The pipeline performs several stages before production deployment:
Code Push | vLint | vUnit Tests | vDocker Build | vPush Image to ECR | vProduction Deployment
The production deployment is gated so that the application passes the required validation stages before being deployed.
This creates a repeatable path from source code to production infrastructure.
9. Reliability and Testing
Machine learning systems require more than model-level validation.
The project included:
- Unit testing
- Load testing
- Containerized deployment
- CI/CD validation
- Kubernetes orchestration
- Infrastructure as Code
- Concurrency testing
The goal was to validate both the ML model and the software system surrounding it.
The model achieved strong detection performance, while the deployment architecture was tested for latency, concurrency, and scaling behavior.
10. Key Lessons
One of the biggest lessons from this project was that deploying an AI model is fundamentally different from simply training one.
A successful AI application requires multiple layers:
Machine Learning +API Development +Containerization +Cloud Infrastructure +Kubernetes +CI/CD +Testing =Production-Ready AI System
The project also demonstrated the importance of testing failure scenarios rather than only testing successful requests.
The initial 52% failure rate during load testing was particularly valuable because it exposed a concurrency problem that would not have been obvious from basic functional testing.
Conclusion
This project took an unsupervised computer vision model and transformed it into a cloud-native inference platform using PyTorch, FastAPI, Docker, Kubernetes, Terraform, AWS EKS, and GitHub Actions.
The final system combined strong ML performance with production-oriented engineering practices:
- 0.985 AUROC
- 0.978 F1
- Approximately 200 ms p50 inference latency
- 2–10 pod autoscaling
- 0% load-test failure rate after fixing the concurrency issue
- Automated CI/CD deployment
- Infrastructure managed through Terraform
The experience reinforced an important principle: building an effective AI model is only half the problem. Building the infrastructure that allows that model to operate reliably, efficiently, and at scale is equally important.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article