AWS Builder Center

How I Built a Production-Grade MLOps Pipeline on AWS EKS - Zero Static Credentials, Zero ClickOps

A walkthrough of building an end-to-end MLOps pipeline on AWS EKS, covering OIDC keyless authentication, Helm deployments, Prometheus observability, and the debugging sessions that documentation doesn't prepare you for.

Associate Solutions Architect | AWS · Kubernetes · Terraform · CI/CD | Technical Author

What I Built

A production-grade MLOps pipeline serving a DistilBERT sentiment classifier on AWS EKS 1.32. The architecture:
GitHub Actions (CI/CD with OIDC)
│
├── Test → Build → Push to ECR
│
└── Helm Deploy → EKS Cluster
│
┌─────┴─────┐
Pod 1 Pod 2
(DistilBERT) (DistilBERT)
└─────┬─────┘
│
LoadBalancer
│
/predict /health /metrics
│
Prometheus → Grafana

Stack at a glance:
Layer
Technology
Model
DistilBERT (HuggingFace)
API
FastAPI
Container
Docker → AWS ECR
Orchestration
Kubernetes (AWS EKS 1.32)
IaC
Terraform
CI/CD
GitHub Actions + OIDC
Package Manager
Helm
Observability
Prometheus + Grafana
Every AWS resource, VPC, EKS cluster, ECR repository, IAM roles, node groups, provisioned via Terraform. Zero manual console interaction. The whole environment can be torn down and rebuilt identically.



Engineering Decision 1: OIDC Keyless Authentication

Most CI/CD tutorials tell you to put AWS access keys in GitHub Secrets. It works. It's also the kind of thing that causes a 3am security incident six months later when someone accidentally commits a secret or a key gets leaked in a log.
I used GitHub Actions OIDC instead. Each workflow run gets a short-lived JWT from GitHub. AWS STS validates it via  AssumeRoleWithWebIdentity  and issues temporary credentials scoped to exactly the permissions the workflow needs. No static keys. Nothing to rotate. Nothing to leak.
The Terraform setup for the OIDC provider:
resource "aws_iam_openid_connect_provider" "github" {
url = "https://token.actions.githubusercontent.com"
client_id_list = ["sts.amazonaws.com"]
thumbprint_list = ["6938fd4d98bab03faadb97b34396831e3780aea1"]
}

resource "aws_iam_role" "github_actions" {
name = "github-actions-mlops"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Principal = {
Federated = aws_iam_openid_connect_provider.github.arn
}
Action = "sts:AssumeRoleWithWebIdentity"
Condition = {
StringLike = {
"token.actions.githubusercontent.com:sub" = "repo:aboodi679/mlops-sentiment:*"
}
}
}]
})
}

Then in the GitHub Actions workflow:
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::ACCOUNT_ID:role/github-actions-mlops
aws-region: us-east-1

Simple. Except it wasn't.

The Bug That CloudTrail Solved

Authentication kept failing with  AccessDenied . The trust policy looked right. The OIDC provider was configured. The role ARN was correct. I spent an hour reading GitHub's OIDC documentation before deciding to actually look at what AWS was receiving.
I enabled CloudTrail and looked at the real  AssumeRoleWithWebIdentity  event. The  sub  claim GitHub was sending:
repo:aboodi679/mlops-sentiment:ref:refs/heads/main

That part was fine. But the  sub  claim also contained something I hadn't seen documented anywhere:
user@214363717/repo@1317687475

GitHub uses numeric IDs in the sub claim, not slugs. The trust policy condition I had written matched slugs. It never matched.
The fix was adding the numeric format explicitly to the condition, or switching to a  StringLike  pattern broad enough to match both. I went with the broader pattern after verifying via CloudTrail that the numeric format was consistent.
The lesson: when OIDC authentication fails silently, CloudTrail is the only way to see what's actually happening at the STS level. The GitHub documentation doesn't mention the numeric sub claim format. I found it by reading raw API logs, not docs.



Engineering Decision 2: Model Weights in the Docker Image

The first version of my Dockerfile downloaded DistilBERT weights at container startup:
# Don't do this
from transformers import pipeline
model = pipeline("sentiment-analysis") # Downloads ~250MB at startup

Cold start on the first request: 45 seconds. Kubernetes liveness probes failed. The pod restarted. Which triggered another cold start. Which failed the liveness probe again.
The fix was downloading the weights at build time:
FROM python:3.11-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt

# Download model weights during build, not at runtime
RUN python -c "from transformers import pipeline; pipeline('sentiment-analysis')"

COPY app/ .
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

The Docker image went from ~500MB to ~1.1GB. Build time went up. But cold start went from 45 seconds to under 2 seconds, and liveness probes stopped failing. The tradeoff was worth it, image size is a one-time cost, cold start latency affects every pod restart and scale event.



Engineering Decision 3: Kubernetes Deployment Configuration

The Helm chart deploys with 2 replicas by default, scaling up to 10 via HPA:
# helm/sentiment-api/templates/deployment.yaml
replicas: 2
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "500m"
memory: "1Gi"
livenessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 10
periodSeconds: 5

The HPA:
# helm/sentiment-api/templates/hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70

Two things I got wrong the first time. First, I set  initialDelaySeconds  to 10 on the liveness probe, too short for a pod that needs to load weights and warm up FastAPI. It killed pods before they were ready. Bumping it to 30 fixed the restart loop.
Second, I forgot resource requests entirely on the first deployment. Without resource requests, the HPA has nothing to calculate utilization against. It never scales. Setting explicit CPU requests is not optional if you want HPA to work.



Engineering Decision 4: Cross-Namespace Prometheus Metrics

The FastAPI app exposes metrics at  /metrics :
from prometheus_client import Counter, Histogram, generate_latest

REQUEST_COUNT = Counter(
'inference_requests_total',
'Total inference requests',
['status']
)

INFERENCE_LATENCY = Histogram(
'inference_latency_seconds',
'DistilBERT inference latency',
buckets=[0.1, 0.25, 0.5, 1.0, 2.5, 5.0]
)

@app.get("/metrics")
def metrics():
return Response(generate_latest(), media_type="text/plain")

Getting Prometheus to scrape these metrics required a  ServiceMonitor :
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: sentiment-api-monitor
namespace: default
spec:
selector:
matchLabels:
app: sentiment-api
endpoints:
- port: http
path: /metrics
interval: 15s

It didn't work. Prometheus was running in the  monitoring  namespace, the app was in  default , and by default the Prometheus Operator only watches ServiceMonitors in its own namespace.
The fix was patching the Prometheus CR:
kubectl patch prometheus kube-prometheus-stack-prometheus \
-n monitoring \
--type=merge \
-p '{"spec":{"serviceMonitorNamespaceSelector":{}}}'

An empty  serviceMonitorNamespaceSelector  tells the operator to watch all namespaces. After that patch, metrics started appearing in Grafana within about 30 seconds.



The CI/CD Pipeline

Three stages, triggered on push to  main :
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run tests
run: |
pip install -r requirements.txt
pytest tests/ -v

build-push:
needs: test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
aws-region: us-east-1
- name: Build and push to ECR
run: |
aws ecr get-login-password | docker login --username AWS --password-stdin $ECR_URI
docker build -t $ECR_URI:${{ github.sha }} .
docker push $ECR_URI:${{ github.sha }}

deploy:
needs: build-push
runs-on: ubuntu-latest
steps:
- name: Deploy to EKS
run: |
aws eks update-kubeconfig --name mlops-cluster --region us-east-1
helm upgrade --install sentiment-api ./helm/sentiment-api \
--set image.tag=${{ github.sha }}

Every deploy uses the commit SHA as the image tag. No  latest . Every deployment is traceable to a specific commit.



What the API Looks Like

curl -X POST http://<EXTERNAL-IP>/predict \
-H "Content-Type: application/json" \
-d '{"text": "This MLOps pipeline finally works and I am unreasonably happy about it"}'

Response:
{"label": "POSITIVE", "score": 0.9997}

The  /health  endpoint returns  {"status": "healthy"}  what the liveness probe checks. The  /metrics  endpoint returns Prometheus-formatted text that Grafana ingests every 15 seconds.



What I Learned That Documentation Won't Tell You

CloudTrail is not just for auditing. When OIDC fails and you have no idea why, CloudTrail shows you the exact request AWS received and rejected. Reading it is faster than reading GitHub's OIDC docs for the fourth time.
Resource requests are load-bearing. HPA without resource requests does nothing. It's not an error, it silently refuses to scale. Set CPU requests first, then configure HPA.
Cold start kills liveness probes. If your pod needs more than 10 seconds to start, set  initialDelaySeconds  accordingly. A pod that restarts every 10 seconds because of a premature liveness check is worse than no liveness check.
Cross-namespace ServiceMonitor requires a Prometheus CR patch. This is documented, but not where you'd think to look. It's in the Prometheus Operator docs, not the ServiceMonitor docs.
The full code is at  github.com/aboodi679/mlops-sentiment . Everything is there, Terraform, Helm chart, GitHub Actions workflows, and the FastAPI app with Prometheus instrumentation.
If you're building something similar and hit the OIDC numeric sub claim issue, check your CloudTrail logs first. The answer is always in the logs.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article