How I Built a Production-Grade MLOps Pipeline on AWS EKS - Zero Static Credentials, Zero ClickOps
A walkthrough of building an end-to-end MLOps pipeline on AWS EKS, covering OIDC keyless authentication, Helm deployments, Prometheus observability, and the debugging sessions that documentation doesn't prepare you for.
What I Built
A production-grade MLOps pipeline serving a DistilBERT sentiment classifier on AWS EKS 1.32. The architecture:
GitHub Actions (CI/CD with OIDC) │ ├── Test → Build → Push to ECR │ └── Helm Deploy → EKS Cluster │ ┌─────┴─────┐ Pod 1 Pod 2 (DistilBERT) (DistilBERT) └─────┬─────┘ │ LoadBalancer │ /predict /health /metrics │ Prometheus → Grafana
Stack at a glance:
Layer | Technology |
|---|---|
Model | DistilBERT (HuggingFace) |
API | FastAPI |
Container | Docker → AWS ECR |
Orchestration | Kubernetes (AWS EKS 1.32) |
IaC | Terraform |
CI/CD | GitHub Actions + OIDC |
Package Manager | Helm |
Observability | Prometheus + Grafana |
Every AWS resource, VPC, EKS cluster, ECR repository, IAM roles, node groups, provisioned via Terraform. Zero manual console interaction. The whole environment can be torn down and rebuilt identically.
Engineering Decision 1: OIDC Keyless Authentication
Most CI/CD tutorials tell you to put AWS access keys in GitHub Secrets. It works. It's also the kind of thing that causes a 3am security incident six months later when someone accidentally commits a secret or a key gets leaked in a log.
I used GitHub Actions OIDC instead. Each workflow run gets a short-lived JWT from GitHub. AWS STS validates it via
AssumeRoleWithWebIdentity and issues temporary credentials scoped to exactly the permissions the workflow needs. No static keys. Nothing to rotate. Nothing to leak.The Terraform setup for the OIDC provider:
resource "aws_iam_openid_connect_provider" "github" { url = "https://token.actions.githubusercontent.com" client_id_list = ["sts.amazonaws.com"] thumbprint_list = ["6938fd4d98bab03faadb97b34396831e3780aea1"]}
resource "aws_iam_role" "github_actions" { name = "github-actions-mlops" assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [{ Effect = "Allow" Principal = { Federated = aws_iam_openid_connect_provider.github.arn } Action = "sts:AssumeRoleWithWebIdentity" Condition = { StringLike = { "token.actions.githubusercontent.com:sub" = "repo:aboodi679/mlops-sentiment:*" } } }] })}
Then in the GitHub Actions workflow:
- name: Configure AWS credentials uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: arn:aws:iam::ACCOUNT_ID:role/github-actions-mlops aws-region: us-east-1
Simple. Except it wasn't.
The Bug That CloudTrail Solved
Authentication kept failing with
AccessDenied . The trust policy looked right. The OIDC provider was configured. The role ARN was correct. I spent an hour reading GitHub's OIDC documentation before deciding to actually look at what AWS was receiving.I enabled CloudTrail and looked at the real
AssumeRoleWithWebIdentity event. The sub claim GitHub was sending:repo:aboodi679/mlops-sentiment:ref:refs/heads/main
That part was fine. But the
sub claim also contained something I hadn't seen documented anywhere:user@214363717/repo@1317687475
GitHub uses numeric IDs in the sub claim, not slugs. The trust policy condition I had written matched slugs. It never matched.
The fix was adding the numeric format explicitly to the condition, or switching to a
StringLike pattern broad enough to match both. I went with the broader pattern after verifying via CloudTrail that the numeric format was consistent.The lesson: when OIDC authentication fails silently, CloudTrail is the only way to see what's actually happening at the STS level. The GitHub documentation doesn't mention the numeric sub claim format. I found it by reading raw API logs, not docs.
Engineering Decision 2: Model Weights in the Docker Image
The first version of my Dockerfile downloaded DistilBERT weights at container startup:
# Don't do thisfrom transformers import pipelinemodel = pipeline("sentiment-analysis") # Downloads ~250MB at startup
Cold start on the first request: 45 seconds. Kubernetes liveness probes failed. The pod restarted. Which triggered another cold start. Which failed the liveness probe again.
The fix was downloading the weights at build time:
FROM python:3.11-slim
WORKDIR /appCOPY requirements.txt .RUN pip install -r requirements.txt
# Download model weights during build, not at runtimeRUN python -c "from transformers import pipeline; pipeline('sentiment-analysis')"
COPY app/ .CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
The Docker image went from ~500MB to ~1.1GB. Build time went up. But cold start went from 45 seconds to under 2 seconds, and liveness probes stopped failing. The tradeoff was worth it, image size is a one-time cost, cold start latency affects every pod restart and scale event.
Engineering Decision 3: Kubernetes Deployment Configuration
The Helm chart deploys with 2 replicas by default, scaling up to 10 via HPA:
# helm/sentiment-api/templates/deployment.yamlreplicas: 2resources: requests: cpu: "250m" memory: "512Mi" limits: cpu: "500m" memory: "1Gi"livenessProbe: httpGet: path: /health port: 8000 initialDelaySeconds: 30 periodSeconds: 10readinessProbe: httpGet: path: /health port: 8000 initialDelaySeconds: 10 periodSeconds: 5
The HPA:
# helm/sentiment-api/templates/hpa.yamlapiVersion: autoscaling/v2kind: HorizontalPodAutoscalerspec: minReplicas: 2 maxReplicas: 10 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70
Two things I got wrong the first time. First, I set
initialDelaySeconds to 10 on the liveness probe, too short for a pod that needs to load weights and warm up FastAPI. It killed pods before they were ready. Bumping it to 30 fixed the restart loop.Second, I forgot resource requests entirely on the first deployment. Without resource requests, the HPA has nothing to calculate utilization against. It never scales. Setting explicit CPU requests is not optional if you want HPA to work.
Engineering Decision 4: Cross-Namespace Prometheus Metrics
The FastAPI app exposes metrics at
/metrics :from prometheus_client import Counter, Histogram, generate_latest
REQUEST_COUNT = Counter( 'inference_requests_total', 'Total inference requests', ['status'])
INFERENCE_LATENCY = Histogram( 'inference_latency_seconds', 'DistilBERT inference latency', buckets=[0.1, 0.25, 0.5, 1.0, 2.5, 5.0])
@app.get("/metrics")def metrics(): return Response(generate_latest(), media_type="text/plain")
Getting Prometheus to scrape these metrics required a
ServiceMonitor :apiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata: name: sentiment-api-monitor namespace: defaultspec: selector: matchLabels: app: sentiment-api endpoints: - port: http path: /metrics interval: 15s
It didn't work. Prometheus was running in the
monitoring namespace, the app was in default , and by default the Prometheus Operator only watches ServiceMonitors in its own namespace.The fix was patching the Prometheus CR:
kubectl patch prometheus kube-prometheus-stack-prometheus \ -n monitoring \ --type=merge \ -p '{"spec":{"serviceMonitorNamespaceSelector":{}}}'
An empty
serviceMonitorNamespaceSelector tells the operator to watch all namespaces. After that patch, metrics started appearing in Grafana within about 30 seconds.
The CI/CD Pipeline
Three stages, triggered on push to
main :jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Run tests run: | pip install -r requirements.txt pytest tests/ -v
build-push: needs: test runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Configure AWS credentials uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: ${{ secrets.AWS_ROLE_ARN }} aws-region: us-east-1 - name: Build and push to ECR run: | aws ecr get-login-password | docker login --username AWS --password-stdin $ECR_URI docker build -t $ECR_URI:${{ github.sha }} . docker push $ECR_URI:${{ github.sha }}
deploy: needs: build-push runs-on: ubuntu-latest steps: - name: Deploy to EKS run: | aws eks update-kubeconfig --name mlops-cluster --region us-east-1 helm upgrade --install sentiment-api ./helm/sentiment-api \ --set image.tag=${{ github.sha }}
Every deploy uses the commit SHA as the image tag. No
latest . Every deployment is traceable to a specific commit.
What the API Looks Like
curl -X POST http://<EXTERNAL-IP>/predict \ -H "Content-Type: application/json" \ -d '{"text": "This MLOps pipeline finally works and I am unreasonably happy about it"}'
Response:
{"label": "POSITIVE", "score": 0.9997}
The
/health endpoint returns {"status": "healthy"} what the liveness probe checks. The /metrics endpoint returns Prometheus-formatted text that Grafana ingests every 15 seconds.
What I Learned That Documentation Won't Tell You
CloudTrail is not just for auditing. When OIDC fails and you have no idea why, CloudTrail shows you the exact request AWS received and rejected. Reading it is faster than reading GitHub's OIDC docs for the fourth time.
Resource requests are load-bearing. HPA without resource requests does nothing. It's not an error, it silently refuses to scale. Set CPU requests first, then configure HPA.
Cold start kills liveness probes. If your pod needs more than 10 seconds to start, set
initialDelaySeconds accordingly. A pod that restarts every 10 seconds because of a premature liveness check is worse than no liveness check.Cross-namespace ServiceMonitor requires a Prometheus CR patch. This is documented, but not where you'd think to look. It's in the Prometheus Operator docs, not the ServiceMonitor docs.
The full code is at github.com/aboodi679/mlops-sentiment . Everything is there, Terraform, Helm chart, GitHub Actions workflows, and the FastAPI app with Prometheus instrumentation.
If you're building something similar and hit the OIDC numeric sub claim issue, check your CloudTrail logs first. The answer is always in the logs.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article