AWS Builder Center

Builder Center

How I Cut AWS EKS Infrastructure Costs by 38.1% with Terraform, Spot Nodes, and an Automated Python FinOps Gate (Plus Argo CD GitOps)

How I Cut AWS EKS Infrastructure Costs by 38.1% with Terraform, Spot Nodes, and an Automated Python FinOps Gate (Plus Argo CD GitOps)

A production-focused AWS EKS infrastructure project combining Terraform, GitOps, DevSecOps, and automated FinOps governance. Explore a four-layer architecture featuring Argo CD self-healing deployments, Kubernetes autoscaling, Prometheus/Grafana observability, and AWS cost optimization that reduces estimated monthly infrastructure spend by 38.1%.

Mohammed Armaan Kazi

BSC IT Student
Spinning up a "textbook" production Kubernetes cluster on AWS is deceptively expensive. Between the Amazon EKS control plane ( $73.00/mo ), redundant multi-AZ NAT Gateways ( $65.70/mo ), On-Demand EC2 worker nodes ( $60.74/mo ), and cross-AZ data transfer fees, an idle baseline cluster easily burns $240.00/month before serving a single real user.
Worse, once a team starts shipping Terraform changes via Pull Requests, a single unreviewed instance resize or extra NAT Gateway can blow up the monthly AWS bill overnight.
In  Part 1 of the CloudPulse series , I built a hardened, non-root FastAPI chaos microservice and a shift-left DevSecOps pipeline that eliminated 3 HIGH-severity container CVEs before publishing to GHCR.
In Part 2 (  cloudpulse-eks-gitops-infra  ), I built the production AWS landing zone, FinOps cost-governance gate, Argo CD GitOps controller, and Prometheus/Grafana observability stack to run that container in production—cutting baseline AWS spend by 38.1% ($148.60/mo vs. $240.00/mo).
Here is the complete breakdown of the 4-layer AWS architecture and how every piece is enforced automatically in CI/CD.


Architecture Overview: The 4 Production Layers

Why separate the application repository ( cloudpulse-devsecops-service ) from the infrastructure repository ( cloudpulse-eks-gitops-infra )?
Following the enterprise GitOps pattern prevents infinite CI/CD loops, isolates IAM permissions between app developers and platform engineers, and creates a clean audit trail where every infrastructure change is a standalone Git commit.
Our infrastructure repository wraps the verified container image in four layers:
  1. Layer 1 — AWS Cloud Networking, EKS Compute & IAM (Terraform): A custom  10.0.0.0/16  VPC across  ap-south-1a  and  ap-south-1b , a KMS-encrypted Amazon EKS v1.31 cluster, EC2 Spot/hybrid managed node groups, and scoped IAM roles.
  1. Layer 2 — Shift-Left FinOps & IaC Security Gates (GitHub Actions): Automated  terraform validate ,  Trivy IaC  scanning,  Kubeconform  strict schema checks, and a custom Python FinOps script that blocks any Pull Request exceeding a $250.00/month AWS budget.
  1. Layer 3 — Zero-Touch GitOps & Self-Healing (Argo CD + Kubernetes HPA): Continuous reconciliation watching  k8s/base/  with  selfHeal: true  and  prune: true , paired with a HorizontalPodAutoscaler scaling from 2 to 6 replicas during traffic surges.
  1. Layer 4 — Real-Time Flash-Sale Observability (Prometheus + Grafana): Custom PromQL telemetry tracking request rates, pod readiness, and flash-sale surge events every 5 seconds.


Layer 1: AWS VPC Networking & Single-AZ NAT Cost Optimization ( vpc.tf )

In  terraform/vpc.tf , we provision an isolated  10.0.0.0/16  Virtual Private Cloud in  ap-south-1  spanning two Availability Zones:
  • 2 Public Subnets ( 10.0.1.0/24  &  10.0.2.0/24 ): Tagged with  kubernetes.io/role/elb  for public ingress load balancers and routed to an Internet Gateway ( igw ).
  • 2 Private Subnets ( 10.0.10.0/24  &  10.0.20.0/24 ): Tagged with  kubernetes.io/role/internal-elb  where our EKS worker nodes run completely isolated from direct public internet access.
  • Zero-Trust Security Groups & VPC Flow Logs: The default VPC security group is stripped of all rules, control-plane ingress is restricted to HTTPS within the VPC CIDR, and all network traffic is logged to Amazon CloudWatch Logs with KMS encryption and a 30-day retention policy.


(AWS Console VPC Resource Map:  vpc-0a5c8076de9bb7174  provisioned via Terraform with public/private subnets across two AZs, route tables, and Internet Gateway routing.)

The FinOps Networking Optimization: Shared Single-AZ NAT Gateway

A standard multi-AZ NAT Gateway deployment costs $65.70/month in fixed hourly charges plus cross-AZ data processing fees. For our workload tier, we implemented a Single-AZ Shared NAT Gateway + Elastic IP architecture routing egress traffic (container image pulls from  ghcr.io  and AWS API calls) from both private subnets through a single optimized NAT tier.
This single architectural choice cuts fixed NAT costs by 50% (-$32.85/mo) and reduces cross-AZ data transfer overhead by 86.5% (-$22.11/mo).


Layer 2: Hardened Amazon EKS v1.31 & Least-Privilege IAM ( eks.tf  &  iam.tf )

Rather than relying on overly permissive default IAM policies,  terraform/iam.tf  provisions three strictly scoped, least-privilege IAM roles:
  1.  control-plane-role  ( eks.amazonaws.com ): Attached exclusively to  AmazonEKSClusterPolicy  and  AmazonEKSVPCResourceController .
  1.  worker-node-role  ( ec2.amazonaws.com ): Scoped strictly to worker node container runtime and CNI networking policies.
  1.  vpc-flow-log-role  ( vpc-flow-logs.amazonaws.com ): Scoped only to publish network audit streams to CloudWatch.


(IAM Console filtered to  cloudpulse-eks-prod : Three isolated least-privilege roles created by Terraform with zero wildcard permissions.)
In  terraform/eks.tf , the Amazon EKS v1.31 cluster ( cloudpulse-eks-prod ) and its managed node group enforce defense-in-depth at the infrastructure layer:
  • KMS Envelope Encryption: Kubernetes secrets are encrypted at rest using a dedicated AWS KMS Customer Managed Key with automatic yearly key rotation enabled.
  • Private-Only API Endpoint & Control Plane Logging: All 5 EKS control-plane audit and diagnostic log streams are enabled by default.
  • Hardened EC2 Launch Templates: Worker nodes enforce IMDSv2 ( http_tokens = "required"  to block SSRF metadata attacks), run on encrypted  gp3  EBS volumes with no public IPs, and utilize EC2 Spot ( t3.medium ) capacity to cut compute spend by 60% (-$36.44/mo).


Layer 3: Pre-Merge FinOps Governance Gate in GitHub Actions

Most teams only discover cloud cost drift when the AWS invoice arrives at the end of the month. To shift FinOps left, I wrote a zero-dependency Python 3 cost-governance scanner ( scripts/finops_cost_guard.py ) that runs on every Pull Request inside  .github/workflows/iac-gitops-ci.yml .
Alongside Terraform validation ( terraform fmt  &  validate ), Trivy IaC security scanning (blocking any  HIGH  or  CRITICAL  misconfigurations), and Kubeconform Kubernetes v1.31 schema validation, the FinOps guard parses  terraform/*.tf  and calculates the monthly run rate before a single resource can be merged:
AWS Resource
Architecture Configuration
Baseline Cost
Optimized Cost
FinOps Savings
Amazon EKS Control Plane
EKS v1.31 ( cloudpulse-eks-prod )
 $73.00 
 $73.00 
 $0.00 
AWS NAT Gateway
Single-AZ Shared NAT vs. Multi-AZ
 $65.70 
 $32.85 
 -$32.85 (-50.0%) 
EC2 Worker Nodes
2x  t3.medium  (Spot / Hybrid)
 $60.74 
 $24.30 
 -$36.44 (-60.0%) 
Amazon EBS Storage
100 GB Encrypted  gp3  (2x 50GB)
 $8.00 
 $8.00 
 $0.00 
CloudWatch Logs
VPC Flow Logs + EKS Audit (30d)
 $6.00 
 $6.00 
 $0.00 
AWS KMS Key
Secrets Envelope Encryption
 $1.00 
 $1.00 
 $0.00 
Network Data Transfer
Cross-AZ Optimization + Shared EIP
 $25.56 
 $3.45 
 -$22.11 (-86.5%) 
TOTAL MONTHLY SPEND
Region:  ap-south-1 
 $240.00 
 $148.60 
 -$91.40 (-38.1%) 
If a developer opens a Pull Request that pushes the estimated monthly spend past our $250.00/mo policy ceiling,  finops_cost_guard.py  exits with code  1  and blocks the merge button. At  $148.60/mo , we maintain $101.40/mo (40.6%) of budget headroom to absorb HPA auto-scaling bursts during peak traffic.


Layer 4: Argo CD GitOps Delivery, HPA Auto-Scaling & Flash-Sale Observability

Once infrastructure passes all four CI gates, Argo CD ( argocd/application.yaml ) takes over continuous delivery. Nobody runs manual  kubectl apply  commands against production:
syncPolicy:

  automated:

    prune: true      # Automatically deletes resources removed from Git

    selfHeal: true   # Reverts manual kubectl edits back to the Git-declared state

  syncOptions:

    - CreateNamespace=true



(Argo CD Application Tree for  cloudpulse-production : Synced to  main  with automated pruning and self-healing active across the Namespace, Service, Deployment, ReplicaSet, Pods, and HPA.)

Self-Healing & Dynamic Flash-Sale Scaling ( k8s/base/ )

Inside  k8s/base/deployment.yaml , our pods run with strict Kubernetes security contexts ( runAsNonRoot: true ,  runAsUser: 10001 ,  readOnlyRootFilesystem: true ,  drop: ["ALL"]  capabilities, and  seccompProfile: RuntimeDefault ).
Liveness and readiness probes hit  GET /healthz  on port  8000  every 10 seconds. If a container returns HTTP 503s for 3 consecutive checks during an outage simulation ( /simulate-outage ), the kubelet terminates and replaces the pod with zero downtime while healthy replicas serve traffic.
Meanwhile, our HorizontalPodAutoscaler ( hpa.yaml ) monitors CPU utilization against a 60% target threshold, scaling smoothly between  minReplicas: 2  and  maxReplicas: 6 :


(Cluster verification: 3/3 nodes Ready, Argo CD application Synced & Healthy, and HPA actively monitoring CPU utilization across replicas.)

Visualizing Traffic Surges in Real Time (Prometheus + Grafana)

Every pod includes Prometheus scrape annotations ( prometheus.io/scrape: "true" , port  8000 , path  /metrics ). When we trigger a flash-sale traffic spike, Prometheus scrapes the custom counters and Grafana ( monitoring/grafana-cloudpulse-dashboard.json ) visualizes the surge across a 5-second rolling dashboard:


(Grafana Production Telemetry Dashboard: Tracking 32.6K total HTTP requests, 51 flash-sale surge events, and  /api/v1/checkout  traffic ramping up to ~300 req/s while maintaining  HEALTHY (UP)  service availability.)

Try It Yourself (Full Source Code)

To avoid burning money on idle AWS control planes outside of live benchmarks, you can run the entire 4-gate Terraform, Trivy, Kubeconform, and FinOps validation pipeline locally—or spin up the full topology on a local  k3d  cluster with Argo CD and Grafana at $0 cost:

# kubernetes# terraform# devops
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article