AWS Builder Center
Don't Scale Blindly, Route Intelligently: ML-Driven Traffic Optimization for AWS ALB

Don't Scale Blindly, Route Intelligently: ML-Driven Traffic Optimization for AWS ALB

Traffic Router is a multi-objective optimization layer on AWS ALB. It predicts backend failure risk with ML and rewrites ALB traffic weights in real time, so traffic moves away from degrading backends before users feel it.

THE PROBLEM AWS ALB and Auto Scaling react to single metrics. Traffic keeps hitting a degrading backend until health checks fail, CPU-only scaling wastes money, and recovery starts after users are already hurt. No AWS-native mechanism optimizes latency, cost and failure risk together, in real time.
WHAT WE BUILT Traffic Router is a multi-objective optimization layer on top of AWS ALB. Every ~2 seconds it:
  1. Polls each backend's P95 latency, error rate and CPU
  1. Scores failure risk with an ML model, calibrated by a Q-learning agent
  1. Solves a constrained optimization for the ideal traffic split
  1. Pushes new weights to the ALB through the ELBv2 API
Objective: minimize α·latency + β·cost + γ·risk + δ‖w‖², with Σw = 1 and 0 ≤ w ≤ 1. Routing becomes a cost/performance/reliability trade-off, not a threshold switch.
WHERE AWS FITS
  • Application Load Balancer is the router: no proxy in the data path
  • Weighted target groups (one per backend) for live traffic shifting, plus native round_robin and least_outstanding_requests as baselines
  • 3 EC2 backends + 1 EC2 engine in a VPC (ap-south-1)
  • boto3 + ELBv2 (modify_listener) for real-time control
  • IAM instance role and security groups for least-privilege access
THE ML MODEL Trained on 1,350 telemetry samples from our own AWS load tests (latency, error rate, CPU, normalized cost), exported as risk_model.pkl and retrained online as new telemetry arrives.
RESULTS Locust on the live ALB, 100 users, 5 min per mode:
  • Round robin: 190,965 requests, 0 failures, P95 49 ms
  • Least outstanding: 191,222 requests, 0 failures, P95 48 ms
  • Traffic Router: 191,040 requests, 0 failures, P95 49 ms 573K requests, zero failures, and the optimizer runs live in the loop with no added latency versus native ALB algorithms.
WHAT FOUGHT BACK
  • A pure linear objective sent 100% of traffic to one "best" backend. A quadratic smoothing term made it strictly convex: healthy backends share load, a degraded one drops toward zero.
  • Moving from Docker to real AWS deleted our router entirely. Weighted routing needs one target group per backend, so switching modes means swapping the listener action.
  • Our first model scored a perfect 100%, which taught us to treat that as a red flag and keep retraining on live telemetry.
WHAT WE LEARNED The ALB is programmable far beyond the console, and convex optimization gives routing decisions that are explainable.
NEXT Failure-injection benchmarks, CloudWatch observability, and scaling triggers only when healthy capacity runs out.
Don't scale blindly. Route intelligently.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article