The zone that was slow but healthy
Say you sell sweets and pickles online from India. Your customers are mostly Indians abroad, in the US and Australia, and they order around festivals. Your application runs across a few AWS Availability Zones so it sits close to them.
Then Diwali arrives. Traffic goes up several times over and one zone starts getting slow.
It is not down. Health checks are green. Endpoint counts are balanced. From the console everything looks fine. But customers hitting that zone are waiting seconds for pages to load, and some of them leave.
That is a brownout, and it is the failure we spent four days on.
Why AWS's routing cannot see it
Amazon ECS Service Connect ships zone aware routing using Envoy sidecars. It prefers endpoints in the caller's own Availability Zone and uses endpoint distribution to decide when traffic should cross zones. When endpoint distribution represents load well, this works, and it saves real money on cross-AZ transfer.
The inputs are endpoint counts and health checks. Neither of those moves when a zone simply becomes slow. So the routing has no signal telling it anything is wrong.
You cannot fix this by tightening health checks either. A health check is binary. It removes the whole zone the moment it trips, when the problem only calls for moving part of the traffic.
What we built
ZoneHeal is a reinforcement learning policy that reads three things every minute: how full a zone is, how its recent latency compares with the SLO, and how much room the other zones have. Five utilisation bands, three latency bands, three spare-capacity bands, so 45 states. Each state picks how much traffic to move: 0, 10, 25 or 50 percent.
We trained it on the Huawei Public Cloud Trace 2025 dataset, 31 days from Region 2, treating its four clusters as Availability Zones. Real arrivals, real pod counts, real service times. We inject the failures into real traffic.
What it learned is a table you can read:
latency: FINE | NEAR SLO | OVER SLO spare: lo mid hi | lo mid hi | lo mid hi
u0 quiet 0 0 1 | 0 0 0 | 0 3 3u1 light 0 0 0 | 0 0 0 | 0 1 3u2 moderate 0 0 0 | 0 0 0 | 0 2 3u3 busy 2 0 0 | 2 0 0 | 1 0 0u4 full 2 0 0 | 2 0 0 | 3 0 0
Quiet and fast, stay local, because moving traffic costs money for nothing. Slow but not busy, move up to half of it, which is the brownout case. Busy, move traffic before latency crosses the line. No room elsewhere, do not pay to move.
We did not write any of those rules.
Results
Held out test days, five seeds:
Condition | Agent | p99 (ms) | ₹/hr | % local |
|---|---|---|---|---|
Brownout | ZoneHeal | 2221.7 | 69.0 | 72.4 |
Brownout | AWS zone aware | 2910.5 | 21.8 | 91.8 |
Normal | ZoneHeal | 728.9 | 9.4 | 95.6 |
Normal | AWS zone aware | 728.8 | 21.8 | 91.8 |
When nothing is wrong, the two are tied on latency within a millisecond. When a zone degrades, endpoint counts do not move, so AWS keeps 91.8% of traffic pointed at the slow zone and ZoneHeal is 24% faster at p99.
The tie is the part we care about. The policy learned to stay quiet.
The stack
ECS on Fargate across three AZs in Sydney, with tasks pinned to subnets so each zone is a real AZ. Service Connect for discovery. S3 holds the trained policy, loaded by each router at startup. Every request writes a structured log line that CloudWatch turns into metrics: latency, cross-AZ bytes, and whether the request kept the 200ms promise.
The learned policy never runs unsupervised. A CloudWatch alarm on p99 needs three consecutive breaching minutes; EventBridge fires a Lambda, and the routers poll that alarm and fall back to a fixed rule within ten seconds. If the CloudWatch call itself fails, they change nothing rather than deciding from missing data.
And when the routing spends money, Bedrock turns the measurements into plain English and SES emails it:
Zone C slowed to about 745ms while its health checks were still passing. The system moved half of its traffic to the other zones, which were responding normally. The transfer cost ₹0.0005.
Every number in there comes from our code. The model only writes the sentences.
What it is not
One dataset, one region. The AWS deployment proves the policy runs as real software, not production-scale performance. The workload is a load generator and the failure is an endpoint we control. Half the training iterations included a brownout, so the policy was trained for the condition it is evaluated on. The cost figures at demo scale are fractions of a rupee.
We are not replacing AWS's routing. We were asking whether the locality decision should be able to see what the system is actually doing, and what that would be worth.
Built for First Commit, AWS × WeMakeDevs, September 2026, by Aarya Upadhya and Anshull M Udyavar.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article