Microservices resilience and debugging on AWS
Microservices on AWS re-reads AWS's 2017 whitepaper — decomposition, discovery, data ownership. Service mesh on AWS settles which mesh, if any, this application runs. This page is about the two things between those decisions: keeping a set of services up when parts of it fail, and finding out what happened when they do not.
It is built from five posts on Jimmy Song's blog, mostly translations:
| post | origin | what it contributes |
|---|---|---|
| Service Mesh 与 Spring Cloud 应用的互通、互联 (microservices HA practice) | Baidu Cloud CNAP team, translated | a hybrid of Spring Cloud and a mesh, Consul as a highly available registry, rate limiting, circuit breaking, routing |
| Microservices Communication and Governance Using Service Mesh | InfoQ virtual panel, April 2018: Matt Klein (Envoy), Oliver Gould (Linkerd), Varun Talwar (Istio), Yuri Shkuro (Jaeger), Dan Berg, Priyanka Sharma, Lachlan Evenson | what the people who built the meshes said they were for |
| How to debug microservices in Kubernetes — proxy, sidecar or service mesh | Jimmy Song, 2021, now archived | Telepresence, Nocalhost and a mesh dashboard as three debugging models |
| Manage microservices traffic using Istio, Istio is not just for microservices | 2018 | early Istio framing, already used in Istio on EKS |
The registry must never take calls down with it
The Baidu team's first rule is the one most designs miss — in translation, the registry must not, through its own failure, affect calls between services. A registry outage should stop changes — new instances, removed instances — from propagating. It should not stop services that already know each other from talking.
Their Consul setup shows the two halves of that:
| half | in the post | on AWS |
|---|---|---|
| the registry itself survives node loss | 3 Consul servers across 3 machine rooms; Raft tolerates n failures with 2n + 1 nodes; 5 is the useful maximum, because more servers slow every write | AWS Cloud Map is a regional managed service, and Kubernetes service discovery is the EKS control plane — there is no quorum for you to size |
| a client survives the registry being down | the SDK keeps a local cache of instances and keeps using it | Cloud Map's DiscoverInstancesRevision returns only a revision number, so a client can poll that cheaply and call DiscoverInstances only when the revision moves — and keep its last answer if Cloud Map cannot be reached |
On EKS, the in-cluster version of the same rule is CoreDNS: a cluster whose DNS falls over takes every name lookup with it, which is the Zalando outage in one sentence. The registry is managed; the client's behaviour when it cannot reach it is still yours.
The post's tenant-isolation trick — filtering a shared registry per tenant in a proxy layer — has a cleaner AWS shape: separate Cloud Map namespaces, or separate VPC Lattice service networks, with IAM deciding who may discover what.
Rate limiting: four algorithms, and the service that runs each
The post lists the algorithms every rate-limiting design picks from. On AWS, each one is already implemented somewhere:
| algorithm | how it behaves | AWS service that implements it |
|---|---|---|
| fixed window | count per window, reset at the boundary; lets twice the limit through across a boundary | — avoid it |
| sliding window | count over the last N seconds | AWS WAF rate-based rules: a Limit per aggregation key over an EvaluationWindowSec look-back |
| token bucket | tokens refill at a rate up to a burst; a request spends one | API Gateway throttling: rateLimit and burstLimit, per stage, method or usage plan |
| leaky bucket / queue | requests wait in a queue and drain at a fixed rate | SQS in front of the worker — the post's "削峰填谷", load levelling |
And the three strategies when a limit is hit map the same way: reject (API Gateway's 429, WAF's block), delay (a queue), and privilege (usage plans with different limits per API key). What no AWS service does is rate-limit between two pods in a cluster; that is a mesh feature, or code.
Circuit breaking: the part no AWS load balancer does for you
The post compares Hystrix and Istio's
outlierDetection and finds two differences that still matter. Istio has no half-open state — an ejected host comes back after a timeout that grows with each ejection — and it breaks per host, removing a failing instance from the pool, where a classic breaker trips for the whole dependency and returns a fallback.Neither exists in an AWS-managed service:
| capability | ALB | VPC Lattice | the application |
|---|---|---|---|
| remove an instance that fails health checks | yes | yes | — |
| shift traffic away from an instance that is answering, but badly | the anomaly mitigation option of weighted-random routing reduces traffic to targets that look anomalous; check the target group attributes in current ALB documentation | no | yes |
| trip a breaker for a whole dependency and serve a fallback | no | no | yes — only here |
So on an AWS-only stack, the circuit breaker with a fallback lives in the application: Resilience4j in Java (Hystrix has been in maintenance mode since 2018), or the equivalent in your language. The load balancer's health checks are the coarse layer underneath, not a replacement.
Retries: the warning every panelist gave
The 2018 panel agreed on what a mesh does well — moving retries, timeouts and breakers out of every codebase into one implementation — and Oliver Gould added the warning that matters most: retries can make an outage worse, through long-running retry loops that hold resources and cascade (paraphrased from the panel's Chinese translation). The post's three fault-tolerance modes name the choice:
| mode | meaning | when |
|---|---|---|
| failover | retry on another instance, a bounded number of times | idempotent reads |
| failfast | fail immediately, no retry | writes, anything not idempotent |
| failback | queue the failed request and retry asynchronously | work that can be late but not lost — on AWS, an SQS queue with a dead-letter queue |
On AWS the retry discipline is already built into one place most services use: the AWS SDKs'
standard and adaptive retry modes draw retries from a retry quota, so a client that keeps failing runs out of retries instead of multiplying load on a struggling dependency. Your own service-to-service calls deserve the same thing: a retry budget, not a retry count. Microservices on AWS works the arithmetic — three retries at each of three layers is twenty-seven calls.What the 2018 panel got right, and what changed
| the panel said | still true? | on AWS in 2026 |
|---|---|---|
| a mesh's hardest problems are networking and observability (Klein) | yes | — |
| adopt a mesh incrementally; trying to do everything at once is the failure (Gould, Berg, Talwar) | yes | Istio on EKS and the gateway-first path in Service mesh on AWS |
| an ESB is centralized and a mesh is distributed; the goals are similar | yes | — |
| a mesh works well on Kubernetes and poorly elsewhere (Talwar, Shkuro) | mostly | Istio ambient and VPC Lattice both reach beyond pods |
| mTLS and identity become feasible at scale through the mesh (Klein, Gould) | yes | SPIFFE in the mesh, or VPC Lattice auth policies on IAM identity |
| Conduit, OpenTracing | renamed or retired | Linkerd 2; OpenTelemetry |
| AWS's own mesh | App Mesh reached end of support on 30 September 2026 | VPC Lattice for service-to-service routing and IAM auth; it is not a mesh — no retries, breakers or rate limits between services |
That last row is the practical consequence of this page. Moving off App Mesh to VPC Lattice keeps routing, discovery and identity managed, and moves retries, timeouts and circuit breaking back into the application — the place the 2018 panel spent its time arguing they should leave.
Debugging: proxy, sidecar or mesh, and the AWS-only version of each
The debugging post names three models, by tool:
| model | tool in the post | what it gives you | on AWS, services only |
|---|---|---|---|
| proxy — your laptop joins the cluster network | Telepresence: a VPN plus volume and environment forwarding | run one service locally against the others in the cluster | Session Manager port forwarding (AWS-StartPortForwardingSessionToRemoteHost) to a service's address, with no VPN and no open port; kubectl port-forward for one pod |
| sidecar — a dev container replaces the pod's | Nocalhost: a VS Code plugin swaps in a dev image and syncs code | edit and run inside the real pod | no AWS service; it is a tool, and it needs cluster access you should scope to a dev namespace |
| mesh — every pod reports | Istio with Kiali | a live service graph | CloudWatch Application Signals service map and SLOs, X-Ray traces |
Two cautions from the post itself. Its Nocalhost walkthrough logs in to the web console as
admin@admin.com / 123456, the published default; change it before the console is reachable by anyone else. And it binds the plugin's ServiceAccount to the admin ClusterRole; on EKS, give a developer an access entry scoped to one namespace instead.The mesh model is the only one that also helps after the fact. A proxy and a sidecar let you reproduce a bug; a service map and traces tell you which service caused the incident you did not reproduce. On an AWS-only stack that is Application Signals or X-Ray, fed by OpenTelemetry — the same signals the SRE page alarms on.
What this repository should take from it
| concern | where it lives here | the gap |
|---|---|---|
| discovery | Kubernetes Services and CoreDNS | CoreDNS replicas and memory are defaults — see Kubernetes troubleshooting on EKS |
| rate limiting | none | an API Gateway or WAF rate-based rule in front of the ALB, before the first load test |
| circuit breaking with fallback | none | the app makes no downstream calls yet; when it does, a breaker in code, not in the load balancer |
| retries | the AWS SDK's defaults, for AWS calls | a budget for service-to-service calls when they exist |
| debugging | kubectl, make deploy | Session Manager port forwarding; Application Signals once there is more than one service |
Verified, and not
From the AWS API models (botocore 1.43.107): Cloud Map's
DiscoverInstancesRevision, WAF rate-based statements' Limit and EvaluationWindowSec, API Gateway's rateLimit and burstLimit, and that VPC Lattice's API has listeners, rules and target groups but no rate-limit or circuit-breaker resource.From the posts: every claim attributed to the Baidu team, the panelists and the debugging guide. The panel is from April 2018 and the HA practice from about 2020; their tools are dated, and the table above says which.
From AWS documentation and announcements, not checked against an API: App Mesh's end of support on 30 September 2026, ALB anomaly mitigation under weighted-random routing, the AWS SDK retry quota's behaviour, the WAF evaluation window values, and Application Signals' EKS support. Check them against current documentation before relying on any one.
Not run. No service, rule or registry was created for this page.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article