Mesh and networking posts, read against AWS
Jimmy Song's site, cloudnative.jimmysong.io, carries hundreds of Chinese posts. Most are translations of English originals, and some are written first in Chinese. This page reads 29 of them, from 2017 to 2025. They range from Matt Klein's 2017 case for a universal data-plane API, through the 2018 Istio 0.8 crop (v1alpha3 routing, Mixer adapters, soft multi-tenancy, cert-manager with Let's Encrypt), to 2023–2025 pieces on BGP load balancers, KEDA, bare metal, Istio 1.26 and LLMs.
Each post answers one of about ten questions that a platform team still asks. "Who is the real client?" "How do two services prove who they are?" "Who configures the proxies?" The page groups the posts by those questions rather than by date, and asks of each: what AWS service now answers it, and does the post's answer still hold?
This page does not repeat what three other pages already cover:
- Service mesh and gateway on AWS : VPC Lattice as the target, Istio evaluated,
istioctl proxy-statusfirst when routing goes wrong. - Istio on EKS : iptables chains, the 15017 webhook port, istio-cni, UID 1337, revisions, ambient mode, and a table of what older sources get wrong.
- Microservices resilience on AWS : rate limiting, circuit breaking, retries and the 2018 mesh panel.
The Kubernetes Handbook read against AWS already has an eight-row table of some of these posts. This page goes further on those eight and adds the other 21. One correction to that table: the handbook row titled "PROXY protocol spec; enabling it on an NLB and the Istio gateway" is two posts. One is a translation of the HAProxy specification. The other is a December 2020 field note about an NLB.
How it was read. All 29 posts were read in full, as scraped Markdown under
k8s/slurps/cloudnative.jimmysong-blog-*.md. They are quoted only in short translated phrases, with attribution. The rest is summary in this page's own words.Four kinds of checks were made, and the page says which applies to each claim:
- Dry-run. Every YAML snippet marked broken or fixed was sent with
kubectl apply --dry-run=server --validate=strictto a kind lab cluster. The lab runs Kubernetes 1.33.13 with CRDs from Istio 1.30.3, cert-manager 1.16.2 and KEDA 2.20.2. Server-side dry-run runs admission and validation but persists nothing. Every command marked broken was run with istioctl 1.30.3 or kubectl 1.33.13. - API model. AWS API names, attribute keys and enums were read from the botocore 1.43.107 models.
- Proto. Istio's
gatewayTopology.proxyProtocolfield was read frommesh/v1alpha1/proxy.protoin istio.io/api (v1.31 alpha module). - Published sources. Dates were checked against AWS, Istio, Cilium and KEDA documentation and announcements fetched on 7 October 2026.
Nothing was deployed to AWS, and no load balancer was created.
The posts at a glance
"Still true?" means: does the post's central argument hold on AWS today? The YAML is judged separately, in the corrections table further down.
| post | year | what it argued | on AWS now | still true? |
|---|---|---|---|---|
| PROXY protocol specification (HAProxy, translated) | 2010–2020 | prepend the client's address to a TCP stream. The receiver must accept only this protocol on that port, or clients can forge their address | NLB target groups: proxy_protocol_v2.enabled (v2 only). AWS's own VPC-endpoint TLV sits in the spec's custom range | yes. The security rule is the part people skip |
| Enabling PROXY protocol on an NLB and the Istio ingress gateway | Dec 2020 | annotate the Service, add an EnvoyFilter for the listener filter, add another for XFF trust | preserve_client_ip.enabled often makes PROXY protocol unnecessary; Istio has a first-class gatewayTopology.proxyProtocol | the goal, yes. Both EnvoyFilters are now rejected |
| mTLS in Kubernetes: options and recommendations (John Howard) | 2024 | sidecar meshes carry most production mTLS; ambient is cheaper; no CNI does mutual TLS; doing it yourself is a last resort | ALB mTLS at the edge; ECS Service Connect TLS; Istio ambient on EKS; Lattice uses IAM SigV4, not mTLS | yes |
| Kubernetes service mesh (Alen Komljen) | Jan 2018 | Services and Ingress are not enough once you need canaries and per-request telemetry | Lattice HTTPRoute weights; ALB weighted target groups | yes, but "Istio supports Kubernetes and Nomad" no longer does |
| Service mesh at Cookpad | May 2018 | Envoy on ECS, an in-house control plane serving xDS v1 JSON from S3, Prometheus and X-Ray | App Mesh did this as a service, then ended on 30 Sep 2026; ECS Service Connect is the ECS successor | the motive, yes. xDS v1 and v2 are gone |
| Tracing gRPC with Istio (Aspen Mesh) | 2018 | the mesh starts spans, but the application must copy trace headers onward; grpc-gateway drops B3 headers | ADOT and X-Ray. The X-Ray SDKs have been in maintenance mode since 25 Feb 2026 | yes. The libraries it uses are archived |
| The universal data plane API (Matt Klein) | Sep 2017 | xDS should become a common API between any proxy and any control plane; value will move to the control plane | AWS sells the control plane and hides the data plane (Service Connect, Lattice). gRPC proxyless xDS exists, but not on AWS | half. xDS won, but no AWS service exposes it |
| Istio security notes (Fleeto) | 2018 | Policy plus DestinationRule for mTLS; Mixer denier and listchecker for allow and deny; ServiceRole for RBAC | PeerAuthentication and AuthorizationPolicy | the model, yes. Every resource in it is gone |
| Ordering containers within a Pod (kubexit) | 2023 | a shared volume plus a wrapper binary to order container start and stop | native sidecars: initContainers[].restartPolicy: Always, GA in Kubernetes 1.33 | no. Use the built-in feature |
| Istio 1.26 released; 1.23 end of life | May 2025 | Gateway API customisation, ambient TCPRoute, ClusterTrustBundle; 1.23 ended 16 Apr 2025 | supported now: 1.31, 1.30 and 1.29, and 1.29's end of life is expected on 12 Oct 2026 | the release cadence, yes |
| A new, more efficient proxy model (eBPF and Cilium) | May 2018 | an eBPF-based per-node data plane could replace per-pod proxies | the idea came back as Istio's ztunnel and Cilium's mesh; on EKS, Cilium replaces the VPC CNI | the direction, yes. The RouteRule in it is gone |
| Pilot xDS interface notes | Jun 2018 | how to query Pilot's v1 REST and v2 debug endpoints | istioctl proxy-status, proxy-config, x internal-debug | no. Every endpoint has moved |
| The rise of the Istio service mesh (Red Hat) | 2018 | Istio answers each fallacy of distributed computing | "transport cost is zero" is now a literal cross-AZ charge | the table, mostly; that row, no |
| linkerd2-proxy and the destination service | c. 2019 | the Rust proxy resolves endpoints over a gRPC stream and balances with power-of-two-choices on peak EWMA | ALB offers least_outstanding_requests and weighted_random with anomaly mitigation, not EWMA | yes, as a description of Linkerd |
| Soft multi-tenancy in Istio | 2018 | one control plane per tenant, made by sed-renaming istio-system, plus RBAC | Istio revisions and discoverySelectors; EKS access policies scoped to namespaces | the conclusion, yes ("soft" only). The method, no |
| Introducing the v1alpha3 routing API | 2018 | Gateway, VirtualService, DestinationRule and ServiceEntry replace RouteRule, DestinationPolicy and EgressRule | networking.istio.io/v1, and Gateway API, which is what the Lattice controller reads | the design, yes. v1alpha3 is still served |
| Lightweight service mesh at UCloud | 2018 | take Pilot out of Istio, run it on etcd without Kubernetes, use explicit proxy ports instead of iptables | ECS Service Connect is AWS's "mesh without Kubernetes"; Lattice needs no proxy at all | the trade-off, yes |
| Istio, Envoy, cert-manager and Let's Encrypt | Jan 2018 | an HTTP-01 workaround that allows one certificate per gateway | cert-manager 1.x with Gateway credentialName. ACM has run a public ACME endpoint since 6 Jul 2026 | no. Nearly every line is obsolete |
| GitOps for Istio (Weaveworks) | 2018 | the cluster pulls state from version control; no kubectl in the pipeline; canaries via Flux and Prometheus | Weaveworks closed in Feb 2024; Flux continues at the CNCF; EKS runs Argo CD as a managed capability | the principle, yes |
| Linkerd as a service mesh proxy at WePay | 2018 | Linkerd 1 (JVM, Finagle) chosen over Envoy, partly for its DaemonSet per-node mode | Linkerd 1 is retired. The per-node idea returned as ztunnel. WePay ran on GKE | historical |
| Building Istio with Minikube in a container and Jenkins | Jan 2018 | privileged Docker-in-Docker with minikube --vm-driver=none for CI | kind in CodeBuild (privileged mode), or a throwaway EKS cluster | the need, yes. The recipe, no |
| LoadBalancer services with BGP and Cilium | Oct 2023 | Cilium announces LoadBalancer IPs to a router over BGP; all six nodes advertise, giving ECMP | inside a VPC: NLB through the AWS Load Balancer Controller. VPC Route Server accepts BGP, but installs one route per prefix | on premises, yes. In a VPC, no |
| Addressing attack vectors with Istio (Aspen Mesh) | Jun 2018 | edge control, mTLS, and Mixer policy for application-layer rules | WAF and ALB mTLS at the edge; Lattice auth policies or Istio AuthorizationPolicy inside | yes. Mixer is gone |
| Istio routing on OpenTracing baggage | 2018 | put user-agent into baggage once, then route on it in every downstream service | W3C baggage header; Lattice and ALB match headers only on the hop they see | yes |
| Why the latest advances in OpenTelemetry matter | 2023 | OTTL, logs, auto-instrumentation, semantic conventions, one Collector | ADOT Collector; CloudWatch's OTLP endpoint; X-Ray reads OpenTelemetry natively | yes |
| Does Kubernetes perform better on bare metal than on VMs? (Gcore) | 2023 | bare metal is 2–5× faster across CPU, RAM, storage and network | 160 .metal instance types in the EC2 model; Nitro offloads I/O | the direction, maybe. The method does not support the numbers |
| KEDA versus vanilla Kubernetes autoscaling (Kedify) | 2023 | the HPA is limited to CPU and memory and cannot scale to zero; KEDA can | KEDA with aws-sqs-queue and aws-cloudwatch scalers and Pod Identity | scale-to-zero, yes. "CPU and memory only", no |
| End of an era: Weaveworks closes | Feb 2024 | a GitOps pioneer runs out of money; Flux should survive | Flux is a graduated CNCF project; AWS chose Argo CD for its managed capability | yes |
| A gentle introduction to LLMs for platform engineers (Christian Posta) | 2025 | an LLM is an HTTP API that bills by the token; put a gateway in front of it | Amazon Bedrock; API Gateway usage plans; Gateway API Inference Extension in-cluster | yes. The sample model is retired |
1. Who is the client? Source IP through NLB and ALB
Two posts carry this question. The PROXY protocol specification explains the mechanism. The December 2020 field note wires it through an NLB into an Istio gateway, and adds a translator's caveat that the content had not been re-verified.
The specification's most important sentence is about security, not the format. The receiver must be configured to expect the header, and must not guess whether it is present. Otherwise any client can claim any address. Its revision history also credits Andriy Palamarchuk of Amazon.com, in 2017, with the checksum, no-op and TLS-related TLV types and the reserved type ranges. AWS later used that custom range for its own TLV, which carries the VPC endpoint ID for PrivateLink traffic.
On AWS, PROXY protocol is often not the right tool. The ELBv2 API model documents these target-group attributes:
| attribute (ELBv2 model) | what the model says | when you want it |
|---|---|---|
| preserve_client_ip.enabled | default off for IP targets on TCP or TLS; on otherwise; cannot be turned off for UDP | turn it on for target-type: ip and the pod sees the client's address with no header parsing. Security groups and the return path must allow it |
| proxy_protocol_v2.enabled | v2 only, default false | when the target needs the address and client IP preservation cannot be used: PrivateLink, or hairpin traffic from a target to its own NLB |
| ALB routing.http.xff_header_processing.mode | append (default), preserve, remove | HTTP behind an ALB. Leave it on append and trust only the rightmost hops you control |
| ALB routing.http.xff_client_port.enabled | default false | when you also need the client's source port |
Combined with Kubernetes' own setting, the choices look like this:
| path | what the pod sees as the source | notes |
|---|---|---|
| NLB, instance targets, externalTrafficPolicy: Cluster | a node IP | kube-proxy SNATs on the second hop |
| NLB, instance targets, externalTrafficPolicy: Local | the client | uneven spread; nodes with no ready pod fail the health check |
| NLB, IP targets, preserve_client_ip.enabled=true | the client | the usual answer on EKS with the VPC CNI |
| NLB, IP targets, preservation off, proxy_protocol_v2.enabled=true | the NLB's address on the socket; the client in the header | the receiver must parse PROXY v2 on that port, and only that port |
| ALB, any target type | the ALB's address on the socket; the client in X-Forwarded-For | the gateway must trust exactly one hop |
What the 2020 post gets wrong now:
- The Service annotations.
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"is the older in-tree path. With the AWS Load Balancer Controller, useaws-load-balancer-type: externalplusaws-load-balancer-nlb-target-type: ip. Set the attributes above throughservice.beta.kubernetes.io/aws-load-balancer-target-group-attributes. These annotation names come from the controller's documentation and were not run here. - Both EnvoyFilters are rejected. The XFF patch uses the Envoy v2 type URL. Istio 1.30.3's validation webhook denied it in the dry-run: "referenced type unknown (hint: try using the v3 XDS API)".
- The listener-filter EnvoyFilter is unnecessary. Istio's gateway topology now has a
proxyProtocolfield (it is inTopologyinmesh/v1alpha1/proxy.proto). Set it on the gateway Deployment:proxy.istio.io/config: '{"gatewayTopology":{"proxyProtocol":{}}}'. - The two trust counts contradict each other. The Service annotation sets
numTrustedProxies: 2, while the EnvoyFilter setsxff_num_trusted_hops: 1. An NLB is a layer-4 hop and adds noX-Forwarded-For, so behind an NLB alone the count is 0. Behind an ALB it is 1. Set too high, the gateway trusts a value the client wrote. - The author's own caveat stands. The post says it does not handle forged
X-Forwarded-For. Neither does the configuration it prints.
2. How do two services prove who they are?
Five posts touch this question: Howard's mTLS options, Fleeto's Istio 0.8 security notes, Aspen Mesh's attack vectors, the cert-manager and Let's Encrypt walkthrough, and WePay's per-service certificates. Howard's 2024 post is the one to keep. He sorts the choices into do-it-yourself, sidecar mesh, node-proxy (ambient) and CNI. He then makes the point that matters most when someone says "we already encrypt": Cilium's mutual authentication and the WireGuard and IPsec options in CNIs are not TLS. In his words, that is not a semantic quibble.
On AWS, the options split the same way, but no single service covers every hop:
| hop | AWS-managed option | is it mutual TLS? | checked |
|---|---|---|---|
| internet → edge | ALB mutual TLS: listener MutualAuthentication with Mode off / passthrough / verify, a trust store (CreateTrustStore, revocation lists, shared trust stores) and AdvertiseTrustStoreCaNames | yes, at the ALB. The target receives the client certificate in headers | API model |
| internet → edge, certificates | ACM ACME endpoint (CreateAcmeEndpoint, External Account Binding, Route 53 DNS prevalidation), announced 6 Jul 2026, issuing public 45-day certificates; ACM public certificates can also be marked exportable (CertificateOptions.Export) | n/a | API model; announcement |
| ECS service → ECS service | ECS Service Connect TLS: issuerCertificateAuthority.awsPcaAuthorityArn, kmsKey, roleArn | TLS from AWS Private CA, managed for you; client-certificate verification is not exposed as a setting | API model |
| pod → pod across VPCs and accounts | VPC Lattice: auth type AWS_IAM with SigV4 per request; listeners HTTP, HTTPS, TLS_PASSTHROUGH | no. Identity comes from IAM, not certificates | API model |
| pod → pod in one cluster | Istio ambient or sidecar, with the mesh CA chained to AWS Private CA via cert-manager's aws-privateca-issuer and istio-csr; Private CA's SHORT_LIVED_CERTIFICATE mode suits workload certificates | yes | Private CA usage-mode enum in the API model; the issuer chain was not run |
| node → node | Nitro in-transit encryption between supported instance types | no: encryption, not identity. This is the same distinction Howard draws for WireGuard | AWS documentation, not checked here |
What the older posts add, and what they get wrong:
- Fleeto's notes are a map of an API that no longer exists. Every resource was dry-run, and every one failed with "no matches for kind":
authentication.istio.io/v1alpha1Policy, and Mixer'sdenier,listchecker,checknothingandrule. TheServiceRoleandServiceRoleBindingit describes went with them. The certificates in/etc/certsare covered in Istio on EKS . - The attack-vectors post says Mixer policy can cache decisions at the edge, so the mesh keeps running if the policy backend fails. Mixer was removed in Istio 1.8. The modern equivalent of that availability argument is
AuthorizationPolicywithCUSTOMexternal authorization, plus a decision about failing open or closed. On AWS, the edge half is WAF on the ALB. - The cert-manager post describes a period when one certificate covered every hostname on the Istio gateway, and the HTTP-01 solver pods had to be exposed by hand. Nearly all of it is now obsolete; see the corrections table. On EKS the current shape is cert-manager 1.x writing a Secret that a Gateway listener references by
credentialName. You can also skip cluster certificates entirely and terminate on an ALB with ACM. The newest option fits this post exactly: cert-manager's ACMEClusterIssuerpointed at an ACM ACME endpoint, with domain validation done once by an administrator. No HTTP-01 solver needs to be reachable from the internet. It was announced in July 2026 and is in the botocore model. This page did not run it. - WePay's per-service certificates show the cost Howard warns about. Each service's certificate was managed separately, and the post calls that useful for some setups and burdensome for others. A mesh CA, or Service Connect's Private CA integration, exists to remove that cost.
3. Who configures the proxies? xDS, and what AWS did with it
Eight posts are about control planes:
- Klein's universal data-plane API
- the Pilot xDS notes
- Cookpad's S3-hosted xDS
- UCloud's Pilot without Kubernetes
- the linkerd2 destination analysis
- the eBPF proxy model
- the Red Hat overview
- Komljen's introduction
Klein's 2017 post predicted two things. Proxies would converge on one management API, and the value would move to the control plane as data planes became commodities. He also described the mechanics: v2 moved from polled JSON to gRPC streams, and SDS was renamed EDS. The translator notes that the promised "KDS" for keys arrived as the Secret Discovery Service.
Both predictions came true, but not as a shared API. AWS built three control planes and exposed none of them as xDS:
| AWS service | data plane | whose control plane | status, 7 Oct 2026 |
|---|---|---|---|
| AWS App Mesh | Envoy, as a sidecar you injected | AWS's, serving xDS to your Envoys | end of support 30 Sep 2026. AWS's guide says console and resource access ends. The API model is still in botocore 1.43.107 (38 operations), so an SDK that still has the model does not mean the service still runs |
| ECS Service Connect | an Envoy-based agent AWS injects and manages | AWS's, not exposed | the documented App Mesh migration target for ECS |
| VPC Lattice | none in your task or pod; the VPC's network does it | AWS's; you write Gateway API HTTPRoute through the AWS Gateway API Controller, or call the Lattice API | the EKS path. It does not retry, break circuits or eject outliers (see Service mesh on AWS) |
Cookpad (2018) is the most useful post here for an AWS reader. Its services ran on ECS, not Kubernetes, which is why it did not adopt Istio. It generated xDS v1 responses from a central repository with a Ruby gem,
kumonos, and served them from S3. A sidecar registered healthy tasks for client-side load balancing, because an internal NLB or a TCP-mode CLB balanced gRPC poorly. The post reports one result worth remembering: proxy retries took the stream of transient network errors from more than 200 an hour down to about one a week. Tracing went to X-Ray. App Mesh was, in effect, this design run as a service. Its successor for the same ECS estate is Service Connect. Its API (ServiceConnectService) offers timeout, with idleTimeoutSeconds and perRequestTimeoutSeconds, and tls. It has no retry or outlier-detection settings. The retries Cookpad valued most are whatever Service Connect's proxy does by default, so check that before migrating.UCloud (2018) pulled Pilot out of Istio, ran it on etcd with no Kubernetes, and used explicit proxy ports instead of iptables capture. Its reasons are still the reasons people give for not adopting a mesh: no Kubernetes, a large estate of existing services, iptables complexity, and databases outside the cluster. Lattice answers three of the four, because it needs no proxy and reaches EC2, ECS, Lambda and (through resource configurations) databases.
The Pilot xDS notes are a list of endpoints that no longer exist:
/v1/registration, /v1/clusters/..., and :8080/debug/edsz. Cluster names have changed too. Pilot wrote out.reviews...|http|version=v1; istiod writes outbound|9080|v1|reviews.<ns>.svc.cluster.local. The current tools are istioctl proxy-status and proxy-config, which Service mesh on AWS already covers, and istioctl x internal-debug for istiod's own view (it is present in 1.30.3).Load balancing. The linkerd2 analysis walks through the Rust proxy: the destination service streams endpoints, and the proxy picks between two random endpoints by peak-EWMA latency. No AWS load balancer does this. The ALB's
load_balancing.algorithm.type offers round_robin, least_outstanding_requests and weighted_random, and the last one can turn on anomaly_mitigation. That is the nearest AWS-managed equivalent of latency-aware balancing with outlier handling.Per-node proxies. WePay chose Linkerd 1 partly for its DaemonSet mode, with one proxy per node, and the 2018 eBPF post proposed a per-node kernel data plane. Both ideas returned in 2022–2024 as Istio ambient's ztunnel. Istio on EKS covers what that means on a node.
The fallacies table. Red Hat's 2018 overview maps each fallacy of distributed computing to an Istio feature. Its "transport cost is zero" row answers with gRPC and Protobuf, meaning fewer bytes on the wire. On AWS that fallacy is now a line item: traffic between Availability Zones is billed in each direction, and Lattice charges per GB processed. Istio's locality load balancing, Kubernetes topology-aware routing and Lattice's pricing belong in that row now.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article