Cross-Cluster Same-Zone Routing with Cilium on EKS

Reduce cross-AZ latency by 67% using Cilium ClusterMesh's eBPF topology-aware routing across EKS clusters connected via Transit Gateway or VPC Peering.

Amuthan Hamaline G K

Published Jun 19, 2026

Last modified Jun 20, 2026

Prerequisites: This guide assumes you are already running Cilium as your CNI on Amazon EKS. ClusterMesh is a Cilium-native feature that extends your existing Cilium deployment with cross-cluster service discovery and topology-aware routing.

Multi-cluster EKS architectures are increasingly common. A simple and frequent pattern: split services by domain — an Orders cluster handling checkout, inventory, and fulfillment, and a Payments cluster handling transactions, billing, and fraud detection. This separation gives you independent scaling, blast radius isolation, and team autonomy.

But these two clusters communicate constantly — every checkout triggers a payment call, every refund triggers an order update. That's continuous, high-volume east-west API traffic crossing cluster boundaries.

The cost problem at 10TB/month inter-cluster traffic:

Via Transit Gateway: Data processing costs $0.02/GB regardless of AZ — that's ~$205/month fixed. Cross-AZ data transfer itself is free through TGW (since April 2022). The cost motivator here is latency — cross-AZ adds ~1ms round-trip vs ~0.3ms same-zone.
Via VPC Peering: Same-AZ peering is free, but cross-AZ costs $0.01/GB each direction. With 3 AZs, roughly 2/3 of randomly-distributed traffic is cross-AZ — that's $137/month. Same-zone routing keeps traffic in the free same-AZ path.

The question: Can we make traffic between the Orders and Payments clusters prefer same-zone endpoints — even though they're in different EKS clusters, different VPCs, or different AWS accounts?

The answer: Yes. Cilium ClusterMesh with trafficDistribution: PreferSameZone does exactly this — at kernel level, with zero latency overhead, using the same CNI you may already be running.

What is Cilium ClusterMesh?

Cilium is an eBPF-based Container Network Interface (CNI) plugin for Kubernetes. It replaces kube-proxy and iptables entirely with eBPF programs running inside the Linux kernel. Instead of walking through iptables rules one by one (O(n)), Cilium uses BPF hash maps for service routing — constant-time lookups regardless of scale.

ClusterMesh is Cilium's multi-cluster feature. It connects the Cilium agents across clusters via per-cluster etcd instances exposed by the clustermesh-apiserver, enabling:

Cross-cluster service discovery — pods in the Orders Cluster can reach services in the Payments Cluster using standard Kubernetes Service names
Shared identity — security policies work across cluster boundaries
Topology-aware load balancing — the trafficDistribution field in the Service spec tells Cilium to prefer same-zone endpoints

The key insight: ClusterMesh doesn't need service mesh sidecars, DNS tricks, or application changes. It operates at Layer 3/4 in the kernel — before your application even sees the packet.

How Same-Zone Routing Works: The Packet Path

When a pod in the Orders Cluster (in us-east-1c) calls a service in the Payments Cluster, here's what happens:

1
2
3
4
5
6
7
8
9
10
11
1. App sends packet (e.g. curl http://payment-service.payments:8080)
2. Packet hits the eBPF program at tc (traffic control) hook — BEFORE it reaches the iptables/netfilter stack
3. Cilium's BPF map lookup finds all endpoints for the service (both local and remote via ClusterMesh)
4. With trafficDistribution: PreferSameZone, the BPF program:
   a. Checks the zone metadata in each backend's BPF map entry (populated from EndpointSlice hints)
   b. Filters backends to those matching the local node's zone (us-east-1c)
   c. If same-zone backends exist → select one (random/maglev)
   d. If no same-zone backends → fall back to any available backend
5. DNAT rewrites the packet destination to the chosen endpoint IP
6. Packet is routed natively to the remote cluster — via Transit Gateway (cross-VPC) or VPC Peering (direct) — no encapsulation in native-routing mode
7. Remote cluster's Cilium agent delivers to the target pod

Latency impact: Steps 2–5 happen in microseconds (BPF hash map lookups). The routing decision adds negligible overhead compared to the actual network transit.

Cross-Cluster Service Discovery: 3 Patterns

Pattern 1: Global Service (Recommended for Most Cases)

Both clusters define the same service with the annotation service.cilium.io/global: "true":

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# In BOTH clusters:
apiVersion: v1
kind: Service
metadata:
  name: payment-service
  namespace: payments
  annotations:
    service.cilium.io/global: "true"
    service.cilium.io/shared: "true"
spec:
  trafficDistribution: PreferSameZone
  ports:
    - port: 8080
      targetPort: 8080
  selector:
    app: payment-service

Traffic from any cluster reaches the nearest same-zone endpoint across all participating clusters.

Pattern 2: Bridge Service (Cross-Namespace/Cross-Cluster Proxy)

Use case: The target service exists only in the remote cluster (Payments), and you want pods in the local cluster (Orders) to reach it without deploying any pods locally.

How it works: You create a Service in the caller's cluster with the same name and namespace as the target — but with no selector. Kubernetes sees 0 endpoints locally. However, because the service is annotated service.cilium.io/global: "true", ClusterMesh populates the BPF map with the remote cluster's endpoints. The caller's pods resolve the service via DNS as usual, and Cilium routes directly to the remote backends.

YAML — deployed in the Orders Cluster (caller) only:

1
2
3
4
5
6
7
8
9
10
11
12
# In Orders Cluster (caller): bridge service — same name as target
apiVersion: v1
kind: Service
metadata:
  name: payment-service
  namespace: payments
  annotations:
    service.cilium.io/global: "true"
spec:
  trafficDistribution: PreferSameZone
  ports:
    - port: 8080

Key points:

No selector → Kubernetes shows 0 endpoints in kubectl get endpoints, but cilium-dbg service list will show the remote backends in the BPF map
The Payments Cluster must have the actual service (with selector + pods) also annotated service.cilium.io/global: "true" and service.cilium.io/shared: "true"
DNS resolution works normally — pods call payment-service.payments.svc.cluster.local
trafficDistribution: PreferSameZone still applies — Cilium picks the closest remote backend by zone

Pattern 3: Headless Global Service (StatefulSet-aware)

Use case: You need to reach individual pods by hostname across clusters — typical for StatefulSets like databases where each pod has a stable identity (e.g., payment-db-0, payment-db-1). In this example, the Payments cluster runs a StatefulSet database that the Orders cluster needs to connect to specific replicas of.

How it works: A headless Service (clusterIP: None) with the global annotation causes ClusterMesh to sync individual pod IPs (not a single VIP) into the remote cluster's DNS. Each pod gets a DNS A record: <pod-name>.<service-name>.<namespace>.svc.cluster.local. Callers can address specific pods directly or let DNS round-robin.

YAML — deployed in the Orders Cluster (caller):

1
2
3
4
5
6
7
8
9
10
11
12
13
# In Orders Cluster — headless bridge to Payments Cluster's StatefulSet
apiVersion: v1
kind: Service
metadata:
  name: payment-db
  namespace: payments
  annotations:
    service.cilium.io/global: "true"
    service.cilium.io/shared: "true"
spec:
  clusterIP: None
  ports:
    - port: 5432

Key points:

clusterIP: None is what makes it headless — DNS returns individual pod IPs instead of a single VIP
EndpointSlice synchronization is enabled by default for headless global services (no extra config needed)
Each remote pod appears as a DNS record: payment-db-0.payment-db.payments.svc.cluster.local
trafficDistribution: PreferSameZone is NOT applicable here — headless services bypass the BPF load balancer (callers connect to specific pods)
Useful for client-side load balancing or when the application needs to be aware of the full topology (e.g., connecting to a specific DB replica)

Comparing Cross-Cluster Routing Options

Feature	ClusterMesh + Cilium	AWS Cloud Map	VPC Lattice
Routing granularity	Pod-to-pod, same-zone	Service-to-service	Service-to-service
Zone-aware routing	Same-zone preference (eBPF)	Latency-based routing (Route 53)	Cross-AZ included in pricing (no separate inter-AZ bill)
Protocol	L3/L4 (kernel)	DNS-based	L4/L7 (managed)
Cross-account	Yes (Transit Gateway or VPC Peering)	Yes (namespace sharing)	Yes (resource shares)
Latency overhead	Microseconds (BPF)	DNS TTL-dependent	Single-digit ms (managed proxy, includes auth + observability)
Cost (infrastructure)	Zero (open-source CNI)	$0.10/resource/month + $1/M API calls	$0.025/hour/service + $0.025/GB
Operational burden	Self-managed (Cilium upgrades, etcd, troubleshooting)	Managed (AWS operated)	Fully managed (zero infrastructure to operate)
Auth	mTLS (Cilium-managed certificates)	IAM + Route 53 policies	Built-in IAM auth (no certificate management)
Observability	Hubble (eBPF-native)	CloudWatch	Access logs + CloudWatch metrics (per-request detail)
L7 routing (path/header)	No (L3/L4 only)	No	Yes (path-based, header-based rules)

When to choose ClusterMesh: You already run Cilium as your CNI, you need pod-level same-zone routing at kernel speed, and you're comfortable owning the Cilium lifecycle (upgrades, etcd operations, troubleshooting).

When to choose VPC Lattice: You need managed L7 routing with path/header-based rules, IAM-native authentication without certificate management, zero operational overhead (fully managed — no infrastructure to patch or upgrade), or you want cross-AZ data transfer costs absorbed into a predictable per-GB fee.

When to choose Cloud Map: You need lightweight DNS-based discovery with latency-based routing via Route 53, and your services communicate primarily via DNS lookups rather than direct pod-to-pod connections.

Implementation Guide

Prerequisites

2+ EKS clusters with Cilium as the CNI (v1.14+), Kubernetes 1.33+ (for trafficDistribution: PreferSameZone; Stable in K8s 1.35)
Cilium Helm config: loadBalancer.serviceTopology=true (enables topology-aware BPF routing)
Clusters connected via Transit Gateway or VPC Peering (pod CIDR ranges must not overlap)
Pod CIDRs routable between clusters (VPC route tables configured — pointing remote pod CIDRs to TGW or Peering connection)
Security groups must allow inter-cluster pod traffic on all required ports
All clusters must use the same Cilium datapath mode (native-routing or encapsulation — cannot mix)
Cilium Helm values: cluster.name and cluster.id must be unique per cluster

Step-by-Step Setup

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# Step 1: Enable ClusterMesh on each cluster (deploys API server + etcd)
$ cilium clustermesh enable --context orders-cluster --service-type LoadBalancer
$ cilium clustermesh enable --context payments-cluster --service-type LoadBalancer

# Step 2: Connect clusters (establishes etcd peering)
$ cilium clustermesh connect --context orders-cluster --destination-context payments-cluster

# Step 3: Verify mesh status
$ cilium clustermesh status --context orders-cluster
# Should show: ✅ Cluster Connections: 1/1 connected

# Step 4: Deploy global service with zone-aware routing (apply in BOTH clusters)
$ kubectl --context payments-cluster apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: payment-service
  namespace: payments
  annotations:
    service.cilium.io/global: "true"
    service.cilium.io/shared: "true"
spec:
  trafficDistribution: PreferSameZone
  ports:
    - port: 8080
  selector:
    app: payment-service
EOF

Validation: Proving It Works

Test 1: Cross-Cluster Connectivity

1
2
3
4
5
6
# Deploy payment-service on Payments Cluster with global annotation
$ kubectl --context payments-cluster apply -f payment-service.yaml

# From Orders Cluster, call the service
$ kubectl --context orders-cluster exec -it client-pod -- curl payment-service.payments:8080
# Response should come from Payments Cluster pod

Test 2: Same-Zone Preference

1
2
3
4
5
6
7
8
9
# Verify BPF map contains backends from BOTH clusters:
# (Note: kubectl get endpoints only shows LOCAL K8s endpoints — remote ClusterMesh
#  backends are in Cilium's BPF maps, visible via cilium-dbg)
$ kubectl --context orders-cluster exec -n kube-system ds/cilium -- cilium-dbg service list | grep payment-service
# Should show backends from both local and remote clusters

# Check which endpoint was selected:
$ kubectl --context orders-cluster exec client-pod -- curl -s payment-service.payments:8080/zone
# Should return: us-east-1c (same zone as caller)

Test 3: Zone Failover

1
2
3
4
5
6
7
8
9
10
# Client pod is on node in us-east-1c
# payment-service pods: one in us-east-1c (Payments Cluster), one in us-east-1a (Payments Cluster)

# Kill the us-east-1c pod:
$ kubectl --context payments-cluster delete pod payment-service-xxxx-c

# Wait 5-15 seconds for BPF map to converge (endpoint removal propagates via kvstore)
# Re-test: should now route to us-east-1a (fallback)
$ kubectl --context orders-cluster exec client-pod -- curl -s payment-service.payments:8080/zone
# Returns: us-east-1a (fallback — no same-zone endpoint available)

Operations & Resilience

Failure Modes

Scenario	Behavior
Remote cluster unreachable	Traffic routes to local endpoints only
All same-zone endpoints down	Falls back to other zones (graceful degradation)
ClusterMesh API server down	Existing routes maintained in cached BPF maps (default: indefinite, configurable via cacheTTL)
Network path disruption (e.g., AZ failure)	TCP retries handle transient drops; BPF maps unaffected. TGW VPC attachments are multi-AZ by design.

Monitoring with Hubble

1
2
3
4
5
6
7
8
9
# Watch cross-cluster flows and inspect destination IPs:
$ hubble observe --from-pod default/client --to-service payments/payment-service -o compact
# Correlate destination IPs with node zones:
$ kubectl get nodes -o custom-columns='NAME:.metadata.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone,IPs:.status.addresses[*].address'

# For zone-aware metrics, configure Hubble Prometheus exporter with zone context:
# (Helm: hubble.metrics.enabled=["flow:destinationContext=pod;sourceContext=pod"])
# Then query in Prometheus/Grafana:
# sum(rate(hubble_flows_processed_total{source_pod_zone!=destination_pod_zone}[5m]))

Day-2 Operations

Scaling: Add more pods in underserved zones → Cilium auto-discovers via kvstore sync (< 5s propagation)
Upgrades: Rolling Cilium upgrades maintain mesh connectivity (eBPF programs persist across agent restarts)
Troubleshooting: cilium-dbg bpf lb list shows the live load balancing table per node

Cost Analysis

Scenario: 10TB/month Inter-Cluster API Traffic (3 AZs)

VPC Peering (cost savings apply):

Metric	Without Same-Zone Routing	With Same-Zone Routing	Savings
Cross-AZ traffic	~6.7 TB (2/3 random)	~1.0 TB (fallback only)	85% reduction
Cross-AZ cost ($0.01/GB × 2 dirs)	$134/month	$20/month	$114/month
Latency (P50)	1.2ms (cross-AZ)	0.4ms (same-zone)	67% lower
Additional infrastructure	N/A	Zero (Cilium is your CNI)	$0

Transit Gateway (latency savings apply):

Metric	Without Same-Zone Routing	With Same-Zone Routing	Savings
TGW processing ($0.02/GB)	$205/month	$205/month	$0 (all traffic traverses TGW)
Cross-AZ data transfer	$0 (free via TGW since Apr 2022)	$0	$0
Latency (P50)	1.2ms (cross-AZ)	0.4ms (same-zone)	67% lower
Additional infrastructure	N/A	Zero (Cilium is your CNI)	$0

Note: With Transit Gateway, cross-AZ data transfer is free (since April 2022), so same-zone routing saves latency, not cost. With VPC Peering, same-AZ is free but cross-AZ costs $0.01/GB each direction — same-zone routing saves both latency and ~$114/month at 10TB scale.

Conclusion & Recommended Rollout

Phase 1: Foundation (Week 1–2)

Enable ClusterMesh between Orders ↔ Payments clusters
Validate basic cross-cluster connectivity

Phase 2: Zone-Aware Routing (Week 3–4)

Add trafficDistribution: PreferSameZone to high-volume services
Deploy Hubble for cross-zone traffic monitoring
Validate improvement — Cost Explorer Data Transfer line item (VPC Peering) or P50 latency metrics (Transit Gateway)

Phase 3: Full Mesh (Week 5–6)

Extend to additional domain clusters
Implement NetworkPolicies across cluster boundaries
Set up alerts for zone imbalance (> 20% traffic to non-local zones)

Bottom line: If you're already running ClusterMesh across your EKS clusters, adding trafficDistribution: PreferSameZone to your global services cuts cross-AZ latency by ~67% with zero application changes. For clusters connected via VPC Peering, you also save 60–85% on cross-AZ data transfer costs. For Transit Gateway, cross-AZ transfer is already free — the win is purely latency and P50 response time.

References

# amazon-eks # networking # aws-networking # advanced-networking

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

Cross-Cluster Same-Zone Routing with Cilium on EKS

Amuthan Hamaline G K

What is Cilium ClusterMesh?

How Same-Zone Routing Works: The Packet Path

Cross-Cluster Service Discovery: 3 Patterns

Pattern 1: Global Service (Recommended for Most Cases)

Pattern 2: Bridge Service (Cross-Namespace/Cross-Cluster Proxy)

Pattern 3: Headless Global Service (StatefulSet-aware)

Comparing Cross-Cluster Routing Options

Implementation Guide

Prerequisites

Step-by-Step Setup

Validation: Proving It Works

Test 1: Cross-Cluster Connectivity

Test 2: Same-Zone Preference

Test 3: Zone Failover

Operations & Resilience

Failure Modes

Monitoring with Hubble

Day-2 Operations

Cost Analysis

Scenario: 10TB/month Inter-Cluster API Traffic (3 AZs)

Conclusion & Recommended Rollout

Phase 1: Foundation (Week 1–2)

Phase 2: Zone-Aware Routing (Week 3–4)

Phase 3: Full Mesh (Week 5–6)

References