AWS Builder Center
Multi-VPC Load Balancing for Amazon EKS workloads

Multi-VPC Load Balancing for Amazon EKS workloads

Authors: Ashok Srirama, Sai Charan Teja Gopaluni, Sai Kumar Samala

Introduction

In traditional Amazon Elastic Kubernetes Service (Amazon EKS)  deployments, when you create a Kubernetes Service of type LoadBalancer , the AWS Load Balancer Controller  provisions a Network Load Balancer (NLB)  in the same Amazon VPC  as your Amazon EKS cluster. However, certain use cases may require placing your load balancers in a different VPC than the one hosting your Amazon EKS cluster. For example, you might need this separation for enhanced security isolation, where public-facing load balancers reside in a dedicated perimeter network VPC. Another common scenario is centralized networking management, where a dedicated networking team maintains all load balancers in a separate VPC to enforce consistent routing policies and simplify compliance requirements.
This article demonstrates how to configure the AWS Load Balancer Controller to create AWS ELBs in a different VPC (VPC B) than the one hosting your EKS cluster (VPC A). This architecture provides enhanced security isolation, network segmentation, and more flexible network design options.

Use Cases

Why would you want to deploy AWS ALB/NLBs in a separate VPC from your EKS cluster?
  1. Security Isolation: Separate public-facing components (ALBs) from your application workloads for enhanced security.
  2. Network Segmentation: Maintain a clear separation between internet-facing infrastructure and internal workloads.
  3. Compliance Requirements: Meet regulatory requirements that mandate separation of public and private resources.
  4. Centralized Load Balancer Management: Manage all load balancers in a dedicated VPC across multiple application VPCs.
  5. Shared Services Model: Create a shared services VPC for common infrastructure components like load balancers.
  6. Legacy Integration: Connect modern EKS workloads with existing load balancer infrastructure in separate VPCs.

Prerequisites

Before you begin, ensure you have:
  1. An AWS account with appropriate permissions
  2. AWS CLI  configured with access to your account
  3. kubectl 
  4. eksctl 
  5. helm 
  6. Two Amazon VPCs:
    • VPC A: Where your Amazon EKS cluster is deployed
    • VPC B: Where you want to deploy your AWS ALB/NLBs

Implementation Overview

Here's a high-level overview of what we'll accomplish:
  1. Create VPC A and VPC B (if they don't already exist)
  2. Deploy an Amazon EKS cluster in VPC A
  3. Set up VPC peering between VPC A and VPC B, alternatively you can utilize AWS Transit Gateway to connect the VPCs
  4. Configure route tables and security groups
  5. Install and configure the AWS Load Balancer Controller with the VPC B ID
  6. Deploy a sample application with appropriate annotations
  7. Verify the load balancers created in VPC B
High Level Architecture
High level architecture

Detailed Implementation Steps

Step 1: Create two VPCs (if they don't exist)

Create two VPCs if they don't exist. We'll use AWS CloudFormation  templates for this purpose.
1
export AWS_REGION="us-west-2" # Replace with your AWS region

For VPC A (EKS Cluster VPC):

1
2
3
4
aws cloudformation create-stack \
--stack-name eks-vpc \
--region $AWS_REGION \
--template-url https://amazon-eks.s3.amazonaws.com/cloudformation/2020-10-29/amazon-eks-vpc-private-subnets.yaml

For VPC B (AWS ELB VPC):

1
2
3
4
5
6
7
aws cloudformation create-stack \
--stack-name elb-vpc \
--region $AWS_REGION \
--template-url https://aws-quickstart.s3.amazonaws.com/quickstart-aws-vpc/templates/aws-vpc.template.yaml \
--parameters ParameterKey=AvailabilityZones,ParameterValue=${AWS_REGION}a\\,${AWS_REGION}b\\,${AWS_REGION}c \
ParameterKey=NumberOfAZs,ParameterValue=3 ParameterKey=PublicSubnetTag1,ParameterValue='kubernetes.io/role/elb=1' \
ParameterKey=CreateNATGateways,ParameterValue=false

Step 2: Create an Amazon EKS Cluster in VPC A

Retrieve VPC A ID

1
2
export VPC_A_ID=$(aws cloudformation describe-stacks --stack-name eks-vpc --query "Stacks[0].Outputs[?OutputKey=='VpcId'].OutputValue" \
--output text --region $AWS_REGION)

Retrieve Subnet IDs

1
2
export SUBNET_IDS=$(aws cloudformation describe-stacks --stack-name eks-vpc --query "Stacks[0].Outputs[?OutputKey=='SubnetIds'].OutputValue" \
--output text --region $AWS_REGION)

Create EKS Cluster with eksctl:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
cat > eks-cluster.yaml << EOF
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: cross-vpc-demo
region: "${AWS_REGION}"
version: '1.33'
vpc:
subnets:
private:
${AWS_REGION}a: { id: $(echo $SUBNET_IDS | cut -d ',' -f3) }
${AWS_REGION}b: { id: $(echo $SUBNET_IDS | cut -d ',' -f4) }
managedNodeGroups:
- name: ng-1
instanceType: t3.medium
desiredCapacity: 2
privateNetworking: true
EOF
eksctl create cluster -f eks-cluster.yaml

Step 3: Set up VPC Peering between VPC A and VPC B

Get VPC B ID

1
2
export VPC_B_ID=$(aws cloudformation describe-stacks --stack-name elb-vpc --query "Stacks[0].Outputs[?OutputKey=='VPCID'].OutputValue" \
--output text --region $AWS_REGION)

Create VPC peering connection

1
2
3
4
5
6
export PEERING_CONNECTION_ID=$(aws ec2 create-vpc-peering-connection \
--vpc-id $VPC_A_ID \
--peer-vpc-id $VPC_B_ID \
--query "VpcPeeringConnection.VpcPeeringConnectionId" \
--region $AWS_REGION \
--output text)

Accept the peering connection

1
2
aws ec2 accept-vpc-peering-connection --vpc-peering-connection-id $PEERING_CONNECTION_ID --region $AWS_REGION
echo "VPC Peering Connection ID: $PEERING_CONNECTION_ID"

Step 4: Update Route Tables

Update route tables in both VPCs to route traffic through the peering connection:

Get CIDR blocks for both VPCs

1
export VPC_A_CIDR=$(aws ec2 describe-vpcs --vpc-ids $VPC_A_ID --query "Vpcs[0].CidrBlock" --output text --region $AWS_REGION)
1
export VPC_B_CIDR=$(aws ec2 describe-vpcs --vpc-ids $VPC_B_ID --query "Vpcs[0].CidrBlock" --output text --region $AWS_REGION)
1
2
3
4
5
export VPC_A_ROUTE_TABLES=$(aws ec2 describe-route-tables \
--filters "Name=vpc-id,Values=$VPC_A_ID" \
--query "RouteTables[*].RouteTableId" \
--region $AWS_REGION \
--output text)

Get route table IDs for VPC B

1
2
3
4
5
export VPC_B_ROUTE_TABLES=$(aws ec2 describe-route-tables \
--filters "Name=vpc-id,Values=$VPC_B_ID" \
--query "RouteTables[*].RouteTableId" \
--region $AWS_REGION \
--output text)

Update route tables in VPC A to route to VPC B

1
2
3
4
5
6
7
8
9
10
echo "$VPC_A_ROUTE_TABLES" | tr '\t' '\n' | while read -r RT_ID; do

echo "Creating route for route table: $RT_ID"

aws ec2 create-route \
--route-table-id "$RT_ID" \
--destination-cidr-block "$VPC_B_CIDR" \
--vpc-peering-connection-id "$PEERING_CONNECTION_ID" \
--region "$AWS_REGION"
done

Update route tables in VPC B to route to VPC A

1
2
3
4
5
6
7
8
9
10
echo "$VPC_B_ROUTE_TABLES" | tr '\t' '\n' | while read -r RT_ID; do

echo "Creating route for route table: $RT_ID"

aws ec2 create-route \
--route-table-id "$RT_ID" \
--destination-cidr-block "$VPC_A_CIDR" \
--vpc-peering-connection-id "$PEERING_CONNECTION_ID" \
--region "$AWS_REGION"
done

Step 5: Install AWS Load Balancer Controller with VPC B ID

Create pod-identity add-on

1
aws eks create-addon --cluster-name cross-vpc-demo --addon-name eks-pod-identity-agent --region "$AWS_REGION

Create Service Account:

1
kubectl create serviceaccount aws-load-balancer-controller -n kube-system

Pod Identity association for AWS Load Balancer controller

1
eksctl create podidentityassociation --cluster cross-vpc-demo --namespace kube-system --service-account-name aws-load-balancer-controller --role-name aws-load-balancer-controller --well-known-policies awsLoadBalancerController --region us-east-1

Install the AWS Load Balancer Controller in your EKS cluster, specifying VPC B as the target VPC

1
2
3
helm repo add eks https://aws.github.io/eks-charts
helm repo update
helm install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=cross-vpc-demo --set serviceAccount.create=false --set serviceAccount.name=aws-load-balancer-controller --set vpcId=$VPC_B_ID
Important: The key configuration here is setting --set vpcId=$VPC_B_ID, which instructs the controller to create load balancers in VPC B instead of the EKS cluster's VPC.

Step 6: Deploy a Sample Application

Get public subnet IDs from VPC B

1
2
3
4
export VPC_B_PUBLIC_SUBNETS=$(aws ec2 describe-subnets \
--filters "Name=vpc-id,Values=$VPC_B_ID" "Name=map-public-ip-on-launch,Values=true" \
--query "Subnets[*].SubnetId" --region $AWS_REGION \
--output json | tr -d '[]" ' | tr ',' ' ')

Create a sample deployment and service

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
cat > sample-app.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: sample-app
spec:
replicas: 2
selector:
matchLabels:
app: sample-app
template:
metadata:
labels:
app: sample-app
spec:
containers:
- name: nginx
image: nginx:1.21
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: sample-app-lb-service
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: "external"
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
spec:
type: LoadBalancer
ports:
- port: 80
targetPort: 80
protocol: TCP
selector:
app: sample-app
EOF

kubectl apply -f sample-app.yaml

Step 6: Verify the NLB is Created in VPC B

Get the service details

1
kubectl get service sample-app-lb-service

Check the NLB in AWS console or using AWS CLI

1
2
aws elbv2 describe-load-balancers \
--query "LoadBalancers[?VpcId=='$VPC_B_ID']" --region $AWS_REGION

Step 7: Create AWS ALB in VPC B

Create a ClusterIP service and Ingress

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
cat > sample-app-ingress.yaml << EOF
apiVersion: v1
kind: Service
metadata:
name: sample-app-service
spec:
type: ClusterIP
ports:
- port: 80
targetPort: 80
protocol: TCP
selector:
app: sample-app
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: sample-app-ingress
annotations:
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
spec:
ingressClassName: alb
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: sample-app-service
port:
number: 80
EOF

kubectl apply -f sample-app-ingress.yaml

Step 8: Verify the ALB is Created in VPC B

Get the service details

1
kubectl get ingress sample-app-ingress

Check the ALB in AWS console or using AWS CLI

1
2
aws elbv2 describe-load-balancers \
--query "LoadBalancers[?VpcId=='$VPC_B_ID']" --region $AWS_REGION

Considerations

  • The LoadBalancer target type must be ip
  • The subnets in VPC B must have following tag kubernetes.io/role/elb: 1 for public ELBs and kubernetes.io/role/internal-elb: 1 for internal ELBs.

Conclusion

In this article, we've demonstrated how to configure the AWS Load Balancer Controller to create AWS NLB and ALBs in a separate VPC from your EKS cluster. This architecture provides enhanced security isolation, network segmentation, and more flexible network design options.
By separating your public-facing load balancers from your application workloads, you can implement a more secure and compliant infrastructure design while still leveraging the power and flexibility of Amazon EKS and the AWS Load Balancer Controller.
This approach can be extended to more complex scenarios, such as using AWS Transit Gateway for connecting multiple VPCs or implementing more sophisticated network security controls between your load balancers and application workloads.

Additional Resources

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article