AWS Builder Center

Builder Center

SaaS Anywhere: Setting Up Amazon EKS Hybrid Nodes

SaaS Anywhere: Setting Up Amazon EKS Hybrid Nodes

This hands-on guide walks through setting up Amazon EKS Hybrid Nodes for SaaS Anywhere deployments. You'll configure IAM roles, install Cilium CNI, establish network connectivity, and onboard customer infrastructure to your EKS cluster. The result is a foundation where your SaaS workloads can run on customer infrastructure while maintaining centralized Kubernetes control.

Schumann, Luca

SaaS Anywhere: Setting Up Amazon EKS Hybrid Nodes

In our foundational articles, we explored how SaaS Anywhere addresses the business drivers that traditional SaaS deployment models can't handle ; compliance requirements, data sovereignty needs, and performance demands that block customer adoption. We established the core architectural principle: separating application-level control (which you retain) from infrastructure-level responsibility (which customers assume) through centralized control planes and distributed application planes.
We then showed how Amazon EKS Hybrid Nodes provides the technical foundation for this architecture , extending your single Kubernetes control plane to manage compute resources wherever they live—in your AWS account, customer environments, or edge locations. This preserves your existing operational investments while enabling workloads to run on customer infrastructure.
Now it's time to build it. This guide walks through important steps of the setup process in three phases:
Phase 1: One-Time Cluster Setup - Configure your EKS cluster with the IAM roles, authentication settings, and Cilium CNI needed to support hybrid nodes across all customer environments.
Phase 2: Network Connectivity - Establish secure connectivity between your AWS VPC and customer environments through VPN or Direct Connect, including CIDR planning and routing configuration.
Phase 3: Customer Onboarding - The repeatable process for adding each new customer environment, from EKS cluster configuration to SSM activation to customer node registration.
By the end, you'll understand the overall process of creating the infrastructure needed for offering SaaS Anywhere through Amazon EKS Hybrid Nodes.

Prerequisites

There is a number of prerequisites you need to consider, when you want to add EKS Hybrid Nodes to an Amazon EKS cluster. Please find a detailled list in the hybrid nodes documentation . The most important ones to follow this article's instructions are:

1 One-Time Cluster Setup

The following steps prepare your EKS cluster to support hybrid nodes. Complete these once, then use the customer onboarding process for each new customer environment.
High-level process:
1.1 Create IAM role and policies for hybrid nodes to authenticate with AWS services
1.2 Verify EKS cluster meets hybrid node requirements (authentication mode, endpoint access)
1.3 Install Cilium CNI since AWS VPC CNI can't run on hybrid nodes

1.1 Create IAM Role for Hybrid Nodes

Servers can only be added as Hybrid Nodes, if they provide valid credentials for a role with specific IAM privileges during the registration process. They need for example access for the Systems Manager API that adds hybrid nodes, read access to parts of the EKS API, and the access to pull images from ECR. The official EKS Hybrid Nodes documentation  provides deatils and a CloudFormation template.
The example role also contains a condition to restrict the ability to deregister nodes through the AWS Systems Manager. For simplicity, we follow the example and use the ARN of the EKS cluster. Please consider updating the resource and condition for this policy to meet your needs.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# Get your cluster ARN for the SSM condition
AWS_REGION="us-east-1" # Replace with the region your cluster is in
CLUSTER_NAME="your-cluster-name" # Replace with your actual cluster name
CLUSTER_ARN=$(aws eks describe-cluster --name $CLUSTER_NAME --query 'cluster.arn' --output text --region $AWS_REGION)

echo "Using cluster ARN: $CLUSTER_ARN"

# Deploy the official CloudFormation template
aws cloudformation deploy \
--stack-name eks-hybrid-nodes-iam \
--region ${AWS_REGION} \
--template-url https://raw.githubusercontent.com/aws/eks-hybrid/refs/heads/main/example/hybrid-ssm-cfn.yaml \
--parameter-overrides \
RoleName=AmazonEKSHybridNodesRole \
SSMDeregisterConditionTagKey=EKSClusterARN \
SSMDeregisterConditionTagValue=$CLUSTER_ARN \
--capabilities CAPABILITY_NAMED_IAM

# Verify the role was created
aws iam get-role --role-name AmazonEKSHybridNodesRole
If the role was successfully created, add an EKS access entry with the new IAM role. This grants the hybrid nodes permission to join the cluster and authenticate with the Kubernetes API server.
1
2
3
4
5
6
# Create access entry with type HYBRID_LINUX
aws eks create-access-entry \
--cluster-name ${CLUSTER_NAME} \
--principal-arn AmazonEKSHybridNodesRole \
--type HYBRID_LINUX \
--region ${AWS_REGION}

1.2 EKS Cluster Preparation

Your existing EKS cluster must meet these requirements for hybrid nodes (based on the official CloudFormation template ):
Cluster Service Role:
  • Must have AmazonEKSClusterPolicy managed policy attached
Authentication Mode:
  • Set to API_AND_CONFIG_MAP or API (required for hybrid nodes)
Endpoint Access:
  • Private endpoint access enabled (hybrid nodes need to reach the API server)
  • Public access is optional depending on your setup

1.3 CNI Installation

The AWS VPC CNI does not run on hybrid nodes, so you will need to install a hybrid nodes supported CNI, as of now that's either Cilium or Calico. We will install Cilium on the cluster. Since we want Cilium to run only on hybrid nodes, we use Kubernetes scheduling controls, specifically the node selector that targets the eks.amazonaws.com/compute-type: hybrid label that EKS automatically applies to all hybrid nodes. First, create the cilium-values.yaml file
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: eks.amazonaws.com/compute-type
operator: In
values:
- hybrid
ipam:
mode: cluster-pool
operator:
clusterPoolIPv4MaskSize: 25 # /25 = 128 IPs per node (126 usable for pods), adapt if necessary
operator:
unmanagedPodWatcher:
restart: false
Next, install Cilium in your cluster using the configuration with Helm. Check the Cilium compatibility matrix  to ensure your Cilium version supports your Kubernetes version, or run kubectl version --short to see your cluster version and match it against Cilium's supported versions.
1
2
3
4
5
6
7
8
CILIUM_VERSION=1.18.2

helm repo add cilium https://helm.cilium.io/

helm install cilium cilium/cilium \
--version ${CILIUM_VERSION} \
--namespace kube-system \
--values cilium-values.yaml
Validate the Cilium Helm release is deployed and healthy. Note that since we haven't registered any hybrid nodes yet, no pods should be deployed at this point.
1
helm status cilium -n kube-system

2 Establishing Connectivity

Before onboarding hybrid nodes, you need secure network connectivity between your AWS VPC and customer environments. This involves three key areas: CIDR planning, network connectivity and AWS-specific configuration.

2.1 CIDR Planning

Before onboarding nodes, it's important to understand the two CIDR ranges you'll configure:
Remote Node CIDR (REMOTE_NODE_CIDR): The IP range where your customer's physical infrastructure resides. This is the network range of the actual machines that will become Kubernetes nodes. EKS needs to communicate with these node IPs for cluster management. In other words, the VPC that your EKS cluster is in needs a route to this IP range.
Remote Pod CIDR (REMOTE_POD_CIDR): The IP range that will be allocated to pods running on the hybrid nodes. This is a virtual overlay network managed by the CNI (Cilium), separate from the physical node network. Each pod gets an IP from this range. Note: While you could technically use a physical network segment for pod IPs, we recommend overlay networks to avoid IP conflicts, simplify routing, and maintain network isolation between pods and infrastructure.
Key differences:
  • Node CIDR = Infrastructure network (where servers / nodes live)
  • Pod CIDR = Application network (where containers / pods get IPs)
  • Pod CIDR is typically larger since you'll have more pods than nodes
Conflict avoidance: Ensure all CIDRs (VPC, node, pod) are non-overlapping. Document the allocation for each customer environment.
Example planning:
  • AWS VPC: 10.0.0.0/16
  • Customer A nodes: 192.168.1.0/24, pods: 10.100.0.0/16
  • Customer B nodes: 192.168.2.0/24, pods: 10.101.0.0/16

2.2 Network Connectivity Options

Establish secure connectivity using one of these options:
  • AWS Site-to-Site VPN or 3rd party VPN solution: Provides encrypted tunnels over the internet
  • AWS Direct Connect: Dedicated network connection for higher bandwidth and consistent performance
Refer to the AWS VPN User Guide  for detailed setup instructions.

2.3 AWS Configuration

Configure routing and security:
  • Route Tables: Add routes from your VPC to customer node CIDRs via VPN/Direct Connect gateway
  • Network ACLs: Ensure they don't block required traffic flows
  • Security Groups: Allow HTTPS (443) from both node and pod CIDRs to EKS cluster security group
Test connectivity before proceeding with node registration.

3 Adding a Hybrid Node

Now that the cluster and connectivity between the environments are set up, we will onboard the first customer node to the cluster. This process can be repeated for any number of nodes joining the cluster.
High-level process:
3.1 Configure EKS cluster with customer network CIDRs and update security groups
3.2 Create SSM activation credentials for customer nodes to register with AWS
3.3 Customer installs nodeadm and joins their infrastructure to your EKS cluster

3.1 Update the EKS Cluster

EKS needs to know about the customer's network topology to properly route traffic and manage hybrid nodes. This step configures the cluster with the customer's network CIDRs and opens the necessary security group rules for communication. If one or more remote networks are already configured on the cluster, retrieve them with this command:
1
aws eks describe-cluster --name $CLUSTER_NAME --region $AWS_REGION --query "cluster.remoteNetworkConfig"
If this is your first remote network: Replace REMOTE_NODE_CIDR and REMOTE_POD_CIDR in the command below with your actual CIDR blocks.
If you already have remote networks configured: Copy the output from the previous command, add your new CIDRs to the existing arrays, and replace the entire --remote-network-config value in the command below with your updated configuration.
1
2
3
4
aws eks update-cluster-config \
--name $CLUSTER_NAME \
--region $AWS_REGION \
--remote-network-config '{"remoteNodeNetworks":[{"cidrs":["REMOTE_NODE_CIDR"]}],"remotePodNetworks":[{"cidrs":["REMOTE_POD_CIDR"]}]}'
Update Cilium for new pod CIDR:
When adding a new customer with a different pod CIDR, update Cilium's configuration to include the new CIDR range. Cilium needs to know about all pod networks it manages to properly allocate IP addresses and route traffic between pods.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# Get current Cilium values
helm get values cilium -n kube-system > current-cilium-values.yaml

# Update Cilium values to include new pod CIDR
# Add the new REMOTE_POD_CIDR to the clusterPoolIPv4PodCIDRList
cat > cilium-values-update.yaml << EOF
ipam:
mode: cluster-pool
operator:
clusterPoolIPv4MaskSize: 25
clusterPoolIPv4PodCIDRList:
- "10.80.2.0/23" # Existing customer CIDRs (if any)
- "REMOTE_POD_CIDR" # New customer CIDR (replace with actual value)
EOF

# Apply the updated configuration
helm upgrade cilium cilium/cilium \
--version 1.18.2 \
--namespace kube-system \
--values cilium-values-update.yaml \
--reuse-values

# Verify the update
helm get values cilium -n kube-system
With Cilium configured, we can now proceed to onboard customer nodes to the cluster. The Cilium agent will automatically be deployed to new hybrid nodes as they join.

3.2: Create SSM Hybrid Activation

The first step is creating a AWS Systems Manager Hybrid Activation, which results in an unique activation ID and activation code, that your end customers use to register servers as hybrid nodes in your EKS cluster. These nodes need to be dedicated to your product, because they will be managed within AWS Systems Manager.
You can generate an AWS Systems Manager Hybrid Activation through the aws CLI. There are a number of arguments  you can use to configure the Hybrid Activation, and we call out three of them explicitly, because they are important for a smooth and efficient onboarding process:
  • You can restrict the number of hybrid nodes your end customers can register at your EKS cluster by setting --registration-limit (integer).
  • You can configure an expiration date of the Hybrid Activation of up to 30 days in the future with --expiration-date (timestamp), to adjust to the time needed by your end customer's IT.
  • You can provide a name for instances that register with the Hybrid Activation and e.g. inject tenant information, using the parameter --default-instance-name (string).
The following example creates a Hybrid Activation for tenant acme-corp, that enables the customer to register 3 serves as hybrid nodes until 5pm at the 17th of July 2025:
1
2
3
4
5
6
7
8
9
# Create SSM hybrid activation
aws ssm create-activation \
--region $AWS_REGION \
--default-instance-name acme-corp-hybrid-node \
--description "Activation for EKS hybrid nodes of customer acme-corp" \
--iam-role AmazonEKSHybridNodesRole \
--tags Key=EKSClusterARN,Value=${EKS_CLUSTER_ARN} \
--registration-limit 3
--expiration-date 2025-07-17T17:00:00
Besides the discussed parameters, there are parameters specifiying the IAM role for the node, the region and the EKS cluster ARN as tag of the instance, to match the condition restricting access to the deregistration API we discussed earlier.

3.3 Customer Node Registration

With the cluster configured and SSM activation ready, it's time for the customer to join their infrastructure to your EKS cluster. The nodeadm CLI is AWS's official tool that automates the entire node bootstrap process. It installs kubelet, containerd, and the Systems Manager agent, then registers the machine as a managed instance using SSM activation. Additionally, nodeadm handles the cluster join process that you'd normally do with kubeadm join, using the IAM role and EKS access entry for authentication instead of bootstrap tokens. The tool works on both x86_64 and ARM physical or virtual machines and automatically applies the eks.amazonaws.com/compute-type: hybrid label that identifies the node as a hybrid node to the Kubernetes scheduler.
Customer runs on their infrastructure:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# Install nodeadm and dependencies (requires sudo/root privileges)
# Use the Kubernetes version provided by your SaaS vendor
sudo nodeadm install 1.31 --credential-provider ssm

# Create node configuration file with values provided by SaaS vendor
cat > nodeConfig.yaml << EOF
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
cluster:
name: my-saas-cluster # Provided by SaaS vendor
region: us-west-2 # Provided by SaaS vendor
hybrid:
ssm:
activationCode: ABC123XYZ # Provided by SaaS vendor (from Step 2.1)
activationId: 12345678-abcd # Provided by SaaS vendor (from Step 2.1)
EOF

# Initialize and join the node to EKS cluster
sudo nodeadm init -c file://nodeConfig.yaml
Customer verification:
Customers should see these services running successfully after nodeadm completes:
1
2
3
4
5
6
# Check if nodeadm completed successfully
sudo systemctl status kubelet
sudo systemctl status containerd

# Check local node status
sudo journalctl -u kubelet --no-pager -l
SaaS vendor verification:
1
2
3
4
5
6
# Check nodes in cluster (will show "Not Ready" until CNI is installed)
kubectl get nodes

# Example output:
# NAME STATUS ROLES AGE VERSION
# mi-036ecab1709d75ee1 Not Ready <none> 1h v1.31.2-eks-94953ac
You should also see the hybrid node registered in the AWS Systems Manager console under "Managed instances" with the instance name you configured during activation.

What You've Built

You now have the foundational infrastructure for SaaS Anywhere:
  • Hybrid-enabled EKS cluster with proper IAM roles and network configuration
  • Cilium CNI configured for distributed pod networking across customer environments
  • Customer onboarding process that extends your Kubernetes control plane to their infrastructure
Your SaaS application can now schedule workloads on customer infrastructure while maintaining centralized control through your EKS cluster. Customers get data locality and compliance benefits without sacrificing the operational advantages of your SaaS platform.

Next: Production-Ready Operations

Having hybrid nodes is just the beginning. Goind forward, we'll tackle the operational challenges of running SaaS Anywhere in production with further deep dive articles:
  • Automated tenant provisioning with GitOps workflows for consistent customer onboarding
  • Cross-environment monitoring and observability patterns for distributed workloads
  • Application deployment strategies using Helm and Argo CD for multi-environment releases
  • Security and compliance patterns for managing workloads across trust boundaries
  • Troubleshooting and maintenance procedures for hybrid node lifecycle management
The goal is transforming this foundation into a production-ready SaaS Anywhere platform that scales operationally as you add customers and environments.
This post layed the fundations for offering SaaS Anywhere with Amazon EKS Hybrid Nodes. In future posts will explore tenant isolation strategies, SaaS control plane implementation patterns, and the operational practices that make distributed SaaS architectures successful at scale.
# kubernetes# saas# amazon-eks# multi-tenant
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article