vCluster Build Multi-Tenant Kubernetes on AWS

Modern platform teams often need to provide developers and application teams with Kubernetes environments without creating a dedicated physical Kubernetes cluster for every team.

Yevgeni Shapiro

Head Of Cloud | Driving Cloud-Native Innovation | Platform Engineering

Published Oct 1, 2026

Last modified Oct 2, 2026

Virtual Kubernetes Clusters · Multi-Tenancy · Platform Engineering · GitOps · AWS

Introduction

Modern platform engineering teams face a common challenge: how to provide development teams with flexible, isolated Kubernetes environments without the operational overhead and cost of provisioning a dedicated physical Kubernetes cluster for every team.

A common solution is namespace-based multi-tenancy. While Kubernetes namespaces provide useful boundaries for resource organization, access control, and quotas, they do not offer a complete cluster-level experience. Teams may still share cluster-scoped resources such as Custom Resource Definitions (CRDs), admission controllers, and other Kubernetes components.

This is where vCluster introduces a different approach to Kubernetes multi-tenancy.

By running multiple virtual Kubernetes clusters on top of a shared host cluster, vCluster enables platform teams to provide developers with independent Kubernetes API environments while maintaining centralized infrastructure management.

In this architecture, AWS provides the underlying infrastructure, Terraform manages infrastructure provisioning, K3s serves as the lightweight host Kubernetes cluster, vCluster creates isolated virtual Kubernetes environments, and GitOps automates application deployment and configuration.

The goal is to build a scalable, cost-efficient, and developer-friendly Kubernetes platform that supports self-service infrastructure and modern platform engineering practices.

1. Architecture Overview

The platform consists of several infrastructure and application layers, each responsible for a specific capability.

Core components

Amazon Web Services (AWS): Provides compute, networking, storage, and security infrastructure.
Terraform: Automates infrastructure provisioning and lifecycle management using Infrastructure as Code.
K3s: Provides a lightweight Kubernetes host cluster running on AWS EC2 instances.
vCluster: Creates multiple virtual Kubernetes clusters within the host cluster.
Argo CD: Implements GitOps-based continuous delivery and application lifecycle management.
Helm: Packages and deploys Kubernetes applications and platform components.
Kubernetes RBAC: Controls access to virtual clusters and their resources.
Prometheus and Grafana: Provide observability for platform infrastructure and workloads.

High-level architecture

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
flowchart TB
    A["Platform Engineering Team"]
    B["Git Repository"]
    C["Terraform"]
    D["AWS Cloud"]
    E["EC2 Instances"]
    F["K3s Host Kubernetes Cluster"]
    G["vCluster Platform"]
    H["Virtual Cluster: Team A"]
    I["Virtual Cluster: Team B"]
    J["Virtual Cluster: Team C"]
    K["Argo CD"]
    L["Application Workloads"]
    M["Prometheus / Grafana"]

    A --> B
    A --> C
    B --> K
    C --> D
    D --> E
    E --> F
    F --> G
    G --> H
    G --> I
    G --> J
    K --> H
    K --> I
    K --> J
    H --> L
    I --> L
    J --> L
    F --> M

The architecture separates infrastructure management from the developer experience.

The platform team controls AWS infrastructure, host cluster operations, virtual cluster provisioning, and governance. Application teams receive access to their own virtual Kubernetes environments and can deploy applications without managing the underlying infrastructure.

2. Provisioning AWS Infrastructure with Terraform

The first step is to create the infrastructure required to run the host Kubernetes cluster.

Terraform provides a declarative approach to managing the AWS resources and makes it possible to reproduce the environment consistently across development, staging, and production.

The infrastructure typically includes:

An AWS VPC with public and private subnets.
Internet Gateway and NAT Gateway, depending on networking requirements.
EC2 instances for K3s control-plane and worker nodes.
Security groups for Kubernetes API access and internal communication.
IAM roles and instance profiles.
EBS volumes for persistent storage.
CloudWatch integration for infrastructure monitoring.

A simplified Terraform example for provisioning an EC2 instance:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
resource "aws_instance" "k3s_node" {
  ami                    = var.ami_id
  instance_type          = var.instance_type
  subnet_id              = var.subnet_id
  vpc_security_group_ids = [aws_security_group.k3s.id]

  iam_instance_profile = aws_iam_instance_profile.k3s.name

  root_block_device {
    volume_size = 30
    volume_type = "gp3"
    encrypted   = true
  }

  tags = {
    Name        = "k3s-platform-node"
    Environment = "development"
    ManagedBy   = "Terraform"
  }
}

For production environments, infrastructure should be designed with availability and security in mind. Consider using multiple Availability Zones, private networking, encrypted storage, restricted administrative access, and automated recovery procedures.

Terraform state should be stored remotely with encryption, access control, and state locking supported by the selected backend.

3. Deploying K3s as the Host Kubernetes Cluster

K3s is a lightweight, certified Kubernetes distribution designed for environments where a smaller operational footprint is useful.

For this platform, K3s serves as the host cluster on which virtual Kubernetes clusters will run.

A basic installation of the initial K3s server can be performed using:

1
2
3
4
5
curl -sfL https://get.k3s.io | \
  INSTALL_K3S_EXEC="server \
  --cluster-init \
  --write-kubeconfig-mode 600" \
  sh -

To join additional server or agent nodes, retrieve the cluster token securely from the initial server and configure the joining nodes with the appropriate server endpoint.

For a multi-node setup, consider:

High availability for the Kubernetes control plane.
Reliable networking between host nodes.
Persistent storage for virtual cluster workloads.
Resource requests and limits for host-level services.
Backup and recovery of host cluster state.
Secure access to the Kubernetes API.

The host cluster is a critical part of the platform. Although virtual clusters provide logical separation, they still depend on the reliability, security, and capacity of the underlying infrastructure.

4. Installing vCluster

Once the host Kubernetes cluster is operational, vCluster can be installed to provide virtual Kubernetes environments.

The vCluster CLI simplifies creating and managing virtual clusters.

Install the CLI:

1
2
3
4
5
curl -L -o vcluster \
  "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64"

chmod +x vcluster
sudo mv vcluster /usr/local/bin/

Check the installation:

1
vcluster version

Create a namespace for the first tenant:

1
kubectl create namespace team-a

Create a virtual cluster:

1
2
vcluster create team-a \
  --namespace team-a

The command provisions a virtual Kubernetes control plane in the host cluster and configures access to the virtual cluster.

Connect to it:

1
2
vcluster connect team-a \
  --namespace team-a

Verify the environment:

1
2
3
kubectl get nodes
kubectl get namespaces
kubectl get pods -A

The virtual cluster presents its own Kubernetes API and logical cluster resources. Its control-plane components run within the host cluster, while workloads can be scheduled onto the host infrastructure according to the configured vCluster synchronization and isolation model.

The exact behavior depends on the vCluster version, configuration, and enabled features.

5. Multi-Tenancy with Virtual Kubernetes Clusters

One of the main benefits of vCluster is its ability to provide independent Kubernetes API environments for different teams.

For example:

Tenant	Virtual cluster	Purpose
Team A	vcluster-team-a	Application development
Team B	vcluster-team-b	QA and integration testing
Team C	vcluster-team-c	Data engineering
Platform Team	Host cluster	Shared platform services

Each virtual cluster can have its own:

Kubernetes namespaces.
Service accounts and RBAC configuration.
Custom Resource Definitions, subject to the supported synchronization and isolation model.
Application workloads.
Deployment configurations.
Kubernetes API access.
Resource policies and quotas.

However, virtual clusters do not automatically guarantee complete isolation.

The host cluster still shares compute, memory, networking, storage, and potentially other platform-level resources. Security boundaries must be designed carefully, especially for workloads that are untrusted or subject to strict compliance requirements.

For stronger isolation, combine vCluster with:

Kubernetes resource quotas and limit ranges.
Network policies.
Pod Security Standards.
Restricted service account permissions.
Node isolation or dedicated node pools.
Admission policies.
Host cluster monitoring and auditing.

The result is a multi-tenant model that offers greater Kubernetes API independence than namespace-based tenancy while retaining shared infrastructure efficiency.

6. GitOps with Argo CD

GitOps is an important component of a modern platform engineering architecture.

Instead of manually deploying applications into each virtual cluster, teams can use Git repositories as the source of truth for application configurations.

Argo CD continuously monitors repositories and synchronizes the desired state with the target Kubernetes environments.

A typical GitOps workflow looks like this:

A developer commits application manifests or Helm values to Git.
Argo CD detects the repository change.
Argo CD compares the desired state with the live state.
The application is deployed or updated in the selected virtual cluster.
The deployment status is reported back to the platform team.

A simplified Argo CD Application definition:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: team-a-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/example/platform-apps.git
    targetRevision: main
    path: applications/team-a
  destination:
    server: https://team-a-vcluster.example.com
    namespace: application
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

The destination server must be configured as a reachable cluster endpoint, and Argo CD must have the appropriate credentials and permissions.

For larger platforms, an application-of-applications or app-of-apps pattern can be used to manage multiple tenant environments from a central GitOps repository.

This enables platform teams to standardize deployment patterns while allowing application teams to manage their own workloads.

7. Developer Self-Service and Platform Engineering

A central objective of platform engineering is to reduce the cognitive load on developers.

Instead of asking infrastructure teams to provision clusters, configure access, and deploy platform components manually, developers can request environments through a self-service workflow.

A possible provisioning process:

A developer requests a new environment through an internal portal or Git repository.
An automation workflow validates the request.
Terraform or Kubernetes-based automation provisions the required supporting infrastructure or tenant configuration.
vCluster creates the virtual Kubernetes environment.
RBAC policies and resource quotas are applied.
Argo CD registers or targets the environment.
The developer receives the connection information and deployment instructions.

This model can be extended with tools such as Backstage, Crossplane, Argo Workflows, or custom automation services.

The platform team remains responsible for governance, security, reliability, and cost management, while developers gain faster access to the environments they need.

8. Observability and Resource Management

Multi-tenant Kubernetes platforms require visibility across both the host cluster and individual virtual environments.

Observability should cover infrastructure, host cluster components, virtual cluster control planes, and tenant workloads.

Prometheus can collect metrics from the host cluster and supported virtual cluster components. Grafana provides dashboards for resource utilization, workload health, and operational trends.

Useful metrics and monitoring areas include:

CPU and memory utilization per tenant.
Pod scheduling failures.
Node capacity and resource pressure.
Virtual cluster control-plane health.
Application availability.
Storage consumption.
Network traffic.
API request rates and errors.

Resource quotas and limit ranges help prevent one tenant from consuming excessive resources.

For example:

1
2
3
4
5
6
7
8
9
10
11
12
apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-a-quota
  namespace: application
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 8Gi
    limits.cpu: "8"
    limits.memory: 16Gi
    pods: "30"

This quota controls resources requested by workloads in the specified namespace. For tenant-wide enforcement, apply the relevant policies across the tenant's namespaces or use supported virtual cluster and host-level controls.

A key operational principle is to monitor both logical tenant utilization and actual host resource consumption. Virtual cluster resource accounting does not necessarily map one-to-one to physical infrastructure consumption.

9. Security and Governance

A shared Kubernetes infrastructure introduces important security considerations.

Although vCluster creates separate Kubernetes API environments, workloads ultimately run on shared infrastructure unless additional isolation is configured.

A secure implementation should include:

Identity and access management

Use AWS IAM for infrastructure-level access and Kubernetes RBAC for cluster and tenant permissions. Avoid distributing host cluster administrator credentials to tenant teams.

Network security

Restrict access to the host cluster API and virtual cluster endpoints. Use network policies and carefully controlled ingress and egress rules.

Workload security

Apply Pod Security Standards, image scanning, admission controls, and least-privilege service account permissions.

Secrets management

Use an appropriate secrets management solution, such as AWS Secrets Manager or AWS Systems Manager Parameter Store, with controlled access and secure synchronization into Kubernetes.

Audit and compliance

Enable Kubernetes audit logging and centralize relevant application, platform, and infrastructure logs. Retain logs according to operational and compliance requirements.

Backup and recovery

Back up important host cluster state, persistent application data, and virtual cluster configuration. Test restoration procedures regularly.

Security boundaries should be validated against the actual threat model. For tenants running mutually untrusted workloads, dedicated clusters or stronger workload isolation may still be necessary.

10. Benefits and Trade-offs

Using vCluster on a shared K3s host cluster introduces several architectural advantages:

Infrastructure efficiency: Multiple virtual clusters can share the underlying compute and storage infrastructure.
Developer autonomy: Teams receive independent Kubernetes API environments.
Faster provisioning: Virtual clusters can be created without provisioning an entire new physical cluster.
Consistent governance: Platform teams can centrally manage infrastructure, policies, and shared services.
GitOps integration: Application configuration and deployment can be managed through version-controlled workflows.
Flexible environments: Development, testing, and ephemeral environments can be provisioned according to team needs.

There are also trade-offs to consider:

Shared failure domains: A host cluster outage can affect multiple virtual clusters.
Resource contention: Tenants compete for shared infrastructure unless resource and scheduling policies are enforced.
Operational complexity: The platform team must manage both host and virtual cluster layers.
Security boundaries: Virtual clusters are not equivalent to separate physical or fully isolated managed Kubernetes clusters.
Control-plane overhead: Running many virtual clusters introduces additional resource and operational costs.

The architecture should be evaluated based on tenant isolation requirements, workload profiles, operational maturity, and total cost of ownership.

Conclusion

Combining AWS, Terraform, K3s, vCluster, and GitOps provides a practical foundation for building a multi-tenant Kubernetes platform.

AWS supplies the infrastructure, Terraform makes it reproducible, K3s provides the host cluster, vCluster enables independent virtual Kubernetes environments, and Argo CD automates application delivery.

Together, these technologies support a platform engineering model in which infrastructure is managed centrally while application teams gain greater autonomy and faster access to Kubernetes environments.

For organizations operating multiple development teams, CI/CD environments, testing workloads, or internal developer platforms, virtual Kubernetes clusters can offer an effective balance between operational control and developer flexibility.

The broader goal is not simply to run more Kubernetes clusters. It is to build a platform that makes infrastructure easier to consume, safer to operate, and more consistent to manage.

Build once. Share infrastructure. Empower every team.

# amazon-eks # infrastructure-as-code # github # digital-transformation # kubernetes

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

vCluster Build Multi-Tenant Kubernetes on AWS

Yevgeni Shapiro

Introduction

1. Architecture Overview

Core components

High-level architecture

2. Provisioning AWS Infrastructure with Terraform

3. Deploying K3s as the Host Kubernetes Cluster

4. Installing vCluster

5. Multi-Tenancy with Virtual Kubernetes Clusters

6. GitOps with Argo CD

7. Developer Self-Service and Platform Engineering

8. Observability and Resource Management

9. Security and Governance

10. Benefits and Trade-offs

Conclusion