How to install an air gapped (no internet access) kubernetes cluster using aws ec2
in this tutorial I will be walking through creating an airgapped environment using a subnet with a routing table that only has local access (no route to an internet gateway or nat gateway) , downloading the files needed to a bastion and transferring them to the first machine that will be used as the control plane and as a worker node and then we will be able to admit other nodes
1.let’s start by creating a new vpc with one public subnet , then go and create a new route table with the only route being the ip cidr block of the vpc
now create a subnet with this route table
2. now let’s create an ec2 instance that will be out bastion , it will be in the public subnet , and it doesn’t need to have a key , as we will access it through session manager (make sure that the role has AmazonSSMManagedInstanceCore policy , the AMI having systems manager agent installed or install it through user data and port 443 is open to the outbound)
3. now let’s create the first node of the cluster which will serve as both a control plane and a worker node , for this we will use the c6a.xlarge instance type with and ebs of 250 gb and the AMI being ubuntu 22, also the security group must have the following ports open tcp 2379, 2380, 6443, 10250, 10257 and 10259 and udp 8472
4. (skip this step if using ubuntu 22 ) now download containerd on the bastion machine in order to use scp to move the .tar.gz file to the first node , follow the steps in option one in this documentation by downloading the files first into the bastion then into the first node https://github.com/containerd/containerd/blob/main/docs/getting-started.md (ie. download containerd , runc and cni plugins into the bastion , use scp to move them and follow the instructions to install them ) , do this step bif you are using ubuntu 24 , see kurl documentation for more details https://kurl.sh/docs/install-with-kurl/system-requirements
5. now go to https://kurl.sh/ and choose the options that you need for your installation , once finished click on “Download airgap installer ” ,copy the curl command and then run it in the bastion machine to download the package and then move it to the main node using scp , now ssh into the node and install the cluster using the command cat install.sh | sudo bash -s airgap
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#(skip this step if using ubuntu 22 ) note that you might need to install a few dependencies first like flo and the packages needed for it , use this simple script in the bastion
sudo apt update
sudo apt clean
sudo apt install --download-only -y fio
scp -i <key>.pem /var/cache/apt/archives/*.deb ubuntu@<private ip>:/home/ubuntu/
#then ssh into the node and run
sudo dpkg -i *.deb
#the same might be needed for conntrack and containerd (again) , so you might need to run this script 3 times , once for each dependency
- after waiting for a few minutes the installation will be done and you can run kubectl commands , one important command will also appear in green.
this command is valid for 24 hours , after that you will need to generate a new one (see instructions in picture ) , to join a new node , simply launch a new ec2 instance with the same configurations , scp the big package you downloaded from kurl , unzip it then run this command
- now let’s run something on this cluster , if you run kubectl get pods -A , you will find a registry pod , this works as your local ecr , get it’s ip address by using the following command
1
2
3
4
5
6
7
kubectl get svc/registry -n kurl -o jsonpath='{.spec.clusterIP}'
and get it’s credentials by using the following command
json=$(kubectl get secret/registry-creds -o jsonpath='{.data.\.dockerconfigjson}')
echo $json | base64 --decodenow you can see the username and password you will use to login into this registry , since this is an airgapped cluster with no access to the internet you will need to bring your image using a .tar file and push it into this registry
8. for this example I will be using littlelink which I used before in another article that you can check here
we will also be using kompose (https://kompose.io/ ) to convert the docker compose to yaml files for deployment and service
now go back to your bastion machine and install docker from the official script https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository
then pull the image and get the .tar file
1
2
docker pull ghcr.io/timothystewart6/littlelink-server:latest
sudo docker save -o littlelink.tar ghcr.io/timothystewart6/littlelink-server:latestnow we will install kompose https://kompose.io/installation/ and then copy the docker compose config from the github repo ( https://github.com/timothystewart6/littlelink-server ) info a compose.yaml file and then run the kompose command “kompose convert -f compose.yaml”
now let’s scp the .tar and the kompose generated files into the node and then use containerd to load the image
sudo ctr -n=k8s.io images import littlelink.tarhowever , this will upload the image to containerd on the node and not to the registry , now we use containerd to upload it to the registry
use the following command to find the image
sudo ctr -n=k8s.io images lsnow tag the image with a new tag that includes the ip address of the registry
sudo ctr -n=k8s.io images tag ghcr.io/timothystewart6/littlelink-server:latest https://<registry ip address>/timothystewart6/littlelink-server:latestthen we push it into the registry
sudo ctr -n=k8s.io images push --platform linux/amd64 --skip-verify --user <registry username>:<registry password> <registry ip address>:443/timothystewart6/littlelink-server:latestnow the image is in the registry , now we need to create a secret that the deployment will use to pull the image from the registry
kubectl create secret docker-registry regcred --docker-server=https://<registry ip address>:443 --docker-username=<registry username> --docker-password=<registry password>now let’s go back to the files komposer created , we can apply the service file without any modifications , however we need to modify the deployment file before applying as we need to tell the deployment to use the registry secret and to pull the image from the registry
don’t forget to change the image tag to include the ip address of the registry
now we can apply the deployment file , and you can see the pod showing up , but …. there seems to be a problem , guess what ? Kubelet can’t pull the image from the registry :) fun …
so what we will do now ?
Well … the “correct way” is to use a certificate to connect to the registry , however , that’s a headache so what we will do instead is the following …
on each and every node , we will change kubelet’s configuration and then restart it …
this will make kubelet skip the certificate verification for the url of the registry (the ip address) and then kubelet will be able to pull the image with an “insecure” connection
I’ll include the config.toml file (and what to modify) below but what we will do is to create a config.toml file , copy it to a specific location then restart containerd
nano config.tomlsudo cp config.toml /etc/containerd/config.tomlsudo systemctl restart containerdbut before , copy the block from below and change the following lines :
63 this image should be in the images in kubelet , if you install a newer version of kurl you might need to change the tag
151 ,153,162,163,262 change the ip address only
151 ,153,162,163,262 change the ip address only
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
disabled_plugins = []
imports = []
oom_score = 0
plugin_dir = ""
required_plugins = []
root = "/var/lib/containerd"
state = "/run/containerd"
temp = ""
version = 2
[cgroup]
path = ""
[debug]
address = ""
format = ""
gid = 0
level = "warn"
uid = 0
[grpc]
address = "/run/containerd/containerd.sock"
gid = 0
max_recv_message_size = 16777216
max_send_message_size = 16777216
tcp_address = ""
tcp_tls_ca = ""
tcp_tls_cert = ""
tcp_tls_key = ""
uid = 0
[metrics]
address = ""
grpc_histogram = false
[plugins]
[plugins."io.containerd.gc.v1.scheduler"]
deletion_threshold = 0
mutation_threshold = 100
pause_threshold = 0.02
schedule_delay = "0s"
startup_delay = "100ms"
[plugins."io.containerd.grpc.v1.cri"]
device_ownership_from_security_context = false
disable_apparmor = false
disable_cgroup = false
disable_hugetlb_controller = true
disable_proc_mount = false
disable_tcp_service = true
drain_exec_sync_io_timeout = "0s"
enable_selinux = false
enable_tls_streaming = false
enable_unprivileged_icmp = false
enable_unprivileged_ports = false
ignore_deprecation_warnings = []
ignore_image_defined_volumes = false
max_concurrent_downloads = 3
max_container_log_line_size = 16384
netns_mounts_under_state_dir = false
restrict_oom_score_adj = false
sandbox_image = "registry.k8s.io/pause:3.10.1" #change if needed
selinux_category_range = 1024
stats_collect_period = 10
stream_idle_timeout = "4h0m0s"
stream_server_address = "127.0.0.1"
stream_server_port = "0"
tolerate_missing_hugetlb_controller = true
unset_seccomp_profile = ""
[plugins."io.containerd.grpc.v1.cri".cni]
bin_dir = "/opt/cni/bin"
conf_dir = "/etc/cni/net.d"
conf_template = ""
ip_pref = ""
max_conf_num = 1
[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "runc"
disable_snapshot_annotations = true
discard_unpacked_layers = false
ignore_rdt_not_enabled_errors = false
no_pivot = false
snapshotter = "overlayfs"
[plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
base_runtime_spec = ""
cni_conf_dir = ""
cni_max_conf_num = 0
container_annotations = []
pod_annotations = []
privileged_without_host_devices = false
runtime_engine = ""
runtime_path = ""
runtime_root = ""
runtime_type = ""
[plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
base_runtime_spec = ""
cni_conf_dir = ""
cni_max_conf_num = 0
container_annotations = []
pod_annotations = []
privileged_without_host_devices = false
runtime_engine = ""
runtime_path = ""
runtime_root = ""
runtime_type = "io.containerd.runc.v2"
BinaryName = ""
CriuImagePath = ""
CriuPath = ""
CriuWorkPath = ""
IoGid = 0
IoUid = 0
NoNewKeyring = false
NoPivotRoot = false
Root = ""
ShimCgroup = ""
SystemdCgroup = false
[plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
base_runtime_spec = ""
cni_conf_dir = ""
cni_max_conf_num = 0
container_annotations = []
pod_annotations = []
privileged_without_host_devices = false
runtime_engine = ""
runtime_path = ""
runtime_root = ""
runtime_type = ""
[plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options]
[plugins."io.containerd.grpc.v1.cri".image_decryption]
key_model = "node"
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = ""
[plugins."io.containerd.grpc.v1.cri".registry.auths]
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."10.96.2.113:443"] # edited line
[plugins."io.containerd.grpc.v1.cri".registry.configs."10.96.2.113:443".tls] # edited line
ca_file = "" # edited line
cert_file = "" # edited line
insecure_skip_verify = true # edited line
key_file = "" # edited line
[plugins."io.containerd.grpc.v1.cri".registry.headers]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."10.96.2.113:443"] # edited line
endpoint = ["https://10.96.2.113:443"] # edited line
[plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
tls_cert_file = ""
tls_key_file = ""
[plugins."io.containerd.internal.v1.opt"]
path = "/opt/containerd"
[plugins."io.containerd.internal.v1.restart"]
interval = "10s"
[plugins."io.containerd.internal.v1.tracing"]
[plugins."io.containerd.metadata.v1.bolt"]
content_sharing_policy = "shared"
[plugins."io.containerd.monitor.v1.cgroups"]
no_prometheus = false
[plugins."io.containerd.runtime.v1.linux"]
no_shim = false
runtime = "runc"
runtime_root = ""
shim = "containerd-shim"
shim_debug = false
[plugins."io.containerd.runtime.v2.task"]
platforms = ["linux/amd64"]
sched_core = false
[plugins."io.containerd.service.v1.diff-service"]
default = ["walking"]
[plugins."io.containerd.service.v1.tasks-service"]
rdt_config_file = ""
[plugins."io.containerd.snapshotter.v1.aufs"]
root_path = ""
[plugins."io.containerd.snapshotter.v1.btrfs"]
root_path = ""
[plugins."io.containerd.snapshotter.v1.devmapper"]
async_remove = false
base_image_size = ""
discard_blocks = false
fs_options = ""
fs_type = ""
pool_name = ""
root_path = ""
[plugins."io.containerd.snapshotter.v1.native"]
root_path = ""
[plugins."io.containerd.snapshotter.v1.overlayfs"]
mount_options = []
root_path = ""
sync_remove = false
upperdir_label = false
[plugins."io.containerd.snapshotter.v1.zfs"]
root_path = ""
[plugins."io.containerd.tracing.processor.v1.otlp"]
[proxy_plugins]
[stream_processors]
[stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
path = "ctd-decoder"
returns = "application/vnd.oci.image.layer.v1.tar"
[stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
path = "ctd-decoder"
returns = "application/vnd.oci.image.layer.v1.tar+gzip"
[timeouts]
"io.containerd.timeout.bolt.open" = "0s"
"io.containerd.timeout.shim.cleanup" = "5s"
"io.containerd.timeout.shim.load" = "5s"
"io.containerd.timeout.shim.shutdown" = "3s"
"io.containerd.timeout.task.state" = "2s"
[ttrpc]
address = ""
gid = 0
uid = 0
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
[plugins."io.containerd.grpc.v1.cri".registry.configs."10.96.2.113".tls]
ca_file = "/etc/kubernetes/pki/ca.crt"
and then restart containerd on the current node using the above command (reminder that you have to do this on every node )
kill the pod if needed and wait for the new pod to be healthy and then get the service ip address and curl it with the port
you should get an output like this
and that’s it , we’ve installed the cluster deployed an application and curled it
congratulations and thank you for reading , don't forget to follow me on linkedin.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article