A 2020 cloud-native manual and a 2022 awesome list, read against AWS
A 2020 cloud-native manual and a 2022 awesome list, read against AWS
What an EKS node runs today
The EKS AMI release published on 1 October 2026 (
v20260930, from awslabs/amazon-eks-ami) builds only AL2023 images for Kubernetes 1.31 through 1.37: standard, NVIDIA and Neuron, on x86_64 and arm64. Its package table for 1.36 lists:containerd2.2.7runcamazon-ssm-agent
The AL2023 build script installs containerd, nerdctl (
dnf install -y nerdctl) and the SOCI snapshotter.crictl is not something the AMI build installs. None of the 74 files under the AMI's AL2023 and shared template directories mentions
crictl or cri-tools. Run command -v crictl on your own node before a runbook depends on it. ctr ships with containerd, and nerdctl is installed by the build script.The node's containerd config comes from
nodeadm. Its template names a single runtime handler and makes it the default:runc(binary/usr/sbin/runc) on standard AMIsnvidia(binary/usr/bin/nvidia-container-runtime) on any AMI where that binary exists, which means the NVIDIA AMIs
That has a consequence for RuntimeClass. The API is
node.k8s.io/v1; v1beta1 was removed in 1.25 according to the lifecycle file. A RuntimeClass's handler must name a runtime that containerd on the node actually has configured. On a stock EKS node there is exactly one handler, so a RuntimeClass with handler: runc matches standard nodes but not NVIDIA nodes, where the only handler is named nvidia. A Pod that points at a missing handler is created but never starts. RuntimeClass only becomes useful on EKS when you add a second runtime yourself, which managed node groups do not support. See the runtime layer page for why gVisor and Kata have no managed path.Getting to crictl on a node
These were checked against crictl 1.36.0's help output. They were not run against an EKS node.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
# reach an AL2023 node without SSH (the SSM agent is in the AMI's package table)
aws ssm start-session --target i-0123456789abcdef0
# on the node: name the endpoint; crictl now warns that its default list is deprecated
export CONTAINER_RUNTIME_ENDPOINT=unix:///run/containerd/containerd.sock
sudo -E crictl pods --namespace payments --name api
sudo -E crictl ps --all --namespace payments --name api -o table
sudo -E crictl logs --previous --tail 200 <container-id>
sudo -E crictl inspectp -o json <pod-id>
sudo -E crictl rmi --prune # removes unused images; -q is its short form
# the tools the AMI build itself installs, in the namespace the kubelet uses
sudo nerdctl --namespace k8s.io ps
sudo ctr --namespace k8s.io images ls
Access is different on other node types:
- Bottlerocket. There is no shell on the host. You go in through the control container, then the admin container, then
sheltiefor a root shell in the host namespaces. - EKS Auto Mode. You get no shell on the node at all.
kubectl debug node/<name> -it --image=<image> --profile=sysadminis the route. kubectl 1.33's help listssysadminamonglegacy,general,baseline,netadminandrestricted.
The Bottlerocket and Auto Mode access paths come from AWS documentation known before this review. They were not re-checked here.
What the CRI notes say that is now wrong
| the notes say | now | how it was checked |
|---|---|---|
| the kubelet has two runtime modules, dockershim and remote | dockershim was removed in Kubernetes 1.24. The kubelet has only the remote CRI client, and EKS AMIs run containerd 2.2.7 with runc | AMI release v20260930 package table |
| the CRI code lives at the 1.18 staging path; the proto shown is the 1.18 one | the CRI API is runtime.v1 only. The cri-api module's pkg/apis/runtime lists v1 and v1alpha2 at v0.25.0, and only v1 at v0.26.0 (Kubernetes 1.26). containerd 2.x serves only v1 | GitHub contents API on kubernetes/cri-api tags |
| RuntimeService has 22 RPCs, ImageService 5 | the master proto has 37 and 6. New since then: CheckpointContainer, CheckpointPod, RestorePod, GetContainerEvents, metrics RPCs (ListMetricDescriptors, ListPodSandboxMetrics), RuntimeConfig, UpdatePodSandboxResources, the Pod-sandbox stats RPCs, and Stream* variants of the list calls | rpc lines counted in api.proto on kubernetes/cri-api master, 7 October 2026 |
| for exec, the runtime returns a URL and the API server connects to the runtime's stream server directly, bypassing the kubelet | that was the redirect mode, removed with --redirect-container-streaming in 1.20. The kubelet now proxies the stream | from knowledge, not re-checked |
| install crictl v1.17.0 from the cri-tools release tarball | the URL pattern still works (crictl-v1.37.0-linux-amd64.tar.gz is the current asset name), but match crictl's minor version to the cluster. crictl 1.36 refuses an endpoint that does not serve CRI v1 ("validate CRI v1 runtime API") | GitHub releases API; crictl 1.36 run against a missing socket |
| crictl finds the runtime on its own | in the cri-tools v1.17 era the default endpoint was the dockershim socket (from knowledge). crictl 1.36 tries containerd, CRI-O, then cri-dockerd, and says this default "is now deprecated and the endpoint should be set" | crictl 1.36 --help |
| the kubelet is told which runtime to use with flags | --container-runtime was removed (from knowledge). The endpoint is the KubeletConfiguration field containerRuntimeEndpoint, and nodeadm writes it on EKS | config/v1beta1/types.go in kubernetes/kubelet v0.34.0 |
The notes do not mention frakti, rktlet or CRI-O's status, but the exajobs list's Container Runtime section does. Its status today:
- frakti is archived under kubernetes-retired (last push December 2020).
- rkt is archived (February 2020).
- hyperd is archived (November 2018).
- railcar is archived (October 2019).
- containerd/cri, the separate plugin repository, is archived. CRI is now built into containerd.
- CRI-O is active (pushed on the day of this review), but it is not an option on EKS-provided AMIs.
2. Kubeflow on AWS: the distribution AWS stopped shipping
What the two notes install, and why none of it applies
The "building Kubeflow" note has two paths, and both are gone:
- Outside China: download
kfctland applykfctl_k8s_istio.v1.0.0.yaml. kubeflow/kfctl is archived (last push August 2023). - Inside China: clone the author's re-hosted manifests (shikanon/kubeflow-manifests, last push July 2022), with images mirrored to a Chinese registry.
Its version table stops at Kubernetes 1.16, and it marks 1.16 incompatible with every Kubeflow release then available.
Storage in both paths is Rancher's local-path provisioner, made the default through the beta annotation
storageclass.beta.kubernetes.io/is-default-class. kubectl still reads the beta annotation, but the GA key is storageclass.kubernetes.io/is-default-class. On EKS, local-path would put pipeline artefacts and notebook home directories on a node's root volume, and they would be lost when Karpenter or a node-group update replaced the node.The "core components" note's YAML would also fail as printed:
- The TensorBoard Deployment uses
apps/v1beta1and the TF Serving Deployment usesextensions/v1beta1. Both were removed in 1.16 according to thek8s.io/apilifecycle files. - The TFJob is
kubeflow.org/v1beta2. Kubeflow Trainer v2 has no TFJob at all: its master branch keeps TensorFlow only underdocs/legacy-v1/, and its built-in runtimes are torch, DeepSpeed, JAX, MLX and XGBoost. TFJob survives only in the v1 line (tag v1.9.4, August 2026). - The pipeline example uses
dsl.ContainerOpwithfile_outputs.kfp.dslon master hascomponentandcontainer_componentand noContainerOp.
What survives is the framing. The note calls Kubeflow a cloud-native "machine learning gift bag" (translated). The four stages it organises Kubeflow around (business understanding, data, training and evaluation, deployment) are a fair way to read the mapping table below.
Resource sizing is the one place the 2020 note and 2026 agree. It asks for at least one worker with 4 CPUs, 12 GB of memory and 50 GB of storage. The current Kubeflow community distribution README totals its components at 4,377 millicores, 12,316 MiB and 65 GB of PVCs.
The AWS distribution: not archived, not maintained
Every check below was made on 7 October 2026.
- awslabs/kubeflow-manifests is not archived. Its README still introduces "Kubeflow on AWS" as a distribution with AWS service integrations.
- Its last release is
v1.7.0-aws-b1.0.3, published 1 September 2023. The releases before it are 1.7.0 builds andv1.6.1-aws-b1.0.2. - Its compatibility page tests Kubeflow 1.7 on EKS 1.23, 1.24 and 1.25 only. The current EKS AMI release builds 1.31 to 1.37, so none of the tested versions is among the ones AWS still builds nodes for.
- Issue #794 is titled "[Not Planned] AWS Distribution for Kubeflow-1.8". It says AWS "is not planning to participate in distribution testing at this time" (quoted from the issue). Users asked whether that meant later or never, and the issue records no answer.
- Commits since 2024 are dependency bumps and one workflow security fix (August 2026).
- kubeflow.org's installation page lists no AWS distribution. It lists Canonical, prokube.ai, Microsoft Azure (AKS, 26.03), Nutanix, QBO and Red Hat. A 2023 pull request to move AWS to a "legacy" list was closed without merging.
- deployKF, the community alternative suggested in that issue, was last pushed in August 2024.
So the honest status is this. AWS never announced an end of support, and the repository was never archived. In practice it stopped at Kubeflow 1.7 and is not usable on any EKS version AWS still builds nodes for. If you run Kubeflow on EKS today, you deploy the upstream community distribution (
26.03.1, June 2026, CI-tested on Kubernetes 1.36 with kind) and do the AWS integration yourself:- gp3 through the EBS CSI driver for read-write-once PVCs
- EFS for shared notebook volumes
- an ALB in front of the Istio ingress gateway
- Cognito or another OIDC provider behind Dex or oauth2-proxy
- S3 and RDS for artefacts and metadata
The AWS distribution used to wire up those integrations for you.
The SageMaker components for Kubeflow Pipelines were removed as well. They lived in kubeflow/pipelines under
components/aws/sagemaker. The directory exists at tag 2.16.0 (42 entries) and is gone at 2.17.0. PR #13481, merged on 5 June 2026, removed every component directory except Google Cloud's, and pointed to a new kubeflow/pipelines-components repository. That repository has no AWS or SageMaker files. The components' own changelog ends at version 2.3.0, which added the Model Monitor components in 2023.The live way to drive SageMaker from Kubernetes is the ACK SageMaker controller (aws-controllers-k8s/sagemaker-controller, v1.10.1, 18 September 2026). It lets a KFP or Argo step create a
TrainingJob or Endpoint custom resource.Each Kubeflow component, and what replaces it on AWS
Component versions are from the community distribution README. AWS operation names are from the botocore 1.43.107 models.
| Kubeflow component (2026 version) | AWS managed service (API) | what you keep | what you give up |
|---|---|---|---|
| Central dashboard, profiles, Istio, Dex / oauth2-proxy | SageMaker AI domain and Studio (CreateDomain, CreateSpace) with IAM Identity Center or IAM | per-user workspaces, single sign-on | namespace-per-team isolation you define, and Istio authorization policies you write |
| Notebooks v1.11 | Studio spaces: AppType includes JupyterLab and CodeEditor (CreateNotebookInstance still exists for the older model) | Jupyter, custom images | notebook Pods on the same cluster, PVCs and GPU pool as the training jobs |
| Pipelines 2.17.2 (Argo underneath) | SageMaker Pipelines (CreatePipeline, StartPipelineExecution) | a DAG with caching and lineage | the KFP SDK and its portability. Steps become SageMaker jobs, not arbitrary Pods on your cluster |
| Katib v0.19.0 | automatic model tuning (CreateHyperParameterTuningJob; strategies Bayesian, Random, Hyperband, Grid) | search spaces, early stopping | Katib's other algorithms and neural-architecture search (from knowledge), and tuning anything that is not a SageMaker training job, such as the RAG top_k sweep on the Kubeflow GenAI page |
| Trainer v2.3.0 (TrainJob; TFJob only in v1) | training jobs (CreateTrainingJob with EnableManagedSpotTraining, CheckpointConfig), or HyperPod with EKS orchestration (CreateCluster, Orchestrator.Eks, NodeRecovery: Automatic) | PyTorch, DeepSpeed and JAX code | with training jobs, the cluster disappears and so do your Kubernetes scheduling rules. With HyperPod on EKS you keep Kubernetes and still run an operator; AWS adds health checks and node replacement. See the AI Infra page |
| KServe v0.21.0 (+ Knative 1.23) | real-time endpoints with inference components (CreateEndpoint, CreateInferenceComponent); or KServe on EKS, self-managed | model-server containers (vLLM, Triton) | the InferenceService resource, Knative revisions and canaries in your cluster. KServe has no EKS add-on |
| Kubeflow Hub v0.3.17 (model registry) | SageMaker Model Registry (CreateModelPackageGroup, CreateModelPackage) | versions, approval status, lineage | a registry that is not tied to one cloud |
| ML Metadata, experiment tracking | managed MLflow (CreateMlflowTrackingServer, TrackingServerSize Small/Medium/Large; CreateMlflowApp also exists) | the MLflow client API | nothing much, if your team already used MLflow rather than KFP's metadata view |
| Spark Operator 2.5.2 | EMR on EKS (emr-containers StartJobRun) | Spark on the same cluster | the SparkApplication CRD; EMR release labels set the Spark version |
| Feast (feature and vector store) | Feature Store (CreateFeatureGroup); for vectors, Bedrock Knowledge Bases (KnowledgeBaseStorageType includes S3_VECTORS, OPENSEARCH_SERVERLESS) | online and offline features | a single store that serves both features and vectors |
| TensorBoard Deployment | Studio's TensorBoard app type | the same UI | nothing |
| MinIO, MySQL (2020 kustomize) | S3, RDS | S3-compatible artefact paths | nothing. Note that minio/minio itself is archived on GitHub (April 2026) |
Do not move it all at once. Pipelines and the notebooks are the parts teams miss most after leaving Kubeflow, because SageMaker Pipelines is a different SDK. Katib, the model registry and metadata are the easiest to hand over to AWS.
The GenAI page, mapped
The Kubeflow GenAI use-cases page names no cloud. On AWS, each of its stages has a managed counterpart:
| Kubeflow GenAI stage | Kubeflow piece | AWS managed counterpart (API checked) |
|---|---|---|
| synthetic data generation | Pipelines | SageMaker Pipelines processing steps; Bedrock for the generator model |
| RAG indexing and retrieval | Feast with Milvus | Bedrock Knowledge Bases (CreateKnowledgeBase) over S3 Vectors (s3vectors CreateIndex) or OpenSearch Serverless |
| RAG preprocessing at scale | Spark Operator | EMR on EKS or EMR Serverless |
| LLM fine-tuning | Trainer BuiltinTrainer / TorchTune | Bedrock model customisation (CustomizationType: FINE_TUNING, CONTINUED_PRE_TRAINING, DISTILLATION, REINFORCEMENT_FINE_TUNING) for supported models; SageMaker training or HyperPod for anything else |
| hyperparameter optimisation | Katib | automatic model tuning, for training jobs only |
| inference at scale | KServe | SageMaker inference components; Bedrock for hosted models |
| evaluation | pipeline steps; BLEU and ROUGE | Bedrock evaluation jobs (CreateEvaluationJob) |
One point of disagreement with the Kubeflow page: it tunes RAG
top_k and temperature to maximise BLEU. BLEU scores n-gram overlap with a reference, so it rewards copying the reference's wording, not grounding the answer in the retrieved text. A retrieval-grounded metric, or a judged evaluation, is a better target, whichever platform runs it.Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article