The three ways, and what each one hands you
| Bedrock | SageMaker AI inference | EKS on GPU / Inferentia | |
|---|---|---|---|
| who owns the weights | AWS, or yours via Custom Model Import | yours, in S3 | yours, in S3 or FSx |
| who loads them | Bedrock | the endpoint, with Fast Model Loader streaming from S3 | your container, from a mount or a pull |
| the unit you pay for | tokens in and out, or provisioned throughput | instance-hours per endpoint, per inference component | instance-hours per node, whatever is on it |
| scaling | none to do; cross-region inference profiles spread load | inference-component autoscaling, including to zero | Karpenter for nodes, HPA or KEDA for pods |
| cold start | none visible | provision plus weight load; Fast Model Loader shortens the second | provision plus image pull plus weight load, all yours |
| isolation between tenants | the API boundary | one endpoint or component per tenant | namespaces, plus MIG or a slice per tenant, plus NetworkPolicy |
| guardrails | Bedrock Guardrails inline, or ApplyGuardrail on any text | ApplyGuardrail from your code | ApplyGuardrail from your code |
| what is not an AWS service | nothing | the inference container is AWS-built or yours | the model server (vLLM or similar) is yours |
| when it is the right one | the model is one Bedrock hosts and per-token pricing fits the traffic | the weights are yours and you want managed hosting | you need the runtime, the batching or the hardware sharing under your control |
The choice is not a maturity ladder. A team that fine-tuned a model and serves it at steady high volume is on EKS for cost, not for lack of ambition; a team whose traffic is bursty and whose model is one Bedrock hosts is on Bedrock for the same reason.
One request through each
The same request, a chat turn with a tool call, takes three shapes:
- Bedrock. The client calls API Gateway, which invokes a Lambda or an app pod holding an IAM role with
bedrock:InvokeModelon one model ARN. The call goes to the Bedrock Converse API with atoolConfig, a Guardrail ID and a cross-region inference profile. The response carriesusageand, when prompt caching applied, the cache read and write token counts. Model invocation logging writes the full request and response to CloudWatch Logs or S3; CloudTrail records that the role made the call. Nothing in the path is yours to patch. - SageMaker AI. The same client path ends at
sagemaker:InvokeEndpoint. The endpoint hosts an inference component that pulled the weights from S3 at scale-up. Guardrails are not inline; the app calls ApplyGuardrail on the prompt and the completion itself. CloudWatch carries the endpoint's invocation and latency metrics; CloudTrail carries the control-plane calls. - EKS. The client reaches an ALB owned by the load balancer controller, which forwards to a Service in front of model-server pods on a GPU node group. The pod's service account carries an EKS Pod Identity association for
s3:GetObjecton the weights bucket, or the weights are already on an FSx for Lustre file system linked to that bucket. The image came from ECR; the GPU came from a Karpenter NodePool on ap5,g6orinf2instance. CloudWatch Container Insights carries the GPU metrics; the app calls ApplyGuardrail, and CloudTrail sees only the AWS calls the pod makes, never the tokens.
The last line is the honest one. On Bedrock every token passes through a service AWS can log and govern. On EKS the tokens stay inside your pod, and the AWS-native controls are on the edges: identity into the bucket, the image scanner, the metrics, the network policy. That is the trade, and it is the reason the wishes below split the way they do.
Where the model lives, and how it gets to the GPU
Weights are the file no other workload has: tens to hundreds of gigabytes that every replica reads at start and never writes. On AWS that file has one home and three ways out of it:
- S3 holds the artifact. Everything else is a cache of it.
- FSx for Lustre linked to the bucket by a data repository association gives every node in an AZ a POSIX mount with the file already staged, if you run the import task before the first pod needs it. This is the fast path for a fleet that scales in and out.
- Mountpoint for Amazon S3 as a CSI driver mounts the bucket read-only into the pod with no file system to run; first-byte latency is S3's, and the local cache narrows it on the second replica on the same node.
- SageMaker Fast Model Loader does the same job for SageMaker endpoints: it streams weights from S3 straight to the accelerator in chunks rather than copying the whole file to disk first.
Which one you pick decides your cold start, and cold start decides whether scale-to-zero is a saving or a latency incident. That chain is the subject of three of the ten infrastructure wishes below.
The wishes, answered with what exists
The twelve AI wishes on the 100-wishes page
Wish 1 (Athena's
ST_Covers) and wish 11 (a confidence contract for Comprehend and Textract) are AI-adjacent, not LLM, and stay on their page . The other ten:- 2 — Bedrock should report what a prompt cache saved. ✓ The Converse
usageblock documents cache read and cache write input-token fields for models with prompt caching enabled. Read those two fields per call and the 50-turn session bills back exactly as the wish asks. If your calls show only input and output tokens, the model or the request has no cache point, which is a different finding from "the field does not exist". Check before filing. - 3 — Say which tools the model was shown. ◐ The response will not tell you. Model invocation logging will: it records the full request body,
toolConfigincluded, keyed by request ID. Join your own log line to it and a tool-less call is distinguishable after the fact. Not in the response, which is the wish, but in a service you can query. - 4 — AgentCore should reach the regions Bedrock reaches. ◐ Run the agent in an AgentCore region and let a cross-region inference profile carry the model calls to wherever the model is. The region constraint moves from the pair to one service, which is the most a workaround can do.
- 5 — A guardrail that sees tool arguments. ◐ ApplyGuardrail evaluates any text you hand it, independent of a model call. Serialise the
toolUseblock's arguments and evaluate them before the tool runs. The policy is still expressed in your code rather than in the guardrail, so ◐, but the guardrail engine is the one doing the judging. - 6 — A first-party token counter, offline. ◐ Bedrock's CountTokens API answers the size question for the models that support it without spending an inference call. It is still a round trip, and the wish said offline, so half.
- 7 — AgentCore Memory should say what it kept. ◐ The memory records are listable and retrievable through the AgentCore Memory data APIs, so "what do you believe" is answerable from the service. "Why did you keep it" is not, and that was the wish.
- 8 — Warm Code Interpreter sessions. ◐ Sessions persist across calls up to a timeout, so an agent that keeps one session ID for a whole turn pays start-up once, not per tool call. A warm pool across turns is still a wish.
- 9 — Model deprecation announced in the API. ◐
ListFoundationModelsreturns a model lifecycle status, active or legacy, and AWS Health publishes deprecation notices as events you can route through EventBridge. The status is in the API; the date is in Health; the wish wanted both in one place. - 10 — SageMaker and Bedrock should share one model-access story. ◐ IAM is the one story that exists:
bedrock:InvokeModelon a model ARN andsagemaker:InvokeEndpointon an endpoint ARN, both under one permission boundary or SCP, both in CloudTrail. Two verbs, one policy, one audit trail. The grant is still made twice. - 12 — Q Developer suggestions attributable to a policy. ○ Nothing in this page's scope answers it, and it is not a serving concern. Still a wish.
The ten AI-infrastructure wishes
These were written beside the GPU design in the lab repository this documentation grew out of, and they are about the EKS column of the table above. In their numbering:
- 1 — Capacity Blocks as a queryable API. ✓ EC2 has a describe-offerings call for Capacity Blocks that returns what is reservable by instance type, count and start window, and a launch that fails for capacity returns a distinct insufficient-capacity error code rather than a configuration error. The scheduler loop the wish describes is buildable on those two. Verify the API name in your SDK version.
- 2 — Topology-aware GPU provisioning. ◐ EKS labels nodes with their network-topology layer and zone, and a placement group under a Capacity Block keeps a job's instances close. Karpenter can be constrained on those labels; it does not yet reason about NVLink domains, and Kubernetes topology-aware scheduling through Kueue is the piece that does. Partly AWS, partly the scheduler you run.
- 3 — An EFA path-health and topology API. ◐ The EC2 instance-topology API reports the network node hierarchy an instance sits in, which answers "are these instances close". Whether a given GPU-to-NIC pair is on the direct PCIe path is still a benchmark, not a metric.
- 4 — A DCGM dashboard pack. ◐ CloudWatch Container Insights collects NVIDIA GPU metrics on EKS with prebuilt views, which is the AWS-native answer; the wish asked Managed Prometheus for the same, and that pack does not exist. If your dashboards are in Grafana, you still write the PromQL.
- 5 — HyperPod recovery as an event stream. ○ HyperPod publishes cluster and node events to CloudWatch, and some to EventBridge; the per-run "this node was cordoned and the job resumed from checkpoint N" join the wish asks for is not a documented stream. Watch that space; do not build on it.
- 6 — A managed model-weight cache with pre-warm. ◐ On SageMaker, Fast Model Loader is exactly this for endpoints. On EKS, FSx for Lustre with a data repository association and a pre-import task before scale-up is the same primitive, but you run the pre-warm job. Managed for one path, hand-built for the other.
- 7 — MIG and vGPU slices as native EKS resources. ◐ MIG profiles schedule as resources once the NVIDIA operator is configured with a MIG strategy, and the slice is the isolation. What is missing is the isolation class the wish wanted the scheduler and quotas to see; that arrives with Dynamic Resource Allocation, general availability in Kubernetes 1.34, which EKS offers. The driver is NVIDIA's, not AWS's.
- 8 — Default-deny tenant networking for GPU namespaces. ◐ The VPC CNI enforces NetworkPolicy natively, and security groups for pods put an AWS security boundary on a pod. Both are opt-in per namespace by a policy you write; a namespace is still open until someone writes it. Configuration closes it; the default does not.
- 9 — Scale-to-zero with warm weights. ◐ SageMaker inference components scale to zero, and Fast Model Loader shortens the path back up; the first request after zero still pays provisioning. On EKS, KEDA to zero and Karpenter to zero nodes work, and the cold path is wish 6.
- 10 — DRA as the supported way to request accelerators. ◐ Kubernetes 1.34 made DRA general availability, and EKS runs 1.34, so the API is there on a standard cluster. The wish also asked for AWS-maintained drivers, and the drivers are still NVIDIA's.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article