Hugging Face on AWS, services only
Source.huggingface/awesome-huggingface, the ecosystem catalogue Hugging Face itself publishes. Read on 2026-09-26 through the GitHub API rather than recalled: the repository is archived, its last push was 2024-04-29, and it carries 1,093 stars. Twenty sections, and the section named ☁ Cloud has exactly one entry — Amazon SageMaker. This page takes the catalogue as the inventory of what people actually build with, and rebuilds it on AWS-managed services only. Nothing here is deployed; the honesty section says what that means.
An archived catalogue is not a dead one. It is a dated one, and the date is the most useful thing about it: it fixes a moment — before the serving-engine consolidation, before AWS had a model API — and the difference between that moment and an AWS account today is exactly the design work this page is about.
The catalogue, read as an artifact
Nine "official libraries", seven NLP toolkits, four inference engines, five model-scalability projects, and one cloud row. Checked entry by entry through the API on the day of writing, the inference and scalability sections have aged hardest:
| catalogue entry | state on 2026-09-26 | what that tells you |
|---|---|---|
| lightseq | archived, last push 2023-05-16 | the hand-written CUDA serving kernels lost |
| FastSeq | archived, last push 2022-08-17 | same |
| nn_pruning | archived, last push 2022-06-21 | structured pruning stopped being a product decision |
| FasterTransformer | last push 2024-03-27 | NVIDIA moved the work into TensorRT-LLM |
| TurboTransformers | last push 2025-07-18 | alive, and niche |
| parallelformers, OSLO | last push 2023-04, 2023-09 | model-parallel wrappers folded into the frameworks |
| autonlp | redirects to autotrain-advanced, active | the only entry whose successor is obvious from the URL |
| knockknock | last push 2023-06-23 | see the row it becomes on AWS — it is two lines of EventBridge |
| transformers | last push 2026-09-27 | the library the whole catalogue orbits did not age at all |
The pattern is worth naming before any AWS service appears: the layers that consolidated are the layers a cloud can sell you, and the layers that did not are the ones you still assemble. Four separate inference engines in the catalogue are, in an AWS account today, a choice of container image. Five model-parallelism projects are a training-job configuration. The tokenizer is still the tokenizer.
The layer map
One row per layer of the Hugging Face stack, the AWS-managed service that carries it, and what the swap costs:
| layer | catalogue entries | AWS-managed answer | what is given up |
|---|---|---|---|
| model and dataset distribution | huggingface_hub, datasets | S3 — weights mirrored into a bucket, datasets as Parquet with a prefix layout | the Hub's discovery, model cards and community; you now own a mirror and its freshness |
| training | transformers, accelerate, trl, autotrain | SageMaker training jobs on the Hugging Face training DLC, or EKS with the same image; SageMaker HyperPod for runs measured in weeks | the job's scheduler is AWS's; a resume after a node loss is yours to write unless HyperPod is taken |
| distributed training | Deepspeed, fairscale, ColossalAI, OSLO, parallelformers | the same DLC on multi-node instances with EFA, placed in a cluster placement group | nothing about the library changes — but see GPUs on AWS: the collective silently falls back to TCP if one piece of the EFA path is missing |
| text generation serving | TurboTransformers, FasterTransformer, lightseq, FastSeq | the vLLM, SGLang or llama.cpp DLC on a SageMaker endpoint, on ECS, or on EKS | the engine is now someone's release cadence rather than your fork; see the engine section |
| embeddings and reranking | Sentence Transformers, SimCSE, WhiteningBERT | the TEI DLC on a SageMaker endpoint, or Bedrock embedding models | a Bedrock embedding model is not on the Hub, so a Hub-trained reranker and a Bedrock embedder are two vocabularies in one system |
| neural search | Jina Integration, Weaviate Integration, ColBERT | OpenSearch Service: BM25, k-NN and RRF in one cluster — the pattern search-platform already runs | no late-interaction (ColBERT-style) scoring; you get a single vector per document unless you model the multi-vector case yourself |
| question answering | Haystack | Bedrock Knowledge Bases over OpenSearch, or the retrieval path above plus a model call | the pipeline's components stop being inspectable individually |
| evaluation | Jury, Spotlight | SageMaker model evaluation / Bedrock evaluations; the metrics themselves in a training job | Spotlight's interactive data-segment exploration has no managed equivalent |
| training notifications | knockknock | EventBridge on the SageMaker job state-change event → SNS | nothing. This is the cleanest swap in the catalogue: a library import becomes a rule |
| adversarial robustness | TextAttack, TextFlint, OpenAttack | nothing | see what AWS has no answer for |
| accelerators | the Hardware section (Qualcomm, Intel) | Inferentia and Trainium through Optimum Neuron and the Neuron DLCs | the model has to compile: an architecture Neuron does not support is not a tuning problem, it is a different instance family |
Four ways to serve a Hub model, and one of them is not an AWS service
This is the decision the rest of the page hangs off, and it is not "which instance".
| option | what you operate | when it is right | what it costs you |
|---|---|---|---|
| Bedrock, a model AWS already hosts | nothing | the model you want is on the menu and you want tokens, not hours | you cannot bring a fine-tune of an architecture Bedrock does not import, and the model id is AWS's to deprecate |
| Bedrock Custom Model Import, your weights in S3 | nothing | you fine-tuned a supported architecture and want the Bedrock API around it | a long list of constraints, read out of the documentation below |
| SageMaker endpoint with a Hugging Face DLC | the endpoint's instance count | any Hub architecture, your container, managed scaling and a private endpoint | you pay for instance-hours whether or not anyone calls it; scale-to-zero is a different product shape |
| EKS with the same DLC | the cluster | you already run EKS, need per-pod control, GPU sharing or a custom scheduler | everything on GPUs on EKS becomes yours |
| Hugging Face Inference Endpoints | nothing | you want HF to run it | this is not an AWS service. It runs on AWS infrastructure under Hugging Face's account and contract, which is a different answer to "who is on call" and a different place for your data to sit |
The last row is in the table precisely because it is the one an AWS-only constraint rules out, and ruling it out silently would be dishonest: it is a good product, and it is not an AWS service.
The engine changed underneath the container
Most "Hugging Face on SageMaker" writing tells you to deploy the TGI container. Checked on 2026-09-26:
huggingface/text-generation-inference is archived and in maintenance mode, archived on 2026-03-21, with Hugging Face's own guidance pointing downstream to vLLM, SGLang, llama.cpp and MLX.The DLC list has already moved, and it is worth reading as a list rather than as a recommendation, because the versions are the interesting part:
| DLC | engine / version | accelerator | registry account |
|---|---|---|---|
| huggingface-pytorch-training | transformers 5.13.1 | GPU | 763104351884 |
| huggingface-pytorch-training-neuronx | transformers 4.55.4, Neuron SDK 2.26.0 | Trainium | 763104351884 |
| huggingface-pytorch-inference | transformers 4.51.3 | CPU / GPU | 763104351884 |
| huggingface-vllm | vLLM 0.29 | GPU | 763104351884 |
| huggingface-vllm-inference-neuronx | vLLM 0.11 via Optimum 0.4.5 | Inferentia | 763104351884 |
| huggingface-sglang | SGLang 0.5.12 | GPU | 763104351884 |
| huggingface-llama.cpp | llama.cpp b9522 | CPU / GPU | 763104351884 |
| tei / tei-cpu | TEI 1.9.3 | CPU / GPU | 683313688378 |
Two things fall out of that table that no prose summary of it would give you.
The TEI images come from a different registry account. Every other DLC is
763104351884; the embeddings container is not. If your ECR pull-through cache, your VPC endpoint policy or your image allowlist is written against one account id, the embeddings endpoint is the one that fails to pull, and it fails at deploy time rather than at review time.The transformers version is not one version. The training DLC ships transformers 5.13.1, the inference DLC ships 4.51.3, the Neuron pair ships 4.55.4, and — below — Bedrock Custom Model Import asks for 4.51.3 specifically. A fine-tune produced on the training DLC and imported into Bedrock crosses a major version boundary in the library that wrote the config. That is not a warning anyone will raise for you; it is an import job that either works or returns an error about your architecture.
The weights are the deployment
The habit the Hub encourages is
from_pretrained("org/model"), which resolves a name at runtime. On AWS that habit is three separate operational problems, and all three are solved the same way: treat the weights as a build artifact.| the habit | what it does in production | the AWS-only fix |
|---|---|---|
| resolving a model name at container start | every pod pulls tens of gigabytes from the internet, through the NAT gateway, on every scale-out | mirror to S3 once; the vLLM, SGLang and TEI DLCs load directly from S3 |
| pulling main | the bytes you serve on Tuesday are not the bytes you tested on Monday | pin the revision SHA in the mirror job, and treat a new SHA as a deploy |
| a gated or private repo | a token has to exist on the instance, and it is usually an environment variable in a manifest | Secrets Manager, read by the task or pod under its own role, as the target architecture argues for the database password |
The NAT line is not hypothetical, and it is the same line the crawler page found: data processing through a NAT gateway is charged per gigabyte, a 70B model in
bf16 is ~140 GB, and an autoscaling group that pulls it per node pays that per node, per scale-out, forever. A VPC endpoint does not help — the Hub is on the internet. An S3 mirror plus an S3 gateway endpoint removes the charge entirely, which is the rare case where the fix is both cheaper and faster.And the format matters for a reason that is not performance: a
.bin checkpoint is a pickle, and a pickle is code that runs at load time. safetensors is the format that is only tensors. Mirroring is the natural place to enforce it, because it is the one point where every byte passes through something you control.What Custom Model Import will actually accept
Read from the Bedrock user guide on 2026-09-26, because this is the row of the serving table with the most constraints and the fewest of them are obvious:
| constraint | value |
|---|---|
| regions | us-east-1, us-east-2, us-west-2, eu-central-1 |
| architectures | Mistral, Mixtral, Flan (T5), Llama 2/3/3.1/3.2/3.3 and Mllama, GPTBigCode, Qwen2/2.5/3 (Qwen3 only Qwen3ForCausalLM and Qwen3MoeForCausalLM), GPT-OSS (20B and 120B, us-east-1 only) |
| weight format | .safetensors, plus config.json, tokenizer_config.json, tokenizer.json, tokenizer.model, in S3 |
| size ceiling | 200 GB for text models, 100 GB for multimodal |
| context ceiling | maximum positional embeddings under 128K |
| library version | transformers 4.51.3 |
| embeddings | not supported — an embedding model cannot be imported at all |
| not available with | batch inference, CloudFormation |
Two of those deserve to be read twice. Embedding models cannot be imported, which means a system that fine-tunes both a generator and a retriever ends up with the generator behind the Bedrock API and the retriever behind a SageMaker endpoint or TEI on EKS — two serving paths, two scaling stories, two bills, decided by a single line in the documentation. And CloudFormation is not supported, which quietly means the import step cannot live in the same infrastructure-as-code run as everything around it.
Then there is the line that is easy to skim past:
Amazon Bedrock overrides llama3rope_scalingvalue with the following values:original_max_position_embeddings=8192,high_freq_factor=4,low_freq_factor=1,factor=8.
A field you set in
config.json is replaced by the service. If a long-context fine-tune depends on a different rope-scaling configuration, the imported model is not the model you trained, and nothing about the import job's success tells you so. Test the imported model on long inputs before believing the import succeeded — the failure is a quality regression, not an error.What AWS has no answer for
The honest end of the mapping. Four catalogue entries have no AWS-managed equivalent, and the gap is different in each case:
| catalogue layer | why there is no AWS row | what teams do instead |
|---|---|---|
| adversarial robustness (TextAttack, TextFlint, OpenAttack) | Bedrock Guardrails filter content at inference; they do not attack a model to measure how it breaks | run the library in a SageMaker processing job and keep the report as an artifact — the tooling is yours, the compute is AWS's |
| interactive dataset and failure-mode exploration (Spotlight) | model evaluation on AWS produces scores, not an interactive view of the segments that produced them | SageMaker Studio with the library installed; it is a notebook, not a service |
| the Hub's social layer — model cards, discussions, provenance | there is no AWS registry of who trained what, on what, under which licence | SageMaker Model Registry holds versions and approval status, and the licence and lineage become fields you populate and nobody validates |
| demo surfaces (Gradio / Spaces, which the catalogue assumes) | closest is App Runner or ECS behind an ALB | you are hosting a web app, and it is now your web app |
The licence row is the one that becomes a real problem rather than an inconvenience: model licences on the Hub range from Apache-2.0 to custom acceptable-use terms, the obligation travels with the weights into your S3 bucket, and — as Bedrock's own import documentation puts it — making sure your use complies with those terms is yours. An S3 object tag carrying the licence, applied by the mirror job, is a one-line habit that makes the question answerable later.
How it fails
| failure | what you see | what the design does |
|---|---|---|
| weights pulled at pod start | scale-out takes minutes and the NAT bill grows with the fleet | mirror to S3, load from S3, gateway endpoint |
| main moved | the same deploy produces different outputs | pin the revision SHA; a new SHA is a deploy, with a canary |
| gated repo, token missing at 3am | the endpoint never becomes healthy, and the message is a 401 from a host nobody on-call recognises | Secrets Manager plus a readiness probe that fails loudly on a credential error |
| a .bin checkpoint | arbitrary code at load time, inside your VPC, under the task role | refuse anything that is not safetensors at the mirror boundary |
| trust_remote_code=True | the model's own repository ships the modelling code that then runs | pin the revision, review the code once, vendor it into the image |
| the architecture is not Neuron-supported | a compile error late in a migration to Inferentia | check Optimum Neuron's support list before the instance decision, not after |
| Bedrock rewrote rope_scaling | quality falls off on long inputs, with no error anywhere | evaluate the imported model at full context before cutting traffic |
| TEI image will not pull | the embeddings endpoint alone fails to start | the registry account for TEI is not the one the other DLCs use |
What it costs
No prices — they move, and a wrong one stated confidently is worse than none. The shape:
| line | driver | the lever |
|---|---|---|
| endpoint instance-hours | accelerator type × replicas × hours held | Bedrock's per-token pricing is the alternative shape; a model you call rarely is the case where tokens win |
| NAT data processing | gigabytes of weights × nodes × scale-outs | the S3 mirror, which removes the line rather than reducing it |
| S3 storage | every mirrored revision you keep | a lifecycle rule; keeping every SHA forever is a decision, not a default |
| training | instance-hours, and everything on GPUs on AWS | Spot plus checkpointing, which is a purchasing decision |
| Bedrock custom-model inference | the imported model's own pricing dimension | whether the fine-tune earns the difference against a hosted model with a prompt |
The one worth stating plainly: the cheapest way to serve a Hub model is often not to hold an accelerator at all. The catalogue predates that option existing, which is why its Cloud section has one row.
What was rejected, and why
| the ecosystem answer | AWS-managed alternative | verdict |
|---|---|---|
| Hugging Face Inference Endpoints | SageMaker endpoint with the same DLC | rejected here only because it is not an AWS service; on the merits it is the least work of any row |
| a self-built model registry on S3 with a JSON index | SageMaker Model Registry | take the registry; it carries versions and approval state, which the JSON index will grow badly |
| Weaviate, Qdrant or Jina for vectors | OpenSearch Service k-NN | one cluster already serves BM25 and RRF here; a second vector store would be a second truth |
| running TGI because the tutorials say so | the vLLM DLC | the tutorials are describing an archived repository in maintenance mode |
| from_pretrained at runtime | S3 mirror, pinned revision | the runtime resolve is the cause of three separate incidents in the table above |
| a self-run Prometheus for endpoint metrics | CloudWatch, and AMP when GPU profiling counters are needed | the same split argued on GPUs on EKS |
| Qualcomm and Intel accelerators (the catalogue's Hardware section) | Inferentia and Trainium via Optimum Neuron | not a judgement on the silicon: they are not in an AWS account |
Verified, and not
Read, not remembered. The catalogue's structure and entries are from
huggingface/awesome-huggingface . Its archived state, last push date (2024-04-29) and star count, and the archived state and last push of lightseq, FastSeq, nn_pruning, FasterTransformer, TurboTransformers, parallelformers, OSLO, knockknock, transformers and text-generation-inference, were read through the GitHub API on 2026-09-26, as was the autonlp → autotrain-advanced redirect. TGI's archived-and-maintenance-mode status is from that API call and from the project's own maintenance-mode notice. The DLC table — image names, engine versions and both registry account ids — is from Hugging Face's AWS documentation on the same day. The Custom Model Import constraints, including the architecture list, the 200 GB and 100 GB ceilings, the 128K context ceiling, transformers 4.51.3, the exclusion of embedding models, batch inference and CloudFormation, and the rope_scaling override, are quoted from the Amazon Bedrock user guide on the same day.Run, not recalled. Nothing. No model was mirrored, no endpoint was deployed, no import job was submitted and no token was measured. This repository provisions no accelerated compute and its Terraform has never been applied.
Not verified, and load-bearing. Three. The DLC versions above are a snapshot of a list that changes every few weeks — the shape of the version skew between the training DLC, the inference DLC and Bedrock's pinned transformers is the durable claim; the exact numbers are not. The NAT arithmetic is multiplication over published pricing dimensions and a stated model size, not an invoice. And the
rope_scaling consequence — that a long-context fine-tune can be silently altered by the import — follows from the documented override and has not been reproduced here; it is the first thing to test if you import a long-context Llama, and the reason it is written down rather than assumed.Questions this page should answer
If the page did its work, each of these is answerable from it alone. The section named after each question is where the argument lives, not a hint.
- The catalogue's Cloud section has exactly one row. What does that date it to, and which layers of the list aged worst? — The catalogue, read as an artifact
- Four ways to serve a Hub model are listed and one of them is ruled out on a technicality rather than on merit. Which, and what is the technicality? — Four ways to serve a Hub model
- Why is "deploy the TGI container" now bad advice, and what do the two oddities in the DLC table cost you — the registry account and the transformers version? — The engine changed underneath the container
from_pretrained("org/model")is three operational problems, not one. Name them, and give the single habit that fixes all three. — The weights are the deployment- Custom Model Import refuses one whole class of model and silently overwrites one config field. Which, and what does each do to a design? — What Custom Model Import will actually accept
- Which catalogue layer has no AWS answer at all, and why is Bedrock Guardrails not it? — What AWS has no answer for
Next
- The platform underneath any of this: AI infrastructure on AWS , and the two layers below it — GPUs on AWS, above the cluster and GPUs on EKS .
- The serving decision argued without the Hub in the picture: LLMs on AWS, services only .
- Where the mirrored weights and datasets live, and why the hop matters: Storage for AI on AWS .
- The retrieval half, already running in this repository: search-platform .
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article