
Generative AI On AWS
The core concepts behind generative AI, so you confidently decide whether to own, rent, or plug into AWS's generative AI services in a way that serves your company's goals without driving up unnecessary cost.
Every business today can access generative AI. What they can't skip is the decision underneath it: own the model, build on one, or just use what's already built in. That call falls on the engineer, and it's rarely just a technical one.
But that call is only as good as the engineer's grasp of what's actually being decided. Before own, rent, or plug in means anything, it helps to understand what generative AI actually is, what a foundation model does under the hood, and why a concept called RAG quietly changes the entire calculation. Only then does the AWS layer decision stop being a guess and start being a strategy.
Understand What Generative AI Is
For most of machine learning's history, the question we asked computers was simple: what is this? Is this email spam. Is this a cat in the picture. Is this transaction fraudulent. The model's job was to look at an input and return a label, a category, or a number.
Instead of asking a model to classify something that already exists, we now ask it to create something that does not exist yet: what would a plausible new example of this look like? The output is no longer a label, it is new content: text, an image, audio, or code.
This is the shift behind everything you already interact with. ChatGPT writing a paragraph, Midjourney generating an image from a description, GitHub Copilot finishing a line of code before you type it. None of these tools were explicitly programmed with rules for what to say. They learned patterns from data and now apply those patterns to produce something new.
A useful way to think about it:
Imagine a musician who has studied thousands of songs. They are not retraining their ear in real time when they improvise. They are drawing on deeply internalized patterns, rhythm, structure, style, to produce a new piece of music that was never played before but still sounds coherent. That is close to what a generative model is doing with language, images, or code.

Foundation Models, and What Powers Them
The engine behind modern generative AI is the foundation model. A foundation model is a large model pretrained on massive, broad datasets, built to be adapted to many different downstream tasks instead of being built from scratch for each one. This is a genuine shift from older machine learning practice, where a new model was usually trained for every specific problem. Now, one base model can answer questions, summarize documents, write code, and hold a conversation, all without being retrained for each of those jobs.
Every AWS service discussed later in this guide, from Bedrock to SageMaker to Amazon Q, is ultimately a different way of packaging access to a foundation model.
A few terms are worth locking down here, because they show up constantly in any conversation about generative AI on AWS:
- Foundation Model (FM): the large pretrained model itself, for example Claude, Titan, Llama, or Stable Diffusion.
- Large Language Model (LLM): a foundation model that specializes in text and language.
- Prompt: the input a person gives the model to steer what it produces.
- Inference: the act of running the model to generate an output from a prompt.
- Fine tuning: further training a foundation model on your own labeled data so it specializes in your specific task or domain.
- Embeddings: numerical vector representations of text or data that capture meaning, used heavily for search and retrieval.
There is an important distinction that often gets misunderstood: a foundation model does not keep learning after it has been trained. Training happens once, offline, and produces a fixed set of internal parameters, known as weights. Once training ends, those weights are frozen.
Every time you send the model a prompt, it performs a fixed, one way computation using those already learned weights to predict the most statistically plausible next piece of text, image, or code. No new learning happens at that moment. The model is recombining and applying the patterns it internalized, grammar, structure, factual associations, style, to compute something novel in response to an input it has never literally seen before.
This distinction matters more than it first appears, because it explains why two other concepts, fine tuning and RAG, exist at all. Since a deployed model cannot learn new things on its own at the moment you ask it a question, if you want it to know about your company's own data and documents, you have exactly two paths: retrain it on that data, which is fine tuning, or feed the relevant information into the prompt at the moment of the question, which is RAG.

How RAG Comes Into Play
Foundation models are frozen after training and generic by design. They know what was in their training data up to a certain point, and nothing about your company's product catalog, internal documents, or last week's news. Fine tuning can partially address this, but it is expensive, slow, and has to be repeated every time the underlying data changes. For most real business use cases, that is not practical.
This is the problem Retrieval Augmented Generation (RAG), was built to solve. The idea is straightforward: instead of trying to make the model memorize your data through retraining, you retrieve the relevant pieces of your data at the moment of the question and hand them to the model as context, alongside the question itself. The model then generates its answer grounded in that fresh, specific information rather than relying only on what it learned during training.
A simple analogy: RAG is like giving a well read expert a short stack of your specific documents to skim right before they answer your question, instead of expecting them to already have memorized everything about your business from years ago. They still bring their reasoning ability and general knowledge, but the specific facts come from what you just handed them.
The mechanics, broken into two stages, look like this:
Preparation stage: your documents are split into smaller chunks, each chunk is converted into an embedding, a vector that captures its meaning, and those embeddings are stored in a vector database.
Query stage: when a user asks a question, that question is also converted into an embedding, the system searches the vector database for the chunks whose meaning is closest to the question, and those retrieved chunks are combined with the original question and passed to the foundation model, which generates an answer grounded in that retrieved context.

On AWS, this entire pipeline can be managed through Amazon Bedrock Knowledge Bases, which handles chunking, embedding, and retrieval without you having to wire each piece together yourself. Common vector store choices on AWS include Amazon OpenSearch Service in serverless mode, Amazon Aurora PostgreSQL with the pgvector extension, and third party options such as Pinecone or Redis.
For an engineer, the business case for RAG is simple to make: it is dramatically cheaper and faster to stand up than fine tuning, it keeps the model's answers current as your documents change, since there is no retraining involved, and it is the approach behind most production generative AI products in use today, from internal knowledge assistants to customer support bots.
Generative AI on AWS: The Different Layers
AWS organizes its generative AI offerings into three layers, and this structure is the clearest way to reason about the own, rent, or plug in decision making. Each layer trades off control, cost, and speed to market differently, and each serves a different business goal.

Infrastructure: Own It
This is the layer for teams building a foundation model from scratch. It means provisioning raw compute, GPU based EC2 instances such as the P4 and P5 families, or AWS's own custom chips, Trainium for training and Inferentia for inference, built specifically to be more cost effective than general purpose GPUs for this workload. Large scale training runs are often orchestrated with SageMaker HyperPod, which manages distributed training across many instances at once.
This layer is for companies where the model itself is the product or a core competitive asset, typically AI labs or organizations with the resources and reason to fully own their model's behavior. The economics are the highest of the three layers: significant upfront capital investment, but full control and full differentiation. It only makes sense when owning the model is genuinely part of the business strategy, not simply because it seems like the most advanced option.
Tools: Rent It
This is where most builders on AWS actually operate. Instead of training a foundation model, you build on top of one that already exists, choosing between two AWS services depending on how much control you want:
- Amazon Bedrock: fully managed, serverless access to multiple foundation models, including Anthropic Claude, Meta Llama, Mistral, Cohere, and Amazon's own Titan and Nova models, all through one unified API. There is no infrastructure to manage, and Bedrock also supports fine tuning, RAG through Knowledge Bases, agents, and guardrails.
- Amazon SageMaker: more control for teams that want it, letting you deploy and fine tune open source foundation models yourself, with a fuller machine learning lifecycle but a heavier operational load than Bedrock.
Because these two services are the ones an engineer will actually be choosing between day to day, the table below lays out the decision in the terms that matter most when you are the one accountable for the infrastructure and its cost:
| Amazon Bedrock | Amazon SageMaker |
|---|---|---|
Management | Fully managed and serverless. AWS handles the underlying infrastructure entirely. | You provision, manage, and scale the infrastructure and endpoints yourself. |
Model choice | A curated set of foundation models accessed through one unified API. | Any model, including your own or open source ones, for maximum flexibility. |
Best for | Fast integration, RAG through Knowledge Bases, and agent based workflows. | Custom training, deep fine tuning, and full control over the ML lifecycle. |
Operational burden | Low. No servers or endpoints to maintain | Higher. Your team owns provisioning, scaling, and upkeep. |
This layer serves most product teams, startups, and enterprises whose business goal is to ship AI features quickly without owning the underlying model. The economics are pay as you go, with no infrastructure investment required. Bedrock trades some flexibility for speed and a lower operational burden, while SageMaker trades more setup effort for deeper customization. For most engineers weighing this decision, it comes down to a straightforward tradeoff: speed versus control, and the honest answer for most workloads, when the goal is serving a company need without inflating cost, is to start with Bedrock and only reach for SageMaker when a specific requirement, like training on proprietary data at scale or deploying a model Bedrock does not offer, actually demands it.
Applications: Plug In
This is generative AI already embedded inside other AWS products, where the end user is not thinking about a foundation model at all, they are just using a feature. Amazon Q Developer autocompleting and explaining code inside an IDE, or generative BI summaries inside Amazon QuickSight, are both examples.
This layer is for companies that want the outcome, faster coding, faster insight, without building or managing anything AI related themselves. The economics are the cheapest and fastest to adopt of the three layers, but also the least differentiated: since every company using the same built in feature has access to the same capability, it functions as a productivity gain and table stakes rather than a competitive edge.
Layer | What It Is | Who It's For | Business Goal | Economics |
|---|---|---|---|---|
Infrastructure | Raw compute (GPUs, Trainium/Inferentia) to train a foundation model from scratch | AI labs and companies where the model is the product | Full ownership and differentiation of the model itself | Highest upfront cost and control, justified only if the model is the competitive moat |
Tools (Bedrock / SageMaker) | Pre built foundation models to build on, fully managed (Bedrock) or hands on (SageMaker) | Most companies: product teams, startups, enterprises adding AI features | Ship AI capabilities fast without training a model | Pay as you go, no infrastructure investment. Bedrock favors speed, SageMaker favors control |
Applications | GenAI already built into AWS products, such as Amazon Q Developer or generative BI in QuickSight | Companies that want the outcome, not the AI work itself | Productivity gains with zero AI expertise required | Cheapest and fastest to adopt, but least differentiated, effectively table stakes |
Bringing It Back to the Decision
Access to generative AI is no longer the hard part. AWS, along with every other major cloud provider, has already built the paths: own the model by training it from scratch, rent a foundation model through Bedrock or SageMaker, or plug into a feature that already has generative AI built in.
What actually separates a good decision from a costly one is whether the engineer making the call understands what is being traded off at each layer, and how that tradeoff maps to what the business is actually trying to achieve.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article