AWS Builder Center

AWS Foundation for Generative AI: From Cloud Infrastructure to AI Agents

AWS Foundation for Generative AI: From Cloud Infrastructure to AI Agents

I'm learning about AWS Solution Architect !

AWS Foundation for Generative AI: From Cloud Infrastructure to AI Agents

1. Generative AI Is More Than Just Calling an API

When developers first start working with Generative AI, the architecture often looks very simple:
User
↓
Backend API
↓
LLM
↓
Response

For example, an application sends a prompt to a foundation model such as Claude or Amazon Nova, receives the response, and returns it to the user.
For a prototype, this architecture may be perfectly sufficient.
However, once the system moves into production, the architecture becomes much more complex.
We need to answer questions such as:
  • Who is allowed to use the AI system?
  • How do we protect sensitive data?
  • How can the AI access internal company data?
  • How do we reduce hallucinations?
  • How do we monitor latency and token usage?
  • How do we control AI costs?
  • How do we limit what an AI Agent is allowed to do?
  • How do we provide governance when multiple teams use AI?
This is where the concept of a Generative AI Foundation becomes important.
A strong AI foundation provides reusable infrastructure, security controls, data capabilities, governance, observability, and AI services that multiple teams can build on.



2. Think of AWS Foundation as Multiple Layers

A useful way to understand a modern AWS AI architecture is to divide it into several layers:
┌──────────────────────────────────────────────┐
│ AI Applications │
│ Chatbot | Copilot | AI Agent | RAG | Search │
├──────────────────────────────────────────────┤
│ AI Services │
│ Bedrock | AgentCore | Knowledge Bases │
├──────────────────────────────────────────────┤
│ AI Gateway │
│ Auth | Rate Limit | Quota | Cost | Logging │
├──────────────────────────────────────────────┤
│ Data Foundation │
│ S3 | RDS | OpenSearch | Vector DB │
├──────────────────────────────────────────────┤
│ Cloud Foundation │
│ IAM | VPC | KMS | CloudWatch | WAF | DNS │
├──────────────────────────────────────────────┤
│ AWS Infrastructure │
│ Region | AZ | EC2 | Lambda | Containers │
└──────────────────────────────────────────────┘

The key idea is:
The AI application is only the top layer.
If identity, networking, data, security, and observability are poorly designed, the AI application will also struggle to become a reliable production system.



3. Layer One: Cloud Foundation

Before discussing Bedrock or AI Agents, we still need the fundamental AWS building blocks.

Identity

AWS IAM is the foundation for controlling access to AWS resources.
For example:
Developer
↓
IAM / SSO
↓
AWS Account
↓
IAM Role
↓
AWS Services

Applications should generally avoid using long-lived IAM user credentials.
Instead, applications should use IAM Roles with only the permissions they actually need.
This follows the principle of:
Least Privilege
For an AI system, this becomes even more important because an AI Agent may need to interact with multiple AWS services and business systems.



Networking

A production AI system may require:
  • Amazon VPC
  • Private Subnets
  • Security Groups
  • VPC Endpoints
  • NAT Gateway
  • AWS WAF
  • Amazon API Gateway
  • CloudFront
A simplified architecture might look like:
Internet
│
▼
CloudFront
│
▼
AWS WAF
│
▼
API Gateway
│
▼
Lambda / ECS
│
├──── S3
├──── DynamoDB
└──── Amazon Bedrock

For enterprise workloads, controlling network access to AI services is particularly important.
The architecture should consider not only whether the application can reach the AI service, but also who can access it, from where, and under what conditions.



4. Layer Two: Data Foundation

A Large Language Model does not automatically know your company's private information.
Suppose a company has:
  • HR policies
  • Internal documentation
  • Product specifications
  • Customer information
  • Engineering documentation
  • Business procedures
We need a mechanism that allows the AI system to retrieve relevant information.
This is where Retrieval-Augmented Generation (RAG) becomes useful.
A simplified RAG pipeline looks like this:
Documents
│
▼
S3
│
▼
Chunking
│
▼
Embeddings
│
▼
Vector Store
│
▼
Semantic Search
│
▼
Relevant Context
│
▼
LLM

For example, imagine a company has 10,000 HR documents.
Instead of putting all of them directly into every prompt, we can:
  1. Store documents in Amazon S3.
  1. Split documents into smaller chunks.
  1. Generate embeddings.
  1. Store embeddings in a vector database.
  1. Search for relevant chunks when a user asks a question.
  1. Provide those chunks as context to the foundation model.
AWS provides Amazon Bedrock Knowledge Bases as a managed approach for building RAG applications.
Other architectures can use services such as Amazon OpenSearch or PostgreSQL with  pgvector .



5. Layer Three: Foundation Models

This is where Amazon Bedrock becomes important.
Without a managed AI platform, an organization might have to manage its own model-serving infrastructure:
Application
│
▼
GPU Cluster
│
▼
LLM

With Amazon Bedrock:
Application
│
▼
Amazon Bedrock
│
├── Amazon Nova
├── Anthropic Claude
├── Other Foundation Models
└── Other AI Models

The major advantage is that developers can consume foundation models through APIs without managing the underlying model-serving infrastructure themselves.
This allows teams to focus more on application and business logic rather than GPU management, model deployment, and infrastructure scaling.



6. An Important Architectural Pattern: AI Gateway

When an organization has only one AI application, the architecture can be very simple:
Application
│
▼
Amazon Bedrock

But imagine an organization with many teams:
Team A ─┐
Team B ─┤
Team C ─┼──> AI Platform ──> Bedrock
Team D ─┤
Team E ─┘

Now we need centralized controls.
An AI Gateway can provide a layer between applications and AI models:
┌── Team A
│
Team B ──> API Gateway ──> AI Gateway ──> Bedrock
│
├── Authentication
├── Authorization
├── Rate Limiting
├── Quotas
├── Logging
├── Cost Tracking
└── Guardrails

This pattern can provide centralized control over how different applications consume foundation models.
For example, Team A might have a monthly AI budget of $1,000 while Team B has a budget of $5,000.
The gateway can become the control point for:
  • Authentication
  • Authorization
  • Rate limiting
  • Usage tracking
  • Cost allocation
  • Model selection
  • Logging
  • Governance
This becomes particularly valuable in large organizations.



7. Layer Four: AI Agents

RAG allows AI to retrieve information.
AI Agents go one step further.
An Agent can reason about a task and interact with external tools or systems.
For example:
User:
"Check order #123 and create a customer
service ticket if the delivery is delayed."

A traditional chatbot might only explain how the user can check the order.
An AI Agent could potentially perform the workflow:
User
↓
AI Agent
↓
Understand Request
↓
Call Order API
↓
Check Delivery Status
↓
Call Ticket API
↓
Create Ticket
↓
Return Result

The architecture becomes:
┌── Database
│
User → Agent ────┼── REST API
│
├── Search
│
└── Business Systems

This is one reason AI Agents are becoming an important architectural topic.
AWS has introduced Amazon Bedrock AgentCore to help developers build, deploy, and operate AI agents for production workloads.



8. The Biggest Security Challenge: Agent Permissions

One of the most important architectural questions is:
What is the AI Agent actually allowed to do?
Imagine an Agent has access to:
Read Database
Write Database
Send Email
Delete Data
Create Ticket
Call Payment API

The Agent is effectively becoming an identity with access to enterprise systems.
Giving it unrestricted access would be extremely dangerous.
Instead, permissions should be explicitly defined:
Agent
│
▼
Authorization Layer
│
├── Read Order ✓
├── Create Ticket ✓
├── Send Email ✓
├── Delete Order ✗
└── Refund Payment ✗

This is where IAM and application-level authorization become critical.
An Agent should have the minimum permissions necessary to perform its job.



9. Observability Must Be Designed from the Beginning

Traditional applications usually monitor metrics such as:
  • CPU
  • Memory
  • Request latency
  • Error rate
  • Throughput
AI applications require additional metrics.
We can think about observability in three major categories:
Application Metrics
+
LLM Metrics
+
AI Business Metrics


Application Metrics

Examples:
  • API latency
  • HTTP errors
  • Lambda duration
  • ECS CPU utilization

LLM Metrics

Examples:
  • Input tokens
  • Output tokens
  • Model latency
  • Model usage
  • Error rate

RAG Metrics

Examples:
  • Retrieval latency
  • Number of retrieved documents
  • Empty retrieval rate
  • Retrieval relevance

Agent Metrics

Examples:
  • Number of tool calls
  • Failed tool calls
  • Number of agent steps
  • Agent execution time
  • Tool latency
Without these metrics, debugging an AI system can become extremely difficult.



10. Cost Management Becomes an Architectural Concern

Traditional applications might estimate costs using:
Compute
+
Storage
+
Network

AI systems introduce additional cost dimensions:
Input Tokens
+
Output Tokens
+
Model
+
Requests
+
Embeddings
+
Vector Search
+
Agent Tool Calls

Consider an apparently simple request:
User
↓
RAG
↓
LLM
↓
Agent
↓
3 API Calls
↓
LLM

One user interaction may trigger multiple model calls and tool calls.
Therefore, cost management should not be treated as something to think about after deployment.
It should be part of the architecture.
For example:
User
↓
AI Gateway
↓
Quota / Rate Limit
↓
Model
↓
Usage Tracking
↓
Cost Allocation

Teams can then track AI usage by:
  • Application
  • Team
  • Environment
  • Model
  • Business unit
This becomes especially important when AI adoption grows across an organization.



11. A Production-Oriented AWS GenAI Foundation

Putting the previous concepts together, a high-level architecture could look like this:
USERS
│
▼
CloudFront
│
▼
WAF
│
▼
API Gateway
│
▼
AI Gateway Layer
┌────────┼─────────┐
│ │ │
Auth Quota Logging
│ │ │
└────────┼─────────┘
│
▼
Application Layer
│
┌─────────────┼─────────────┐
│ │ │
▼ ▼ ▼
RAG Agent Direct LLM
│ │ │
▼ ▼ ▼
Knowledge Base Tools/APIs Amazon Bedrock
│ │ │
└─────────────┼─────────────┘
│
▼
Data Layer
┌─────────┼─────────┐
│ │ │
S3 RDS DynamoDB
│
▼
Vector Store

───────────────────────────────

IAM | KMS | CloudWatch | VPC
CI/CD | Secrets | Monitoring
Governance | Cost Management

This is not a universal architecture.
Different applications will require different components.
The important lesson is the architectural thinking behind the layers.



12. From AWS Foundation to an AI Platform

As AI adoption grows, having every team independently build its own AI infrastructure creates significant duplication.
For example:
Team A
├── Bedrock
├── RAG
├── Logging
└── IAM

Team B
├── Bedrock
├── RAG
├── Logging
└── IAM

Team C
├── Bedrock
├── RAG
├── Logging
└── IAM

A better approach can be a centralized AI Platform:
AI PLATFORM
│
┌─────────────┼─────────────┐
│ │ │
Model Access RAG Agents
│ │ │
└─────────────┼─────────────┘
│
Security / IAM
│
Observability
│
Cost Management
│
Governance

The platform team provides reusable capabilities.
Application teams then focus on business problems:
HR Team
↓
HR Assistant

Sales Team
↓
Sales Copilot

Support Team
↓
Customer Agent

Engineering Team
↓
Developer Assistant

This is the fundamental idea behind an enterprise AI platform.



13. What Does This Mean for an AWS Solution Architect?

If AWS is learned only as:
EC2 runs servers. S3 stores objects. Lambda is serverless. RDS is a database.
the knowledge can quickly become fragmented.
A Solution Architect needs to think at a higher level:
How do I combine these services to create a secure, scalable, reusable platform?
With Generative AI, the question is no longer simply:
"How do I call Claude?"
Instead, the architectural question becomes:
"How do I build a secure, scalable, observable, and cost-efficient AI platform?"
That is a real Solution Architecture problem.



14. A Practical AWS Learning Roadmap for GenAI

If you are starting from AWS fundamentals, a logical learning path is:
1. AWS Fundamentals
↓
2. IAM
↓
3. VPC
↓
4. S3
↓
5. Lambda
↓
6. API Gateway
↓
7. DynamoDB / RDS
↓
8. CloudWatch
↓
9. Amazon Bedrock
↓
10. Embeddings + Vector Databases
↓
11. RAG
↓
12. AI Gateway
↓
13. AI Agents
↓
14. Agent Security
↓
15. Governance + Cost Optimization

This approach is more valuable than memorizing AWS services individually because each topic builds on the previous one.



15. Conclusion

Generative AI is changing the way we think about cloud architecture.
A foundation model is only one component.
A production-grade AI system requires:
Identity + Networking + Data + Models + Security + Observability + Governance + Cost Management
Therefore, learning AWS Foundation in 2026 should not simply mean memorizing AWS services.
Instead, learn how to combine them into a reusable foundation.
The evolution looks something like this:
AWS Infrastructure
↓
Cloud Foundation
↓
Data Foundation
↓
AI Foundation
↓
AI Platform
↓
AI Applications
↓
AI Agents

The most important shift is from service-oriented thinking to architecture-oriented thinking.
Instead of asking:
"What does this AWS service do?"
start asking:
"What problem does this service solve, what are its trade-offs, and how does it fit into the overall architecture?"
That mindset is what turns AWS knowledge into Solution Architecture skills.
And as Generative AI continues to evolve, the ability to design the foundation underneath AI applications may become just as important as understanding the models themselves.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article