RAG and Fine-tuning System Design — on AWS, services only
RAG and Fine-tuning System Design — on AWS, services only
RAG and Fine-tuning System Design — on AWS, services only
Source design.rag-finetuning/README.mdin the system-design corpus. This page keeps that design's requirements and numbers and rebuilds it on AWS-managed services only. Nothing here is deployed, and the honesty section says exactly how this page was produced.
What the source design is
Comprehensive system design documentation and implementations for a Retrieval-Augmented Generation (RAG) and Fine-tuning platform.
The numbers it has to survive
Taken from the source document, not re-derived:
| quantity | value |
|---|---|
| Vector DB | 4 TB (500M chunks) |
| Document Storage | 100 TB |
| Daily Growth | 32.7 GB/day |
What the source covers, and where each part lands on AWS
| the source's section | on AWS |
|---|---|
| Documentation | no AWS-specific change: this section is about the problem, not the platform |
| Code Implementations | the architecture above; the managed services change the operational half and leave the logic |
| C++ Implementation | the architecture above; the managed services change the operational half and leave the logic |
| Rust Implementation | the architecture above; the managed services change the operational half and leave the logic |
| Go Implementation | the architecture above; the managed services change the operational half and leave the logic |
| System Overview | no AWS-specific change: this section is about the problem, not the platform |
| Scale | quotas and service limits, checked per service before launch rather than derived from server counts |
| Key Components | the architecture above; the managed services change the operational half and leave the logic |
| Key Design Decisions | CodePipeline and CodeBuild, ECR with immutable tags and Inspector scanning |
| Cost Optimization | Cost Explorer and Budgets, with the data-transfer lines called out separately because they do not appear in a diagram |
| Quick Start | no AWS-specific change: this section is about the problem, not the platform |
| Core Concepts | no AWS-specific change: this section is about the problem, not the platform |
| Related Systems | no AWS-specific change: this section is about the problem, not the platform |
The AWS-only architecture
The shape this design takes when every component has to be an AWS-managed service:
- Bedrock for models, so there is no inference fleet until there has to be
- OpenSearch Serverless with k-NN as the vector store
- S3 for the corpus, with Glue or Lambda for the embedding pipeline
- SageMaker only when the model is your own
The source already reaches for
s3, so those parts of the design carry over unchanged.The request path
1
2
3
4
5
client -> API Gateway -> Lambda -> Bedrock (model)
| ^
| | retrieved context
+-> OpenSearch Serverless (k-NN)
S3 (corpus) -> Lambda/Glue (chunk + embed via Bedrock) -> OpenSearchThe decisions that survive the swap
Managed services remove the operational decisions and leave the design ones. For this system they are:
- Where the source of truth is, and what is only a projection. Every derived store — a search index, a cache, a materialised view — must be rebuildable from the source of truth, and nothing may write to two stores in one request. On AWS the rebuild path is usually a Lambda over a DynamoDB Stream or an S3 prefix, and it should be exercised, not assumed.
- Idempotency, because every AWS delivery primitive is at-least-once. SQS, DynamoDB Streams, Kinesis and EventBridge can all deliver twice. A client-supplied key plus a conditional write is the standard defence, and it has to be designed in rather than added after the first duplicate.
- Where the lock lives. A conditional write in DynamoDB is a compare-and-swap inside the storage engine: it needs no lock service, cannot be lost on failover, and fails loudly. Anything that reaches for a distributed lock should first be checked against that.
- Identity is IAM, tenancy is the application's. IAM roles protect the service; they do not scale to one role per end user. The tenant must come from the credential the application validates, never from a request header.
How it fails
| failure | what the user sees | what the design does |
|---|---|---|
| a managed service hits a quota | errors that look like an application bug | quotas are per account and per region: raise them before the event, and alarm on Service Quotas usage rather than discovering the ceiling during a launch |
| a region degrades | the whole system, if it is single-region | multi-AZ is the default in these services and multi-region is a deliberate, priced decision — this design states which one it has taken |
Where this design usually goes wrong
- Standing up an inference fleet before Bedrock has been ruled out on cost or capability, which turns a product question into a GPU-capacity project.
- A vector store chosen before the retrieval quality has been measured with the simplest possible baseline.
- Paying for embeddings on the whole corpus when only the ambiguous cases need them.
The scaling path
The version to build first is not the version in the source document, and the AWS-only path between them is short:
| stage | what it is | the change from the previous stage |
|---|---|---|
| first version | one Lambda or one Fargate service, DynamoDB or Aurora, S3, CloudFront in front | none — this is where every design should start, and it carries more load than most teams expect |
| under growth | split the read path from the write path; add the projection (OpenSearch, a materialised view) fed by a stream rather than a dual write | the first place a second store is justified, and the first place a rebuild path has to exist |
| at the source document's numbers | the architecture above in full: managed streaming, per-tenant partitioning, multi-AZ everywhere and a stated multi-region position | mostly capacity and quota work, plus the failure modes below becoming real rather than theoretical |
What was rejected, and why
| rejected | in favour of | the reason |
|---|---|---|
| a role per end user in IAM | application-level tenancy, checked against the credential | IAM entity quotas are per account and are orders of magnitude below a user base; IAM protects the service, the application protects the tenant |
| a distributed lock service | a DynamoDB conditional write | compare-and-swap inside the storage engine cannot be lost on failover and needs nothing running |
| a second store written in the same request | one source of truth plus a projection rebuilt from it | dual writes fail independently, and the failure is silent until someone compares the two |
| a self-run metrics or tracing stack | CloudWatch, and X-Ray through ADOT | the alerting has to survive the outage it is reporting on, which is the argument against running it beside the workload |
Operating it
The alarms this design needs, in the order they earn their place:
- The user-visible SLI first — error ratio and latency on the entry point (ALB, API Gateway or CloudFront), as a burn-rate alarm against a stated objective rather than a static threshold.
- Saturation of the one resource that cannot burst — a provisioned store, a connection pool, or a shard count. CloudWatch has the metric; the design has to name which one it is.
- Queue or stream lag, where the design has one: it is the earliest signal that the system is falling behind without erroring.
- Quota usage, through Service Quotas — the ceiling that turns a successful launch into an incident.
- Cost as a signal: a Budgets alarm on the line item this design is most able to get wrong, which is usually data transfer.
Identity, secrets and audit
- Workload identity is an IAM role — IRSA or EKS Pod Identity on EKS, a task role on ECS, an execution role on Lambda. No access key exists anywhere in the design.
- Credentials live in Secrets Manager and are read by the workload itself at startup, not rendered into a manifest or synced into a Kubernetes Secret, which is base64 in etcd and readable by anyone with
get secretin the namespace. - Encryption is KMS everywhere it is offered, with a customer-managed key where the audit trail of key use matters.
- Every API call is in CloudTrail, which is what makes an agent or an automation reviewable after the fact — and the reason destructive actions should be denied by a permissions boundary rather than merely un-scripted.
What it costs
- S3 storage, requests and — the line most designs forget — data transfer out through CloudFront or NAT
- Data transfer: NAT gateway processing and CloudFront egress are the two lines that surprise people, because neither appears in a component diagram.
Verified, and not
How this page was produced. It was rendered by
tools/system-design-render.py from ~/dev/system-design/rag-finetuning/README.md. The title, the summary, the requirement list and every number in the table above are extracted from that document, not recalled. The AWS mapping, the decisions and the failure table come from the mapping in that tool, which is written once and applied to all 620 designs so that they agree with each other.What that means for the reader. This is a correct mapping, not a bespoke design review: the swap for each named technology is right and its cost is stated, but the argument specific to this system — the one decision that makes it interesting — is the part a human still has to write. The pages in the corpus index marked as hand-written have had that pass; this one has not yet.
Run, not recalled. Nothing. No resource was created, no load was applied and no cost was measured. Every AWS behaviour cited is from documentation and moves: verify before acting on one.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article