Amazon S3 for AI Projects: Storing Documents and Datasets
Set up Amazon S3 properly for an AI project. Plan a prefix layout, check safe defaults, choose encryption, turn on versioning, add lifecycle rules, write least-privilege policies and keep an eye on sensitive data.
Series: Amazon Bedrock (13 articles)
- …
- 13Amazon S3 for AI Projects: Storing Documents and Datasets This article
A practical setup for the bucket behind your AI app: how to lay it out, lock it down, keep versions, and stop it growing forever.
Almost every AI project on AWS ends up with an Amazon S3 bucket at its centre. It holds the documents a knowledge base reads, the datasets you test with, the outputs your app produces and the files that pile up along the way. It's also where small mistakes quietly become big ones: a bucket that's too open, files nobody can find, or storage costs that creep up month after month.
In this article we'll set up a bucket properly in a few steps: a layout that makes sense, safe defaults, versioning, lifecycle rules and a least-privilege policy for your app.
What you need
- An AWS account and the AWS CLI configured
- A Region in mind. Use the same Region as your Amazon Bedrock work, since services such as knowledge bases read from a bucket in your Region.
- About thirty minutes
Before you start, check the current Amazon S3 documentation and pricing page, as defaults and prices can change.
Step 1: Plan the layout first
S3 has no real folders. It stores objects under keys, and a "folder" is just a shared prefix in the key name. But prefixes are how you organise, apply rules and control access, so plan them before you upload anything.
A simple layout for an AI project:
Plain text
my-ai-project-dev/raw/ original files exactly as receivedprocessed/ cleaned or converted versionskb-docs/ the documents your knowledge base readsoutputs/ results your app generatestmp/ scratch files that can be deleted
Why this works:
- Raw stays untouched. If a processing step goes wrong, you can rerun it from the original.
- Each prefix can have its own rules. You can expire
tmp/quickly and keepraw/for much longer.
- Permissions can follow the prefixes. Your app might read
kb-docs/and writeoutputs/, and nothing else.
Use one bucket per environment, such as
-dev and -prod , instead of mixing test and live data together. Bucket names are globally unique, so pick something distinctive.Step 2: Create the bucket
With the AWS CLI:
Bash
aws s3 mb s3://my-ai-project-dev --region us-east-1
Or use the S3 console, choosing the same Region you use for Bedrock.
When you create it, check three settings:
- Block Public Access should be on. This is the default for new buckets, and for an AI project holding your own documents there's almost never a reason to turn it off.
- Bucket owner enforced for object ownership, which disables old-style ACLs and keeps access control simple. This is also the default for new buckets.
- Default encryption is applied to new objects. Step 3 explains your options.
The defaults are good. The point is to check them, not to assume them.
Step 3: Choose how to encrypt
Amazon S3 encrypts new objects by default. You have two main options:
Option | What it means | When to choose it |
|---|---|---|
SSE-S3 | S3 manages the keys | The default. Fine for most projects |
SSE-KMS | Your data is encrypted with an AWS KMS key you control | When you need control over the key, audit trails on its use or the ability to revoke access |
SSE-KMS gives you more control, but it adds a few things to manage: the KMS key's own policy, KMS request charges and extra permissions. If another service reads your bucket, such as a Bedrock knowledge base, its role needs permission to use the key too. A missing KMS permission is a classic cause of baffling access-denied errors.
Start with the default. Move to KMS when you have a reason.
Step 4: Turn on versioning
Versioning keeps earlier versions of an object when it's overwritten or deleted. For AI projects that's a real safety net. A bad batch job that overwrites your documents, or an accidental delete, becomes recoverable.
Bash
aws s3api put-bucket-versioning \--bucket my-ai-project-dev \--versioning-configuration Status=Enabled
The catch is cost. Old versions are stored, and charged for, until you remove them. That's what the next step handles.
Step 5: Add lifecycle rules
Lifecycle rules let S3 tidy up for you. Here are three that suit most AI projects:
- Expire scratch files. Delete anything under
tmp/after seven days.
- Clean up old versions. Remove noncurrent versions after thirty days.
- Abort failed uploads. Incomplete multipart uploads sit there invisibly and cost money. Clear them after seven days.
Save this as
lifecycle.json :JSON
{"Rules": [{"ID": "expire-tmp","Filter": { "Prefix": "tmp/" },"Status": "Enabled","Expiration": { "Days": 7 }},{"ID": "clean-old-versions","Filter": { "Prefix": "" },"Status": "Enabled","NoncurrentVersionExpiration": { "NoncurrentDays": 30 }},{"ID": "abort-incomplete-uploads","Filter": { "Prefix": "" },"Status": "Enabled","AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }}]}
Then apply it:
Bash
aws s3api put-bucket-lifecycle-configuration \--bucket my-ai-project-dev \--lifecycle-configuration file://lifecycle.json
For older data you rarely read, lifecycle rules can also move objects to cheaper storage classes after a set number of days. That's where S3 Intelligent-Tiering can help, because it moves objects between access tiers automatically based on how often you use them. Check the pricing page for the details that matter: there's a small monitoring charge per object, and very small objects aren't eligible for automatic tiering. It suits large, unpredictable datasets better than a few small documents.
Be careful with expiry on anything you can't recreate. Test your rules on a dev bucket first.
Step 6: Lock the door
Good defaults keep the public out. Now control who inside your account can get in.
Require encrypted connections. Add a bucket policy that denies any request not sent over TLS:
JSON
{"Version": "2012-10-17","Statement": [{"Sid": "DenyInsecureTransport","Effect": "Deny","Principal": "*","Action": "s3:*","Resource": ["arn:aws:s3:::my-ai-project-dev","arn:aws:s3:::my-ai-project-dev/*"],"Condition": { "Bool": { "aws:SecureTransport": "false" } }}]}
Give your app only what it needs. For an app that reads the knowledge base documents and nothing else, attach a policy like this to its role:
JSON
{"Version": "2012-10-17","Statement": [{"Sid": "ReadDocs","Effect": "Allow","Action": "s3:GetObject","Resource": "arn:aws:s3:::my-ai-project-dev/kb-docs/*"},{"Sid": "ListDocsOnly","Effect": "Allow","Action": "s3:ListBucket","Resource": "arn:aws:s3:::my-ai-project-dev","Condition": { "StringLike": { "s3:prefix": "kb-docs/*" } }}]}
Notice the two parts. Reading objects applies to the object ARNs, and listing applies to the bucket itself, restricted to a prefix. Mixing these up is the most common S3 policy mistake.
Step 7: Upload and use it
From the command line:
Bash
aws s3 sync ./my-documents s3://my-ai-project-dev/kb-docs/
From Python:
Python
import boto3s3 = boto3.client("s3")s3.upload_file(Filename="refund-policy.pdf",Bucket="my-ai-project-dev",Key="kb-docs/refund-policy.pdf",ExtraArgs={"Metadata": {"source": "policy-team", "version": "2026-10"}},)response = s3.list_objects_v2(Bucket="my-ai-project-dev", Prefix="kb-docs/")for item in response.get("Contents", []):print(item["Key"], item["Size"])
A habit worth building: add a little metadata, such as the source and a version or date, to documents you upload. When an AI answer cites a file, you'll be glad to know where it came from and how old it is.
If you're using the bucket for a Bedrock knowledge base, remember to sync the knowledge base after changing files, or answers will reflect the old content.
Step 8: Keep an eye on what's in there
A bucket of documents for AI often collects things nobody planned to store: customer details, contracts, personal data. Two tools help:
- Amazon Macie can scan S3 for sensitive data, such as personal information, so you know what you're holding. It has its own pricing, so start with the buckets that matter most.
- Access logging and CloudTrail record who touched what. Turn on at least the basic logging for any bucket that holds sensitive material.
And think before you upload. If a document contains personal or confidential data that the AI doesn't need, remove it first. It's much easier than protecting it later.
What it costs
S3 is inexpensive, but not free. You pay for:
- Storage, per GB per month, which varies by storage class
- Requests, such as reads, writes and lists
- Data transfer, especially data leaving AWS or crossing Regions
- Extras, such as KMS requests, Macie scanning and Intelligent-Tiering monitoring
The lifecycle rules in Step 5 are your main cost control. Add a budget alert as well, as in the cost article earlier in this series.
Common mistakes
- Turning off Block Public Access to "make it work." Fix the permissions instead.
- Using
s3:*on*for the app's role. Scope it to the one prefix it needs.
- Forgetting versions are billed. Add a rule for old versions.
- Mixing test and live data in one bucket.
- Putting a KMS key behind the bucket and forgetting the service role needs access to the key.
- Uploading everything, including documents with personal data, without checking.
Clean up
When you've finished experimenting, delete the objects and the bucket. A versioned bucket holds older versions and delete markers, which all have to go before S3 will let you delete the bucket. The easiest route is the console's empty-bucket option, then delete the bucket. Check your bill afterwards to make sure nothing is still charging.
A checklist
- Is the bucket in the right Region, with a clear prefix layout?
- Is Block Public Access on?
- Is encryption set the way you intend?
- Is versioning on, with a lifecycle rule for old versions?
- Does your app's role have access to one prefix only?
- Do you know what sensitive data the bucket holds?
- Do you have a budget alert?
Where we go next
With storage sorted, we can give an app a proper database. In the next article we'll look at Amazon DynamoDB for AI applications: storing conversations, sessions and results.
S3 defaults, features, storage classes and pricing change often. Check the current Amazon S3 documentation and pricing page before you follow any step here.
Series: Amazon Bedrock (13 articles)
- …
- 13Amazon S3 for AI Projects: Storing Documents and Datasets This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article