Amazon EC2 & S3: Building a Shared Agent Workspace with Amazon EC2 and S3 Files
Multi-agent AI systems often rely on a central database for shared state. But for workloads heavily dependent on documents, source code, and artifacts, a shared file system might be a better abstraction. Here is how Amazon S3 Files enables file-oriented agent memory.
Do AI Agents Really Need a Database for Memory?
Multi-agent AI systems often need to share information.
One agent researches a document.
Another analyzes security risks.
Another summarizes the findings.
A final agent combines everything into a report.
The common architecture is to introduce a database as the central state store:
Agent → Database → Query → Transform → Agent
That works well when the shared state is structured data.
But what happens when the agents are primarily working with documents, source code, reports, logs, prompts, intermediate artifacts, and other files?
Maybe the database isn't always the best abstraction.
This is where Amazon S3 Files becomes interesting.
From Object Storage to a Shared File System
Amazon S3 has traditionally been accessed through object APIs.
Applications upload and download objects:
1
2
3
4
5
Application
↓
S3 API
↓
ObjectBut AI workloads frequently operate on files.
Agents need to:
- read documents
- create reports
- modify files
- inspect directories
- share intermediate artifacts
- persist working state
- collaborate on a common workspace
Amazon S3 Files changes the interface.
It provides file-system access to S3 data, allowing supported compute services such as Amazon EC2, ECS, EKS, and Lambda to access S3 through a shared file-system interface.
Instead of thinking only in terms of objects and API calls, an application can work with a mounted workspace such as:
1
/mnt/s3files/That is a very different abstraction for agent workloads.
The Agent Workspace
Imagine an AI research system running several agents on EC2.
The shared workspace could look like:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
/mnt/s3files/
├── tasks/
│ ├── customer-review/
│ └── market-analysis/
│
├── agents/
│ ├── research/
│ ├── security/
│ └── analysis/
│
├── reports/
│ ├── draft/
│ └── final/
│
├── shared/
│
└── memory/Now the agents don't necessarily need to exchange every intermediate result through a database.
One agent can produce:
1
/tasks/customer-review/research.mdAnother can read it and create:
1
/tasks/customer-review/security.mdA third agent can analyze both.
Finally, the reporting agent produces:
1
/reports/customer-review/final-report.mdThe filesystem becomes part of the agent coordination layer.
Now Add an Event-Driven Pipeline
The architecture becomes even more interesting when combined with event-driven processing.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
Customer
│
▼
Amazon S3
Input Bucket
│
│ S3 Event
▼
Amazon SQS
Task Queue
│
▼
Amazon EC2
AI Agent Swarm
│
│
▼
Amazon S3 Files
Shared Agent Workspace
│
├── Agent 1 → Research
├── Agent 2 → Security
├── Agent 3 → Analysis
└── Agent 4 → Summary
│
▼
Final AI ReportA document arrives in S3.
An event places a task onto SQS.
An EC2-based agent swarm consumes the task.
The agents use the shared S3 Files workspace for intermediate state and artifacts.
The final agent produces the completed report.
This creates a clean separation between:
Event ingestion → Agent execution → Shared state → Final output
Why This Is Interesting for AI Agents
The biggest benefit isn't simply storage.
It is the abstraction.
Traditional approach:
1
2
3
4
5
6
7
8
9
10
11
Agent
↓
Database
↓
Query
↓
Deserialize
↓
Process
↓
Another AgentFile-oriented approach:
1
2
3
4
5
Agent
↓
Shared S3 Files Workspace
↓
Another AgentFor workloads built around files, the second model can be much more natural.
Agents can leave behind artifacts that other agents can immediately consume.
For example:
1
2
3
4
5
research.md
security-analysis.md
code-review.md
test-results.json
final-report.mdThe files themselves become persistent collaboration artifacts.
But Don't Confuse Shared Files with a Database
This does not mean databases are obsolete.
A database is still the better choice when the application needs:
- transactional updates
- complex queries
- relational relationships
- indexes
- high-frequency structured state changes
- strict transactional guarantees
- analytical queries over structured records
S3 Files is particularly interesting when the state is primarily file-oriented.
Think:
Documents + Reports + Source Code + Artifacts + Agent Workspace
rather than:
Transactions + Users + Orders + Relational Records
The question isn't:
"Should databases disappear?"
The better question is:
"What is the natural representation of the state my agents are sharing?"
The Consistency Detail Matters
There is an important architectural detail here.
S3 Files provides file-system semantics and NFS close-to-open consistency.
That means after a client closes a file, a subsequent open can see the latest version.
However, synchronization between the file system and the underlying S3 bucket is eventually consistent.
AWS also documents that S3 changes can be imported into the file system within seconds, while filesystem writes are exported to S3 after batching.
So the architecture should not assume:
1
2
3
4
5
Write file
↓
Immediately query S3 API
↓
Guaranteed identical stateInstead, understand the synchronization boundary when designing the system.
And What About Race Conditions?
This is another important consideration for multi-agent systems.
If several agents can modify shared state simultaneously, simply putting them on the same filesystem does not automatically solve every distributed coordination problem.
For example:
1
2
3
Agent A ──┐
├──> shared task
Agent B ──┘Both agents might attempt to process the same task.
A production system should therefore introduce explicit coordination strategies such as:
- task claiming
- idempotent processing
- unique task identifiers
- clear ownership conventions
- retry handling
- conflict detection
The shared filesystem provides the workspace.
Your application still needs to define the coordination protocol.
The IAM Trap
A shared filesystem does not mean unrestricted access.
S3 Files introduces specific client permissions such as:
1
2
3
s3files:ClientMount
s3files:ClientWrite
s3files:ClientRootAccessThe application also needs appropriate S3 permissions for the underlying data, depending on the architecture.
For example:
1
2
3
4
5
6
7
EC2 IAM Role
│
├── S3 Files permissions
│
├── S3 object permissions
│
└── SQS permissionsThe correct approach is still least privilege.
An agent that only needs to read research files should not automatically receive write access to every shared directory.
For multi-agent systems, filesystem permissions and IAM policies become part of the security architecture—not an afterthought.
The Architecture
The architecture follows a simple event-driven flow:
Customer → Amazon S3 → Amazon SQS → Amazon EC2 AI Agent Swarm → Amazon S3 Files → Final AI Report
Inside the EC2 agent swarm:
Research Agent + Security Agent + Analysis Agent + Summary Agent
All agents access the same Amazon S3 Files shared workspace, containing:
/tasks · /agents · /reports · /shared · /memoryThe supporting security and operations layer includes:
IAM Least Privilege · S3 Files Policies · CloudWatch · Network Controls
The key architectural idea is:
Agents collaborate through shared state in the S3 Files workspace rather than passing every intermediate artifact through a database.
Architecture Flow
1
2
3
4
5
6
7
8
9
10
11
12
13
Customer
↓
Amazon S3
↓
Amazon SQS
↓
Amazon EC2
↓
AI Agent Swarm
↓
Amazon S3 Files
↓
Final AI ReportInside the Agent Swarm:
1
2
3
4
Research ─┐
Security ─┤
Analysis ─┼──→ S3 Files Shared Workspace
Summary ──┘So, Should We Stop Using Databases?
No.
And that's exactly what makes this architecture interesting.
The goal isn't to replace every database with a filesystem.
Instead, choose the storage abstraction based on the workload.
Use a database when you need:
Structured state → queries → transactions → relationships
Consider S3 Files when you need:
Files → artifacts → shared workspace → agent collaboration
And in many real systems, the answer will be both.
For example:
1
2
3
4
5
6
7
8
9
10
AI Agent Platform
│
┌────────────┴────────────┐
▼ ▼
Database S3 Files
Structured State Agent Workspace
Metadata Documents
Users Reports
Tasks Code
Status ArtifactsThe database can maintain structured control-plane state while S3 Files provides the shared data-plane workspace.
That may be a more realistic architecture than trying to force everything into one storage system.
The Bigger Question
AI agents are changing how we think about application state.
Traditional applications often revolve around:
APIs + Databases + Transactions
Agentic applications increasingly involve:
Context + Files + Tools + Memory + Intermediate Artifacts
That doesn't make databases less useful.
It means we may need to reconsider what "agent memory" actually means.
Sometimes memory is a row.
Sometimes it's a vector.
Sometimes it's a document.
Sometimes it's a code file.
And sometimes it is an entire workspace shared by multiple agents.
Final Thought
Amazon S3 Files makes an interesting architectural possibility much more practical:
What if an AI agent's shared memory is not a database at all—but a shared filesystem?
For document-heavy and artifact-heavy workloads, that could simplify how agents persist, inspect, and exchange intermediate state.
The real design question is not:
Database or filesystem?
It is:
What representation makes the agent workflow simplest, safest, and easiest to scale?
Community Question
If you were building a multi-agent AI system today, where would you keep its shared state?
A database, a shared filesystem like S3 Files, or a hybrid of both?
I'd be interested in hearing where you draw the line.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article