I Built a Backup System Instead of Just Learning S3
What started as a simple backup project turned into a deeper dive into S3-compatible storage, reliability, checksums, retries, retention, and observability. Here’s what I learned while building it.
I Built a Backup System Instead of Just Learning S3
When I started exploring object storage, I didn't want to stop at creating an S3 bucket and uploading a file.
I wanted to understand what actually happens when a backup system has to deal with failures, retries, corrupted data, storage limits, and recovery.
So I decided to build one.
I called it StorVault — a Linux-based backup and recovery system designed around S3-compatible object storage.
The idea
At a high level, the system does this:
Files → Backup Agent → Object Storage → Verification → Retention
And when something goes wrong:
Storage/File Failure → Retry → Verification → Recovery
The goal wasn't to create a production-ready enterprise backup platform. The goal was to understand the engineering problems behind one.
What I used
- C++ — multithreaded backup agent
- Python — orchestration and automation
- Bash/Linux — system-level operations
- S3 API — object-storage interface
- MinIO — S3-compatible object storage
- Ceph RGW — another S3-compatible storage backend
- Prometheus — monitoring and metrics
One of the interesting parts was that the application could communicate with different storage backends through the same S3 interface.
That made me realize something important:
An API abstraction becomes really powerful when you actually have multiple implementations behind it.
Why checksums matter
A backup isn't useful just because the file was successfully uploaded.
What if the file changes during transfer?
What if the stored object becomes corrupted?
What if the upload appears successful but the recovered file isn't identical to the original?
That's where checksums come in.
The system calculates a checksum for the data and uses it during verification and recovery.
Conceptually:
1
2
3
4
5
6
7
8
9
10
11
12
13
Original File
↓
Calculate Checksum
↓
Upload
↓
Download / Restore
↓
Calculate Checksum Again
↓
Compare
↓
✓ Valid / ✗ Corrupted
This was one of the first moments where I started thinking about backups as a reliability problem, rather than simply a storage problem.
Failures are part of the design
Another thing I wanted to understand was retries.
A network request can fail.
A storage service can temporarily become unavailable.
A connection can timeout.
Instead of treating every failure as a fatal error, the system can retry operations where it makes sense.
That led me to think about questions like:
- How many times should an operation retry?
- How long should we wait between retries?
- Which errors should be retried?
- What happens if the retry also fails?
- How do we prevent duplicate or inconsistent backups?
These questions are much closer to real-world infrastructure engineering than simply running a successful upload command.
Adding retention
Keeping every backup forever isn't practical.
So I also explored retention policies — deciding how long backups should remain available before they can be removed.
For example:
1
2
3
4
5
Day 1 → Backup A
Day 2 → Backup B
Day 3 → Backup C
...
Day 30 → Older backups become eligible for deletion
This introduces another important infrastructure trade-off:
Recovery history vs. storage cost.
Why Prometheus?
Once a system starts performing automated backups, I don't want to constantly look at logs to figure out whether it's healthy.
I want metrics.
Things like:
- Number of backups completed
- Backup failures
- Restore failures
- Retry counts
- Backup duration
- Objects processed
- Storage-related errors
Prometheus gives the system a way to expose these metrics so its behavior can be monitored over time.
What I learned
The biggest lesson wasn't about S3.
It was about thinking in failure scenarios.
When you're building a small project, it's easy to think:
"What happens when everything works?"
Infrastructure engineering forces you to ask:
"What happens when everything doesn't work?"
That shift changed how I approach projects.
I now think more about:
Reliability → Failure handling → Observability → Recovery
rather than simply:
Code → Run → Done
What's next?
There are still plenty of things I'd like to experiment with:
- Encryption at rest
- Incremental backups
- Deduplication
- Backup scheduling
- More detailed observability
- Kubernetes deployment
- Automated disaster-recovery testing
- Cost-aware storage tiers
That's what I like about building infrastructure projects.
You start with one simple question and suddenly discover ten more interesting problems.
And that's probably the best part of learning cloud engineering.
Don't just learn the service. Build something that makes you understand why the service exists.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article