What happens when you upload something in AWS S3 ?
Every AWS Cloud user has used S3 at least once. But do we know what happens internally? Here is a detailed article about AWS S3 Standard — one of the largest distributed storage systems. Every AWS Cloud user has used S3 at least once. But do we know what happens internally? Here is a detailed article about AWS S3 Standard — one of the largest distributed storage systems.
𝗦𝟯 (𝗦𝗶𝗺𝗽𝗹𝗲 𝗦𝘁𝗼𝗿𝗮𝗴𝗲 𝗦𝗲𝗿𝘃𝗶𝗰𝗲) is an 𝗼𝗯𝗷𝗲𝗰𝘁-𝘀𝘁𝗼𝗿𝗮𝗴𝗲 service provided by Amazon Web Services (AWS) , used to store files such as images, videos, documents, backups, logs, and application data.
It has prominent features like:
• High durability
• High availability
• Scalability
• Data integrity
• Automatic failure recovery
• High durability
• High availability
• Scalability
• Data integrity
• Automatic failure recovery
So, What Happens When We Upload a File or Something to a Bucket?
✨ 𝗨𝗽𝗹𝗼𝗮𝗱
When we upload a file, S3 treats it as an object containing the data, key/name, and metadata.
✨ 𝗗𝗶𝘀𝘁𝗿𝗶𝗯𝘂𝘁𝗲𝗱 𝗦𝘁𝗼𝗿𝗮𝗴𝗲
When we upload a file, S3 treats it as an object containing the data, key/name, and metadata.
✨ 𝗗𝗶𝘀𝘁𝗿𝗶𝗯𝘂𝘁𝗲𝗱 𝗦𝘁𝗼𝗿𝗮𝗴𝗲
S3 doesn't depend on a single disk. The data is distributed across a large storage infrastructure and, for multi-AZ storage classes, across multiple Availability Zones.
✨ 𝗗𝗮𝘁𝗮 𝗣𝗿𝗼𝘁𝗲𝗰𝘁𝗶𝗼𝗻
S3 uses redundancy and erasure coding to protect data. Data can be divided into pieces with additional parity information, allowing missing data to be reconstructed. S3 uses Reed-Solomon erasure coding.
✨ 𝗗𝗮𝘁𝗮 𝗜𝗻𝘁𝗲𝗴𝗿𝗶𝘁𝘆
S3 uses checksums to detect data corruption and verify the integrity of stored data. Generally, by default, it uses the CRC64NVME checksum algorithm for checksum calculations.
✨ 𝗙𝗮𝗶𝗹𝘂𝗿𝗲 𝗥𝗲𝗰𝗼𝘃𝗲𝗿𝘆
S3 is designed around the assumption that hardware failures will happen. When a failure occurs, S3 detects it, recovers the affected data, and restores the required redundancy.
✨ 𝗖𝗼𝗻𝘁𝗶𝗻𝘂𝗼𝘂𝘀 𝗕𝗮𝗰𝗸𝗴𝗿𝗼𝘂𝗻𝗱 𝗪𝗼𝗿𝗸
Behind the scenes, S3 continuously performs tasks such as repair, replication, storage management, tiering, and data placement.
✨ 𝗛𝗲𝗮𝘁 𝗠𝗮𝗻𝗮𝗴𝗲𝗺𝗲𝗻𝘁
S3 also distributes workloads across its storage infrastructure to avoid creating hotspots on heavily accessed storage devices.
✨ 𝗦𝗵𝗮𝗿𝗱𝗦𝘁𝗼𝗿𝗲
Amazon has publicly discussed ShardStore, a lower-level storage component used within S3's storage architecture for managing data on individual storage devices.
In Short:𝗨𝗽𝗹𝗼𝗮𝗱 → 𝗗𝗶𝘀𝘁𝗿𝗶𝗯𝘂𝘁𝗲 → 𝗣𝗿𝗼𝘁𝗲𝗰𝘁 → 𝗩𝗲𝗿𝗶𝗳𝘆 → 𝗗𝗲𝘁𝗲𝗰𝘁 𝗙𝗮𝗶𝗹𝘂𝗿𝗲𝘀 → 𝗥𝗲𝗰𝗼𝘃𝗲𝗿 → 𝗥𝗲𝗽𝗮𝗶𝗿 → 𝗖𝗼𝗻𝘁𝗶𝗻𝘂𝗼𝘂𝘀𝗹𝘆 𝗠𝗮𝗻𝗮𝗴𝗲That's the engineering behind what looks like a simple 𝗣𝗨𝗧 𝗢𝗯𝗷𝗲𝗰𝘁 𝗔𝗣𝗜 call.
Follow to stay connected, and let’s meet again soon to explore another interesting AWS service! . Comment Your Thoughts and Which Service You need Explanation. Happy Clouding ☁️.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article