S3 - Deleting recent data from 2.5 PB + versioned bucket.
Deleting the most recent data from a pool of 500 million, 2.5 PetaBytes of versioned S3 bucket.
Overview
2.5 PB in size. Over 500 million objects. That was what I inherited from ten years of production data in a single S3 bucket.
When I joined my current company as an AWS architect, my first task was to do a cost analysis on the production account as the bill amount was quite high and it affected the profit margin. S3 was one of the three resources consuming 65% of the bill, around $35,000 per month.
The S3 bucket had Intelligent tiering enabled, meaning the newest object sits in the frequent access list, costs as much as the standard storage and older objects sit in the infrequently accessed storage type, which costs less compared to the standard storage. For my use case I had to remove the frequent access storage class data.
Here, the production data was already replicated to a different storage and what I needed was to:-
- Move 1-3 year data to S3 Glacier Deep Archive.
- Delete data 3 years and above.
- Delete 0-1 year data (versioned).
The real challenge for me was to identify the data that needed to be deleted in a cost effective and faster way. The AWS S3 console does not give an option for my use case because the UI caps at 1000 object selection per page and does not offer a date filter.
Alternative is to use S3 APIs to list and delete objects one by one using the date parameter in the command.
Move 1-3 year data to S3 Glacier Deep Archive
I configured a S3 lifecycle policy to move all the objects older than 1 year, policy is as below:-
{ "Rules": [ { "ID": "archive-1y-to-3y", "Status": "Enabled", "Filter": { "And": { "Prefix": "<filter_folder>", "ObjectSizeGreaterThan": 131072 } }, "Transitions": [ { "Days": 365, "StorageClass": "DEEP_ARCHIVE" } ] } ]}Deep archive transition costs $0.05 per 1,000 objects. “ObjectSizeGreaterThan” filters the small objects, saving the transition fee.

Delete data 3 years and above.
For deleting current and non current versions of objects, I configured a S3 lifecycle policy as below.
{ "Rules": [ { "ID": "delete_3Year+", "Status": "Enabled", "Filter": { "Prefix": "<filter_folder>" }, "Expiration": { "Days": 1095 }, "NoncurrentVersionExpiration": { "NoncurrentDays": 1 }, "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 } } ]}
My requirement was to permanently delete objects older than 3 years (1095 days), the S3 bucket had versioning enabled, thus an additional policy was added to permanently delete the non-current version of objects. If the lifecycle policy for the non-current version is not added, only the current version will be deleted and the older version remains.
After the current version expires, the non-current version of objects is set to expire a day after. The deletion action creates a delete marker and it is not removed by default, which requires an additional policy, since it can’t co-exist with the expiration rule.
{ "ID": "cleanup_expired_delete_markers", "Status": "Enabled", "Filter": { "Prefix": "<filter_folder>" }, "Expiration": { "ExpiredObjectDeleteMarker": true }}Delete 0-1 year data (versioned).
This part gets more interesting. Now the remaining 0-1 years old data had to be deleted. Deleting the most recent data is something which cannot be achieved from lifecycle rules. I achieved this using a three step process.
A: Enable inventory of S3 bucket objects.
B: Filter the inventory to create a new delete manifest.
C: Use a Lambda function to delete the objects from the inventory using the delete manifest.
A : AWS S3 inventory is an AWS storage management feature that automatically generates scheduled lists of objects and their metadata within S3 bucket. This is a cost effective solution to track your AWS S3 bucket. You can have your inventory written to the same or different bucket.
aws s3api put-bucket-inventory-configuration --bucket $BUCKET --id <Inventory_name>\--inventory-configuration '{ "Id":"<Inventory_name>","IsEnabled":true, "Destination":{"S3BucketDestination":{"Bucket":"arn:aws:s3:::<destination_bucket>","Format":"CSV"}}, "IncludedObjectVersions":"All", "Schedule":{"Frequency":"Daily"}, "Filter":{"Prefix":"<folder_prefix_if_any>"}, "OptionalFields":["Size","LastModifiedDate","StorageClass"] }' --region $REGIONThe first report will land in 24-48h to the path
<destination_bucket>/…/data , along with a file manifest.json and gzipped CSV files (these are the data files). Basically this manifest.json is similar to an indexer of the gzipped CSV file.Turn off the inventory once the first report is generated, or else daily inventory reports will cost on S3 storage as each report is ~500MB.
B : We’ll need the
manifest.json generated from the latest run, download it. I used the below script to filter out the objects I need to delete.
import json, gzip, csv, os, boto3REGION, DESTBUCKET = "<REGION>", "<destination_bucket>"CUTOFF = "2025-06-01T00:00:00.000Z" ##Desired cutoff date; should be updated in each run.ARCHIVE = {"DEEP_ARCHIVE","GLACIER","GLACIER_IR"} ##Check your own storage filters3 = boto3.client("s3", region_name=REGION)mani = json.load(open("manifest.json"))sch = [c.strip() for c in mani["fileSchema"].split(",")]iB,iK,iV = sch.index("Bucket"), sch.index("Key"), sch.index("VersionId")iDM,iLM,iSC = sch.index("IsDeleteMarker"), sch.index("LastModifiedDate"), sch.index("StorageClass")out = csv.writer(open("delete-manifest.csv","w",newline=""))delete = keep = 0for f in mani["files"]: # only THIS run's data files local = os.path.basename(f["key"]) s3.download_file(DESTBUCKET, f["key"], local) with gzip.open(local,"rt") as fh: for r in csv.reader(fh): is_marker = r[iDM].strip().lower() == "true" sc = r[iSC] if iSC < len(r) and r[iSC] else "STANDARD" if (not is_marker) and r[iLM] >= CUTOFF and sc not in ARCHIVE: out.writerow([r[iB], r[iK], r[iV]]); delete += 1 # keys stay URL-encoded else: keep += 1print(f"versions_to_delete={delete} versions_kept={keep}")INTELLIGENT_TIERING is not added in the ARCHIVE set so that the INTELLIGENT_TIERING passes the filter and gets deleted, which is intended.This script downloads the required gzipped data files and generates a new
delete-manifest.csv file. Upload this to the destination bucket.This is what we use as input to delete. We will have to generate an
ETag for the uploaded version. ETag is a unique identifier assigned to a specific version of an object uploaded. Primarily used to verify data integrity. Use the below command to generate ETag for the uploaded file.aws s3api head-object --bucket <bucket_name> \ --key <path_to_file/>delete-manifest.csv \ --query ETag --output text --region <region> | tr -d '"' This command gets the
ETag without double quotes, this is the point where I spent some time due to the double quotes for ETag .Commands provided by most of the documentations,
ETag value comes with double quotes and feeding that to inline JSON produces a double quote that breaks the parser.C : Here we use Batch Operation to invoke AWS Lambda function fed with
delete-manifest.csv to delete the targeted versions. S3 Batch operation does not have delete by versionID operation by native. So using Lambda function is the only route to delete versioned objects.First we need a batch operation job. Create a batch operation role, add permissions to
- Delete object from the respective bucket.
- Fetch delete-manifest from destination bucket.
- Upload report to destination bucket.
Create a Lambda function with runtime Python 3.12, timeout 1 minute. Use the below code in the Lambda function.
import loggingfrom urllib import parseimport boto3from botocore.exceptions import ClientErrorlog = logging.getLogger(); log.setLevel("INFO")s3 = boto3.client("s3")def lambda_handler(event, context): task = event["tasks"][0] key = parse.unquote_plus(task["s3Key"], encoding="utf-8") vid = task.get("s3VersionId") bkt = task["s3BucketArn"].split(":")[-1] code, msg = "Succeeded", "" try: kwargs = {"Bucket": bkt, "Key": key} if vid: kwargs["VersionId"] = vid s3.delete_object(**kwargs) msg = f"deleted {key} v={vid}" except ClientError as e: ec = e.response["Error"]["Code"] code = "TemporaryFailure" if ec in ("InternalError","SlowDown","ServiceUnavailable") else "PermanentFailure" msg = f"{ec}: {e}"; log.exception(msg) except Exception as e: code, msg = "PermanentFailure", str(e); log.exception(msg) return {"invocationSchemaVersion": event["invocationSchemaVersion"], "treatMissingKeysAs": "PermanentFailure", "invocationId": event["invocationId"], "results": [{"taskId": task["taskId"], "resultCode": code, "resultString": msg}]}Use the below command to create the Batch Operation
aws s3control create-job --account-id <AWS_account_id> --region <region> --no-cli-pager \ --operation '{"LambdaInvoke":{"FunctionArn":"arn:aws:lambda:<region>:<AWS_account_id>:function:<Lambda_function_created>"}}' \ --manifest '{ "Spec":{"Format":"S3BatchOperations_CSV_20180820"}, "Location":{"ObjectArn":"arn:aws:s3:::<manifest_uploaded_bucket>/path_to/delete-manifest.csv","ETag":"<Paste_ETAG_Here>"}}' \ --report '{"Bucket":"arn:aws:s3:::<manifest_uploaded_bucket>","Prefix":"path_to/job-reports","Format":"Report_CSV_20180820","Enabled":true,"ReportScope":"AllTasks"}' \ --priority 10 --role-arn arn:aws:iam::<AWS_account_id>:role/batch_deletion_role \ --confirmation-requiredUpdate the command with the necessary values. The command creates a batch operation, but won’t run it unless done manually after verifying.
Batch operation result metric
Metric | Value |
|---|---|
Object versions deleted | 88,314,766 |
Failures | 0 (100% success) |
Wall clock | 3h 23m 13s |
Throughput | ~7,243 deletes/sec |
Manifest size | ~10.9 GB |
Events Outcome
Event | Effect |
|---|---|
Deep Archive transition lifecycle policy | 807 TB moved GIR to Deep Archive |
3-year expiration lifecycle policy | ~429 TB removed permanently |
Batch Operations job | 322 TB, 88.3M objects removed permanently |
Cost Outcome
| Before | After |
|---|---|---|
Monthly S3 spend | ~$35,000 / month | Less than $8,000 / month |
Reduction | | 77% ($324,000 Annually) |
88M objects deleted at 7,200/sec roughly 3.5 hours. Same work in single-client delete-object takes 10 days at 100/sec rate.
Neither the AWS console nor the API supports deleting objects by date. I’m working on a more convenient system to sort this out.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article