AWS Builder Center

S3 - Deleting recent data from 2.5 PB + versioned bucket.

Deleting the most recent data from a pool of 500 million, 2.5 PetaBytes of versioned S3 bucket.

Solutions Architect by profession, Tech enthusiast by nature.

Overview

2.5 PB in size. Over 500 million objects. That was what I inherited from ten years of production data in a single S3 bucket.
When I joined my current company as an AWS architect, my first task was to do a cost analysis on the production account as the bill amount was quite high and it affected the profit margin. S3 was one of the three resources consuming 65% of the bill, around $35,000 per month.
The S3 bucket had Intelligent tiering enabled, meaning the newest object sits in the frequent access list, costs as much as the standard storage and older objects sit in the infrequently accessed storage type, which costs less compared to the standard storage. For my use case I had to remove the frequent access storage class data.
Here, the production data was already replicated to a different storage and what I needed was to:-
  1. Move 1-3 year data to S3 Glacier Deep Archive.
  1. Delete data 3 years and above.
  1. Delete 0-1 year data (versioned).
The real challenge for me was to identify the data that needed to be deleted in a cost effective and faster way. The AWS S3 console does not give an option for my use case because the UI caps at 1000 object selection per page and does not offer a date filter. 
Alternative is to use S3 APIs to list and delete objects one by one using the date parameter in the command.

Move 1-3 year data to S3 Glacier Deep Archive

I configured a S3 lifecycle policy to move all the objects older than 1 year, policy is as below:-
{
"Rules": [
{
"ID": "archive-1y-to-3y",
"Status": "Enabled",
"Filter": {
"And": {
"Prefix": "<filter_folder>",
"ObjectSizeGreaterThan": 131072
}
},
"Transitions": [
{ "Days": 365, "StorageClass": "DEEP_ARCHIVE" }
]
}
]
}
Deep archive transition costs $0.05 per 1,000 objects. “ObjectSizeGreaterThan” filters the small objects, saving the transition fee.
Console output of Glacier Deep Archive policy


Delete data 3 years and above.

For deleting current and non current versions of objects, I configured a S3 lifecycle policy as below.
{
"Rules": [
{
"ID": "delete_3Year+",
"Status": "Enabled",
"Filter": { "Prefix": "<filter_folder>" },
"Expiration": { "Days": 1095 },
"NoncurrentVersionExpiration": { "NoncurrentDays": 1 },
"AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
}
]
}
Console output of Object delete policy

My requirement was to permanently delete objects older than 3 years (1095 days), the S3 bucket had versioning enabled, thus an additional policy was added to permanently delete the non-current version of objects. If the lifecycle policy for the non-current version is not added, only the current version will be deleted and the older version remains.
After the current version expires, the non-current version of objects is set to expire a day after. The deletion action creates a delete marker and it is not removed by default, which requires an additional policy, since it can’t co-exist with the expiration rule.
{
"ID": "cleanup_expired_delete_markers",
"Status": "Enabled",
"Filter": { "Prefix": "<filter_folder>" },
"Expiration": { "ExpiredObjectDeleteMarker": true }
}

Delete 0-1 year data (versioned).

This part gets more interesting. Now the remaining 0-1 years old data had to be deleted. Deleting the most recent data is something which cannot be achieved from lifecycle rules. I achieved this using a three step process. A: Enable inventory of S3 bucket objects. B: Filter the inventory to create a new delete manifest. C: Use a Lambda function to delete the objects from the inventory using the delete manifest. A : AWS S3 inventory is an AWS storage management feature that automatically generates scheduled lists of objects and their metadata within S3 bucket. This is a cost effective solution to track your AWS S3 bucket. You can have your inventory written to the same or different bucket.
aws s3api put-bucket-inventory-configuration --bucket $BUCKET --id <Inventory_name>\
--inventory-configuration '{
"Id":"<Inventory_name>",
"IsEnabled":true,
"Destination":{"S3BucketDestination":{"Bucket":"arn:aws:s3:::<destination_bucket>","Format":"CSV"}},
"IncludedObjectVersions":"All",
"Schedule":{"Frequency":"Daily"},
"Filter":{"Prefix":"<folder_prefix_if_any>"},
"OptionalFields":["Size","LastModifiedDate","StorageClass"] }' --region $REGION
The first report will land in 24-48h to the path  <destination_bucket>/…/data , along with a file  manifest.json  and gzipped CSV files (these are the data files). Basically this manifest.json is similar to an indexer of the gzipped CSV file.
Turn off the inventory once the first report is generated, or else daily inventory reports will cost on S3 storage as each report is ~500MB.
B : We’ll need the  manifest.json  generated from the latest run, download it. 
I used the below script to filter out the objects I need to delete.
import json, gzip, csv, os, boto3
REGION, DESTBUCKET = "<REGION>", "<destination_bucket>"
CUTOFF = "2025-06-01T00:00:00.000Z" ##Desired cutoff date; should be updated in each run.
ARCHIVE = {"DEEP_ARCHIVE","GLACIER","GLACIER_IR"} ##Check your own storage filter
s3 = boto3.client("s3", region_name=REGION)
mani = json.load(open("manifest.json"))
sch = [c.strip() for c in mani["fileSchema"].split(",")]
iB,iK,iV = sch.index("Bucket"), sch.index("Key"), sch.index("VersionId")
iDM,iLM,iSC = sch.index("IsDeleteMarker"), sch.index("LastModifiedDate"), sch.index("StorageClass")
out = csv.writer(open("delete-manifest.csv","w",newline=""))
delete = keep = 0
for f in mani["files"]: # only THIS run's data files
local = os.path.basename(f["key"])
s3.download_file(DESTBUCKET, f["key"], local)
with gzip.open(local,"rt") as fh:
for r in csv.reader(fh):
is_marker = r[iDM].strip().lower() == "true"
sc = r[iSC] if iSC &lt; len(r) and r[iSC] else "STANDARD"
if (not is_marker) and r[iLM] &gt;= CUTOFF and sc not in ARCHIVE:
out.writerow([r[iB], r[iK], r[iV]]); delete += 1 # keys stay URL-encoded
else:
keep += 1
print(f"versions_to_delete={delete} versions_kept={keep}")
 INTELLIGENT_TIERING  is not added in the  ARCHIVE  set so that the  INTELLIGENT_TIERING  passes the filter and gets deleted, which is intended.
This script downloads the required gzipped data files and generates a new  delete-manifest.csv file. Upload this to the destination bucket.
This is what we use as input to delete. We will have to generate an  ETag  for the uploaded version.  ETag  is a unique identifier assigned to a specific version of an object uploaded. Primarily used to verify data integrity. Use the below command to generate  ETag  for the uploaded file.
aws s3api head-object --bucket <bucket_name> \
--key <path_to_file/>delete-manifest.csv \
--query ETag --output text --region <region> | tr -d '"'
This command gets the  ETag  without double quotes, this is the point where I spent some time due to the double quotes for  ETag .
Commands provided by most of the documentations,  ETag  value comes with double quotes and feeding that to inline JSON produces a double quote that breaks the parser.
C : Here we use Batch Operation to invoke AWS Lambda function fed with  delete-manifest.csv  to delete the targeted versions. S3 Batch operation does not have delete by versionID operation by native. So using Lambda function is the only route to delete versioned objects.
First we need a batch operation job. Create a batch operation role, add permissions to 
  • Delete object from the respective bucket.
  • Fetch delete-manifest from destination bucket.
  • Upload report to destination bucket.
Create a Lambda function with runtime Python 3.12, timeout 1 minute. Use the below code in the Lambda function.
import logging
from urllib import parse
import boto3
from botocore.exceptions import ClientError
log = logging.getLogger(); log.setLevel("INFO")
s3 = boto3.client("s3")
def lambda_handler(event, context):
task = event["tasks"][0]
key = parse.unquote_plus(task["s3Key"], encoding="utf-8")
vid = task.get("s3VersionId")
bkt = task["s3BucketArn"].split(":")[-1]
code, msg = "Succeeded", ""
try:
kwargs = {"Bucket": bkt, "Key": key}
if vid: kwargs["VersionId"] = vid
s3.delete_object(**kwargs)
msg = f"deleted {key} v={vid}"
except ClientError as e:
ec = e.response["Error"]["Code"]
code = "TemporaryFailure" if ec in ("InternalError","SlowDown","ServiceUnavailable") else "PermanentFailure"
msg = f"{ec}: {e}"; log.exception(msg)
except Exception as e:
code, msg = "PermanentFailure", str(e); log.exception(msg)
return {"invocationSchemaVersion": event["invocationSchemaVersion"],
"treatMissingKeysAs": "PermanentFailure",
"invocationId": event["invocationId"],
"results": [{"taskId": task["taskId"], "resultCode": code, "resultString": msg}]}
Use the below command to create the Batch Operation
aws s3control create-job --account-id <AWS_account_id> --region <region> --no-cli-pager \
--operation '{"LambdaInvoke":{"FunctionArn":"arn:aws:lambda:<region>:<AWS_account_id>:function:<Lambda_function_created>"}}' \
--manifest '{
"Spec":{"Format":"S3BatchOperations_CSV_20180820"},
"Location":{"ObjectArn":"arn:aws:s3:::<manifest_uploaded_bucket>/path_to/delete-manifest.csv","ETag":"<Paste_ETAG_Here>"}}' \
--report '{"Bucket":"arn:aws:s3:::<manifest_uploaded_bucket>","Prefix":"path_to/job-reports","Format":"Report_CSV_20180820","Enabled":true,"ReportScope":"AllTasks"}' \
--priority 10 --role-arn arn:aws:iam::<AWS_account_id>:role/batch_deletion_role \
--confirmation-required
Update the command with the necessary values. The command creates a batch operation, but won’t run it unless done manually after verifying.
Batch operation result metric
Metric
Value
Object versions deleted
88,314,766
Failures
0 (100% success)
Wall clock
3h 23m 13s
Throughput
~7,243 deletes/sec
Manifest size
~10.9 GB
Events Outcome
Event
Effect
Deep Archive transition lifecycle policy
807 TB moved GIR to Deep Archive
3-year expiration lifecycle policy
~429 TB removed permanently
Batch Operations job
322 TB, 88.3M objects removed permanently
Cost Outcome

Before
After
Monthly S3 spend
~$35,000 / month
Less than $8,000 / month
Reduction

77% ($324,000 Annually)
88M objects deleted at 7,200/sec roughly 3.5 hours. Same work in single-client delete-object takes 10 days at 100/sec rate.
Neither the AWS console nor the API supports deleting objects by date. I’m working on a more convenient system to sort this out.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article