AWS Builder Center
Save Money on AWS S3: Automate Storage Tiering with Terraform ๐Ÿ’ฐ

Save Money on AWS S3: Automate Storage Tiering with Terraform ๐Ÿ’ฐ

Automate S3 Storage Tiering with Terraform

Series: Save on AWS with Terraform (21 articles)

  1. 2
    Save Money on AWS S3: Automate Storage Tiering with Terraform ๐Ÿ’ฐ This article
If you're storing data in AWS S3, you've probably noticed that storage costs can add up quickly. The good news? You can significantly reduce these costs by automatically moving objects that haven't been accessed in a while to cheaper storage tiers. In this article, I'll show you how to implement S3 lifecycle policies using Terraform to do exactly that.

๐Ÿ“Š Understanding S3 Storage Classes

Before diving into the code, let's quickly review the S3 storage classes:
  • ๐Ÿ”ฅ S3 Standard - Frequently accessed data (~$0.023/GB)
  • ๐ŸŒก๏ธ S3 Standard-IA - Infrequent access (~$0.0125/GB) - 46% savings
  • ๐Ÿง  S3 Intelligent-Tiering - Auto-optimizes based on access patterns
  • ๐ŸงŠ S3 Glacier Instant Retrieval - Archive with instant access (~$0.004/GB)
  • โ„๏ธ S3 Glacier Deep Archive - Long-term archive (~$0.00099/GB) - 95%+ savings

๐Ÿ’ธ The Problem: Forgotten Data Costing You Money

Many organizations have S3 buckets filled with objects that were uploaded once and rarely accessed again:
๐Ÿ“‹ Old application logs
๐Ÿ’พ Backup files
๐Ÿ“ˆ Historical reports
๐Ÿ“ Archived uploads
๐Ÿ—‚๏ธ Legacy projects
Without lifecycle policies, all this data sits in S3 Standard storage, bleeding money every month.

โœจ The Solution: Terraform Lifecycle Policies

Let's create Terraform configurations that automatically transition objects to cheaper storage tiers based on their age.

๐ŸŽฏ Basic Example: Simple Tiering Strategy

Here's a straightforward lifecycle policy that moves objects through different storage tiers:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
resource "aws_s3_bucket" "cost_optimized_bucket" {
bucket = "my-cost-optimized-bucket"
}

resource "aws_s3_bucket_lifecycle_configuration" "bucket_lifecycle" {
bucket = aws_s3_bucket.cost_optimized_bucket.id

rule {
id = "transition-old-objects"
status = "Enabled"

transition {
days = 30
storage_class = "STANDARD_IA"
}

transition {
days = 90
storage_class = "GLACIER_IR"
}

transition {
days = 180
storage_class = "DEEP_ARCHIVE"
}
}
}
What this does:
  • ๐Ÿ“… Day 30 โ†’ Standard-IA
  • ๐Ÿ“… Day 90 โ†’ Glacier Instant Retrieval
  • ๐Ÿ“… Day 180 โ†’ Deep Archive

๐Ÿš€ Advanced Example: Multiple Rules for Different Prefixes

Different data types need different strategies. Here's how to handle logs, backups, uploads, and temp files:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
resource "aws_s3_bucket" "application_data" {
bucket = "my-application-data-bucket"
}

resource "aws_s3_bucket_lifecycle_configuration" "advanced_lifecycle" {
bucket = aws_s3_bucket.application_data.id

# Rule for application logs - aggressive archiving
rule {
id = "archive-logs"
status = "Enabled"

filter {
prefix = "logs/"
}

transition {
days = 7
storage_class = "STANDARD_IA"
}

transition {
days = 30
storage_class = "GLACIER_IR"
}

transition {
days = 90
storage_class = "DEEP_ARCHIVE"
}

expiration {
days = 365 # Delete after 1 year
}
}

# Rule for backups - immediate archival
rule {
id = "archive-backups"
status = "Enabled"

filter {
prefix = "backups/"
}

transition {
days = 1
storage_class = "GLACIER_IR"
}

transition {
days = 30
storage_class = "DEEP_ARCHIVE"
}

expiration {
days = 2555 # 7 years for compliance
}
}

# Rule for user uploads - moderate archiving
rule {
id = "transition-user-content"
status = "Enabled"

filter {
prefix = "uploads/"
}

transition {
days = 60
storage_class = "STANDARD_IA"
}

transition {
days = 180
storage_class = "GLACIER_IR"
}
}

# Rule for temporary files - auto-cleanup
rule {
id = "cleanup-temp-files"
status = "Enabled"

filter {
prefix = "temp/"
}

expiration {
days = 7
}
}
}

๐Ÿง  Using Intelligent-Tiering for Unknown Access Patterns

Not sure about access patterns? Let AWS handle it automatically:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
resource "aws_s3_bucket" "intelligent_bucket" {
bucket = "my-intelligent-bucket"
}

resource "aws_s3_bucket_lifecycle_configuration" "intelligent_lifecycle" {
bucket = aws_s3_bucket.intelligent_bucket.id

rule {
id = "enable-intelligent-tiering"
status = "Enabled"

transition {
days = 0
storage_class = "INTELLIGENT_TIERING"
}
}
}

resource "aws_s3_bucket_intelligent_tiering_configuration" "intelligent_config" {
bucket = aws_s3_bucket.intelligent_bucket.id
name = "EntireBucket"

tiering {
access_tier = "ARCHIVE_ACCESS"
days = 90
}

tiering {
access_tier = "DEEP_ARCHIVE_ACCESS"
days = 180
}
}

๐Ÿท๏ธ Filtering by Tags

Create lifecycle rules based on object tags for more granular control:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
resource "aws_s3_bucket_lifecycle_configuration" "tag_based_lifecycle" {
bucket = aws_s3_bucket.cost_optimized_bucket.id

rule {
id = "archive-by-tag"
status = "Enabled"

filter {
tag {
key = "archive"
value = "true"
}
}

transition {
days = 1
storage_class = "GLACIER_IR"
}
}

rule {
id = "delete-temporary"
status = "Enabled"

filter {
and {
prefix = "temp/"
tags = {
type = "temporary"
}
}
}

expiration {
days = 30
}
}
}

๐Ÿ† Complete Production-Ready Example

This example includes versioning, noncurrent version management, and cleanup rules:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
resource "aws_s3_bucket" "production_data" {
bucket = "my-production-data-${var.environment}"

tags = {
Environment = var.environment
ManagedBy = "Terraform"
Purpose = "Cost-optimized storage"
}
}

resource "aws_s3_bucket_versioning" "production_versioning" {
bucket = aws_s3_bucket.production_data.id
versioning_configuration {
status = "Enabled"
}
}

resource "aws_s3_bucket_lifecycle_configuration" "production_lifecycle" {
bucket = aws_s3_bucket.production_data.id

# Current version lifecycle
rule {
id = "transition-current-versions"
status = "Enabled"

transition {
days = 30
storage_class = "STANDARD_IA"
}

transition {
days = 90
storage_class = "GLACIER_IR"
}

transition {
days = 180
storage_class = "DEEP_ARCHIVE"
}
}

# Noncurrent version lifecycle
rule {
id = "transition-noncurrent-versions"
status = "Enabled"

noncurrent_version_transition {
noncurrent_days = 7
storage_class = "STANDARD_IA"
}

noncurrent_version_transition {
noncurrent_days = 30
storage_class = "GLACIER_IR"
}

noncurrent_version_expiration {
noncurrent_days = 90
}
}

# Clean up incomplete multipart uploads
rule {
id = "cleanup-incomplete-uploads"
status = "Enabled"

abort_incomplete_multipart_upload {
days_after_initiation = 7
}
}

# Clean up delete markers
rule {
id = "cleanup-delete-markers"
status = "Enabled"

expiration {
expired_object_delete_marker = true
}
}
}

๐Ÿ’ต Calculating Your Savings

Example: 10 TB of data in S3 Standard that hasn't been accessed in 6 months
Before:
  • 10 TB ร— $0.023/GB = $230/month
After:
  • 10 TB in Glacier Deep Archive ร— $0.00099/GB = $9.90/month
๐Ÿ’ฐ Savings: $220.10/month (95.7% reduction!)
๐Ÿ“ˆ Annual savings: $2,641.20

โœ… Best Practices

  1. ๐ŸŽฏ Start Conservative - Begin with longer transition periods and adjust based on data
  2. ๐Ÿ“Š Monitor Access Patterns - Use S3 Storage Lens or CloudWatch
  3. ๐Ÿงช Test First - Apply policies to a test bucket before production
  4. ๐Ÿ’ธ Consider Retrieval Costs - Glacier has retrieval fees
  5. ๐Ÿง  Use Intelligent-Tiering - When uncertain, let AWS optimize
  6. ๐Ÿ—‚๏ธ Clean Up Old Versions - Don't forget noncurrent versions in versioned buckets

๐Ÿ“ˆ Monitoring and Validation

Track your lifecycle policies effectiveness:
1
2
3
4
5
6
# Check lifecycle rules
aws s3api get-bucket-lifecycle-configuration --bucket your-bucket-name

# View storage metrics
aws s3api list-objects-v2 --bucket your-bucket-name \
--query 'Contents[*].[Key,StorageClass,LastModified]' --output table
Use AWS Cost Explorer to track storage cost reductions over time.

โš ๏ธ Common Pitfalls to Avoid

  • โฉ Too Aggressive Transitions - Moving data to Glacier too quickly incurs retrieval costs
  • โฑ๏ธ Minimum Storage Durations - Standard-IA and Glacier have 30-90 day minimums
  • ๐Ÿ“ฆ Small Objects - Objects <128 KB are charged for 128 KB in IA/Glacier
  • ๐Ÿงน Incomplete Uploads - Clean these up to avoid unnecessary costs
  • ๐Ÿ“š Noncurrent Versions - Don't forget old versions in versioned buckets

๐ŸŽ‰ Conclusion

Implementing S3 lifecycle policies with Terraform is one of the easiest ways to reduce your AWS bill. By automatically transitioning infrequently accessed data to cheaper storage tiers, you can save thousands annually while maintaining compliance and data retention.
Start with the examples in this article, adjust the transition periods to match your access patterns, and watch your storage costs drop! ๐Ÿ’ฐ

๐Ÿ“š Resources


๐Ÿ’ฌ Have you implemented S3 lifecycle policies? Share your strategies in the comments!

Series: Save on AWS with Terraform (21 articles)

  1. 2
    Save Money on AWS S3: Automate Storage Tiering with Terraform ๐Ÿ’ฐ This article
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article