AWS Builder Center
Efficient Asset Archiving Connecting AEM with AWS S3 Glacier

Efficient Asset Archiving Connecting AEM with AWS S3 Glacier

As DAM repositories grow into terabytes, AEM performance and storage costs suffer. This guide integrates AEM with S3 Glacier using AWS SDK for Java v2: an OSGi service, a workflow that streams originals to Glacier IR, and a restore path back into the DAM.

https://www.linkedin.com/in/franciscomurillo03b7uita85b0/
In modern Digital Asset Management (DAM), storage costs and system performance are critical factors. As repositories grow into terabytes, keeping every high-resolution asset in primary storage becomes expensive and can impact AEM performance. A powerful solution is to integrate Adobe Experience Manager (AEM) with AWS S3 Glacier for intelligent asset archiving.

The Library: AWS SDK for Java v2

The core of this integration is the AWS SDK for Java v2. Unlike its predecessor, v2 is built with a non-blocking architecture and better modularity. For AEM integrations, we specifically utilize the following modules:
S3 Client (s3): The primary client for interacting with Amazon S3. It handles the API calls for object management.
URL Connection HTTP Client (url-connection-client): A lightweight HTTP client. In AEM’s OSGi environment, this is often preferred over the default Netty-based client because it avoids complex classloading issues and dependency conflicts within the container.
Authentication (auth): Handles the signing of requests using the provided credentials.

Maven Dependency Configuration

To implement this, the following dependencies are added to the AEM core module:
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3</artifactId>
<version>2.x.x</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>url-connection-client</artifactId>
<version>2.x.x</version>
</dependency>

1. OSGi Configuration & Lifecycle

The connection is managed as an OSGi service. By using @Designate and @Activate, the service automatically initializes the S3 client when the configuration is updated in the AEM Web Console.
@ObjectClassDefinition: Defines the UI fields for the configuration (Access Key, Secret Key, etc.).
getS3Client(): A helper method that builds the client using a StaticCredentialsProvider and the UrlConnectionHttpClient. This ensures that every request is properly signed and transmitted through a compatible HTTP layer.
Sample OSGi Configuration
@ObjectClassDefinition(name = "AWS S3 Service Configuration")
public @interface AWSS3ServiceConfig {
@AttributeDefinition(name = "AWS Access Key")
String awsAccessKey();
@AttributeDefinition(name = "AWS Secret Key")
String awsSecretKey();
@AttributeDefinition(name = "AWS Region")
String awsRegion();
@AttributeDefinition(name = "AWS Bucket Name")
String awsBucket();
}
S3 Client Initialization
protected S3Client getS3Client(String accessKey, String secretKey, String region) {
SdkHttpClient httpClient = UrlConnectionHttpClient.create();
return S3Client.builder()
.region(Region.of(region))
.credentialsProvider(
StaticCredentialsProvider.create(
AwsBasicCredentials.create(accessKey, secretKey)
)
)
.httpClient(httpClient)
.build();
}

2. The Archiving Workflow (Upload Process)

Archiving is typically implemented as a Workflow Process (WorkflowProcess interface).
S3 Streaming: The original rendition of the asset is retrieved as an InputStream and streamed directly to S3. Using RequestBody.fromInputStream allows us to handle large binaries efficiently without loading them into memory.
Storage Class Selection: The PutObjectRequest specifically sets the storage class to GLACIER_IR (Instant Retrieval). This ensures the asset is stored at a lower cost but remains accessible for immediate download if needed.
Metadata Keying: Once the upload is successful, the S3 object key is saved back to the asset’s metadata in AEM (e.g., as a property like s3key). This creates a permanent link between the local AEM record and the remote S3 binary.
Press enter or click to view image in full size
Asset Upload Logic
private String uploadOriginalToS3(Asset asset, S3Client s3Client, String bucket) {
Rendition original = asset.getOriginal();
InputStream inputStream = original.getStream();
long contentLength = original.getSize();
String s3Key = "archive/" + asset.getName(); // Generalized key logic

PutObjectRequest putObjectRequest = PutObjectRequest.builder()
.bucket(bucket)
.key(s3Key)
.contentType(original.getMimeType())
.contentLength(contentLength)
.storageClass(StorageClass.GLACIER_IR)
.build();

s3Client.putObject(putObjectRequest, RequestBody.fromInputStream(inputStream, contentLength));
return s3Key;
}

3. JCR Metadata Preservation

A critical part of the process is preserving the asset’s intelligence in AEM while the binary is archived.
Recursive Metadata Copying: A custom utility copies all JCR properties (tags, titles, descriptions) from the original asset to a representative “proxy” asset. This ensures that the DAM remains fully searchable even after the original high-resolution file is removed.
Lifecycle Management: After the metadata is backed up and the binary is safe in S3, the original asset is deleted from the JCR to reclaim disk space.
Press enter or click to view image in full size
Recursive Metadata Copy Utility
private static void copyMetadata(Node src, Node destination) throws RepositoryException {
PropertyIterator it = src.getProperties();
while (it.hasNext()) {
Property property = it.nextProperty();
String name = property.getName();
if (name.startsWith("jcr:")) continue; // Skip system properties
if (property.isMultiple()) {
destination.setProperty(name, property.getValues());
} else {
destination.setProperty(name, property.getValue());
}
}
// Logic continues recursively for child nodes...
}

4. The Restoration Mechanism

The restore process is the inverse operation, but with added logic for status monitoring:
Status Checking: For deeper archive classes, the service uses headObject to check if a restore request is complete.
Re-ingestion: Once the object is available, it is downloaded and re-inserted into the DAM using the `AssetManager` API.
User Notification: Since restoration can take time (depending on the S3 storage class), the MessageGatewayService is used to send an HTML email to the user who requested the asset, providing them with a direct link to the restored file.
Press enter or click to view image in full size
OSGi lifecycle diagram
Press enter or click to view image in full size
IT shows how @ObjectClassDefinition exposes config fields in the Web Console, how @Activate/@Modified rebuilds the S3Client on any config change (credential rotation with zero redeploy), and how consumer workflows reference the singleton service.

Key Benefits of this Architecture

- Modularity: The use of the AWS SDK v2 modules keeps the AEM bundle size optimized.
- Reliability: By leveraging OSGi configurations, credentials can be rotated without code deployments.
- Operational Efficiency: Automated workflows handle the heavy lifting of moving terabytes of data, while AEM users continue to work with a fast, lightweight repository.
By combining these specialized parts, the integration provides a professional-grade solution for long-term asset management.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article