AWS Builder Center

Optimizing S3 Uploads: Reducing Server Overhead by 85% with Pre-Signed URLs

Learn how to bypass server-side memory bottlenecks for file uploads to AWS S3. We discuss architecture, security, and how pre-signed PUT URLs save server CPU and memory.

Introduction

When building web applications that allow users to upload files (like photos, PDFs, or documents), the traditional approach is to stream the file from the user's browser to the backend server (e.g., a Flask or Node.js API), and then have the server upload it to AWS S3.
While this works for small files, it quickly becomes a bottleneck. Your server's CPU and RAM get consumed by handling file streams, leading to high latency and scaling costs.
In this article, we’ll explore how to bypass the backend entirely and upload files directly from the client browser to Amazon S3 securely using Pre-signed PUT URLs.



The Architecture: Traditional vs. Pre-Signed

Traditional Upload Flow (High Server Cost):

  1. Client Browser ➔ Streams File ➔ Backend Server (Consumes CPU & RAM)
  1. Backend Server ➔ Uploads File ➔ Amazon S3

Pre-Signed Upload Flow (Zero Server Cost):

  1. Request URL: Client requests an upload slot from the Backend Server.
  1. Generate URL: Backend Server calls AWS S3 API, requests a short-lived pre-signed URL, and returns it to the client.
  1. Direct Upload: Client Browser uploads the file directly to the S3 bucket using the pre-signed URL.



Implementation: Generating the URL with Boto3 (Python)

To generate the pre-signed URL securely on the backend, we use the AWS SDK for Python ( boto3 ). It is critical that the EC2 instance hosting the backend uses an IAM Instance Profile with write permissions to the S3 bucket rather than hardcoding AWS Access Keys.
Here is the Python implementation:
import boto3
from botocore.exceptions import ClientError

def generate_s3_presigned_put_url(bucket_name, object_name, expiration=600):
"""
Generates a pre-signed PUT URL to upload a file directly to S3.
:param bucket_name: Target S3 bucket name
:param object_name: Desired object key inside S3
:param expiration: URL validity in seconds (default is 10 minutes)
"""
# Initialize the S3 client (inherits permissions from IAM role)
s3_client = boto3.client('s3', region_name='us-east-1')
try:
response = s3_client.generate_presigned_url(
ClientMethod='put_object',
Params={
'Bucket': bucket_name,
'Key': object_name,
'ContentType': 'application/octet-stream' # Restricts format
},
ExpiresIn=expiration
)
except ClientError as e:
print(f"Error generating pre-signed URL: {e}")
return None
return response
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article