AWS Builder Center

Builder Center

Streaming Files From a URL Directly to Amazon S3 in an S3Client

Streaming Files From a URL Directly to Amazon S3 in an S3Client

How to build a URL-to-S3 streaming pipeline in Electron using Node.js streams and `@aws-sdk/lib-storage`, with progress reporting, cancellation, redirect handling, and region recovery, without writing files to local disk.

Vidit Shah

Importing a remote file into Amazon S3 looks simple from the UI: paste a URL, choose a key, and start the transfer.
The implementation is more interesting. A naive approach would download the file to a temporary path, then upload that local file to S3. That works for small files, but it creates avoidable disk usage, cleanup logic, and a second transfer phase that has to report progress separately.
The cleaner approach is to treat the remote response body as the upload body. In other words:
  1. Open an HTTP or HTTPS request to the source URL.
  2. Pipe the response through a small counting stream.
  3. Hand that stream directly to S3 multipart upload.
  4. Report progress through the same upload events used by local file uploads.
This article walks through that pattern in an Electron app. The examples are shortened from a working implementation, but they keep the important control flow intact.

Why This Use Case Matters

URL-to-S3 transfer is a small feature on the surface, but it removes a common bottleneck in cloud storage workflows.
Many files that end up in S3 do not start on the user's machine. They start as vendor exports, signed download links, public datasets, CDN-hosted artifacts, build outputs, media files, logs, backups, or one-off files shared over HTTPS. Requiring the user to download those files locally and then upload them again adds extra time, extra disk usage, and another failure point.
Direct streaming solves that by turning the desktop app into a controlled transfer coordinator instead of a local staging area. The user still decides the destination bucket, key, headers, and cancellation behavior, but the bytes do not need to rest on disk.
That matters most when:
  1. The file is large enough that local staging is slow or inconvenient.
  2. The user's machine has limited disk space.
  3. The source URL is temporary, signed, or authenticated.
  4. The same upload UI should handle both local files and remote URLs.
  5. The workflow needs auditability and explicit user control instead of a headless server-side import.
In practice, this pattern is useful anywhere a user needs to move data from "a URL I can access" to "an object key in S3" with progress, cancellation, and predictable error handling.

What the Feature Needs to Handle

The user flow is intentionally small:
  1. Paste a remote URL.
  2. Optionally provide request headers for protected sources.
  3. Choose the target object key.
  4. Start the transfer.
  5. See progress and completion in the normal upload UI.
The engineering requirements are where most of the work is:
  1. No local disk staging.
  2. Support large files through multipart upload.
  3. Validate URL schemes in the renderer and the main process.
  4. Follow source redirects without leaking authorization headers across origins.
  5. Allow cancellation while connecting, streaming, or uploading.
  6. Recover when S3 reports that the destination bucket is in a different region.
  7. Keep the transfer service small enough to extract as open-source reference code.

Architecture

The implementation is split into four pieces:
  1. A renderer form that handles user input and immediate validation.
  2. A narrow preload API that exposes startUrlUpload.
  3. A main-process IPC handler that validates again and starts the transfer.
  4. A streaming service that owns HTTP fetch, progress counting, S3 upload, cancellation, and retry behavior.
That boundary is useful because only one part of the system needs to know about streams and multipart upload. The renderer stays focused on input. The main process stays focused on application state. The service stays focused on the transfer lifecycle.

Renderer Validation

The renderer should catch obvious input problems early, but it should not be the only validation layer. In this implementation, the form validates that the URL is http or https and that the filename is a single object-name segment.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
const urlUploadSchema = z.object({
url: z
.string()
.trim()
.min(1, { message: "URL is required." })
.refine(
(value) => {
try {
const parsed = new URL(value);
return parsed.protocol === "http:" || parsed.protocol === "https:";
} catch {
return false;
}
},
{ message: "Enter a valid http or https URL." }
),
fileName: z
.string()
.trim()
.min(1, { message: "Filename is required." })
.refine((name) => !name.includes("/"), {
message: "Filename cannot contain '/' characters.",
}),
headersText: z.string().optional(),
});
The form also derives a default filename from the URL path while the user has not edited the filename field:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
export function deriveFileNameFromUrl(rawUrl: string): string {
if (!rawUrl) return "";
try {
const parsed = new URL(rawUrl.trim());
const segments = parsed.pathname.split("/").filter(Boolean);
const last = segments[segments.length - 1];
if (!last) return "";
try {
return decodeURIComponent(last);
} catch {
return last;
}
} catch {
return "";
}
}
This is purely a usability feature. The main process still validates the payload before any transfer starts.

A Narrow IPC Contract

The renderer sends one compact payload:
1
2
3
4
5
6
{
bucketName: string;
s3Key: string;
url: string;
headers?: Record<string, string>;
}
The renderer is responsible for joining the current prefix and filename into the final s3Key. That keeps the IPC contract explicit and makes the streaming service easier to reuse in another app.
The main process validates the required fields, parses the URL again, restricts the scheme, rejects duplicate transfers for the same destination key, and then starts the transfer asynchronously:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
if (!bucketName?.trim() || !s3Key?.trim() || !url?.trim()) {
return { success: false, error: "bucketName, s3Key, and url are required." };
}

let parsed: URL;
try {
parsed = new URL(url);
} catch {
return { success: false, error: "Invalid URL." };
}

if (parsed.protocol !== "http:" && parsed.protocol !== "https:") {
return { success: false, error: "Only http and https URLs are supported." };
}

const uploadKey = getUploadKey(bucketName, s3Key);
if (hasActiveUrlUpload(uploadKey)) {
return {
success: false,
error: `An upload is already in progress for ${s3Key}.`,
};
}

void startUrlUpload(
{ bucketName, s3Key, url, headers },
{
mainWindow,
s3Client,
profileName,
region,
sessionId,
uploadKey,
isCanceled: () => permanentlyCanceled.has(uploadKey),
getS3Client: (region) => clientManager.getClient(region),
}
);

return { success: true, sessionId: String(sessionId) };
The sessionId returns immediately. Progress and completion arrive later through events, which keeps the UI responsive.

Fetching the Source URL

The fetch layer uses Node's http and https modules instead of fetch because the implementation needs direct control over request lifecycle, redirects, response streams, and abort behavior.
It only accepts http: and https: URLs:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
export async function fetchUrlStream(
rawUrl: string,
headers: Record<string, string> | undefined,
abortSignal: AbortSignal,
redirectsRemaining = MAX_REDIRECTS,
originUrl?: NodeURL
): Promise<FetchedStream> {
const url = new NodeURL(rawUrl);
if (url.protocol !== "http:" && url.protocol !== "https:") {
throw new Error(`Unsupported URL scheme: ${url.protocol}`);
}

const transport = url.protocol === "https:" ? https : http;
const effectiveHeaders: Record<string, string> = { ...(headers ?? {}) };

if (originUrl && !isSameOrigin(originUrl, url)) {
for (const k of Object.keys(effectiveHeaders)) {
if (k.toLowerCase() === "authorization") {
delete effectiveHeaders[k];
}
}
}

// request setup omitted for brevity
}
Stripping Authorization on cross-origin redirects is important. If the user provides a bearer token for files.example.com, a redirect to another host should not receive that token.
Redirects are followed with a fixed hop limit:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
if (status >= 300 && status < 400 && res.headers.location) {
res.resume();

if (redirectsRemaining <= 0) {
reject(new Error(`Too many redirects (>${MAX_REDIRECTS})`));
return;
}

const nextUrl = new NodeURL(res.headers.location, url).toString();
fetchUrlStream(
nextUrl,
effectiveHeaders,
abortSignal,
redirectsRemaining - 1,
url
).then(resolve, reject);
return;
}
When the response succeeds, the service keeps the response stream and metadata that S3 can use later:
1
2
3
4
5
resolve({
body: res,
contentLength,
contentType,
});

Streaming Into S3

The upload service registers the transfer as active before it awaits the HTTP response. That small ordering detail is what makes cancellation work while the source server is still connecting.
1
2
3
4
5
6
7
8
9
const abortController = new AbortController();

const entry: ActiveUrlUpload = {
abortController,
upload: null,
sourceBody: null,
};

activeUrlUploads.set(uploadKey, entry);
After the source stream is available, the service derives metadata, inserts a counting transform, and passes the transform into @aws-sdk/lib-storage:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
const fetched = await fetchUrlStream(url, headers, abortController.signal);
entry.sourceBody = fetched.body;

const totalBytes = fetched.contentLength;
const contentType =
fetched.contentType ||
(mimeLookup(s3Key) as string | false) ||
"application/octet-stream";

counter = makeCounterStream(
uploadKey,
s3Key,
totalBytes,
baseCtx,
mainWindow,
(n) => {
transferredBytes = n;
}
);

fetched.body.pipe(counter);

const upload = new Upload({
client: currentS3Client,
params: {
Bucket: bucketName,
Key: s3Key,
Body: counter,
ContentType: contentType,
ContentDisposition: "inline",
},
queueSize: determineQueueSize(totalBytes),
partSize: determinePartSize(totalBytes),
leavePartsOnError: false,
});

entry.upload = upload;
const result = await upload.done();
At no point does the service create a temporary file. The remote server's response stream becomes the S3 upload body.

Progress Reporting

The AWS SDK emits upload progress, but for stream bodies it is useful to count bytes before the stream reaches the uploader. That gives the UI a consistent progress signal when the source provides Content-Length, and a useful transferred-byte signal when it does not.
The counting stream is just a Transform:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
function makeCounterStream(
uploadKey: string,
s3Key: string,
totalBytes: number | undefined,
baseCtx: Record<string, unknown>,
mainWindow: BrowserWindow | null,
onBytes: (n: number) => void
): Transform {
let transferredBytes = 0;

return new Transform({
transform(chunk: Buffer, _enc, cb) {
transferredBytes += chunk.length;
onBytes(transferredBytes);

if (shouldEmitProgress(uploadKey)) {
emit(mainWindow, "upload-progress", {
...baseCtx,
s3Key,
transferredBytes,
totalBytes: totalBytes ?? transferredBytes,
isPseudoProgress: false,
});
}

cb(null, chunk);
},
});
}
The event names intentionally match the local file upload pipeline:
  1. upload-progress
  2. upload-file-complete
  3. upload-session-complete
That means a URL transfer can appear in the existing upload list without creating a separate progress UI.

Cancellation

Cancellation needs to stop whichever parts of the transfer already exist:
  1. The outbound HTTP request.
  2. The source response stream.
  3. The S3 multipart upload.
The active upload entry stores all three handles as they become available:
1
2
3
4
5
interface ActiveUrlUpload {
abortController: AbortController;
upload: Upload | null;
sourceBody: NodeJS.ReadableStream | null;
}
The cancel function tears them down in that order:
1
2
3
4
5
6
7
8
9
10
export function cancelUrlUpload(uploadKey: string): boolean {
const active = activeUrlUploads.get(uploadKey);
if (!active) return false;

active.abortController.abort();
active.upload?.abort().catch(() => {});
active.sourceBody?.destroy?.(new Error("Upload canceled by user"));

return true;
}
The running upload still emits the terminal completion event. That keeps cancellation behavior consistent with the rest of the upload system and avoids double-emitting completion from the cancel path.

Recovering From S3 Region Redirects

The destination bucket can be in a different region than the current S3 client. In that case, S3 may respond with a PermanentRedirect error.
Instead of failing immediately, the service treats that as a recoverable condition:
  1. Detect the redirect-shaped S3 error.
  2. Extract the bucket's actual region.
  3. Destroy the partially consumed source stream.
  4. Resolve a fresh S3 client for the correct region.
  5. Emit a progress status so the UI does not look stuck.
  6. Retry once.
The retry branch looks like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
const decision = decideRedirect({
error: attemptErr,
currentRegion: effectiveRegion ?? "",
hasRedirected,
});

if (decision.kind !== "redirect" || !ctx.getS3Client) {
throw attemptErr;
}

hasRedirected = true;

entry.sourceBody?.destroy?.();
counter?.destroy();

const oldRegion = effectiveRegion ?? "";
currentS3Client = ctx.getS3Client(decision.newRegion);
effectiveRegion = decision.newRegion;
baseCtx = {
bucketName,
profileName,
region: effectiveRegion,
sessionId,
};

ctx.onRegionRedirect?.(oldRegion, decision.newRegion);

emit(mainWindow, "upload-progress", {
...baseCtx,
s3Key,
transferredBytes: 0,
totalBytes: 0,
statusMessage: `Switching to bucket's region (${decision.newRegion})...`,
isPseudoProgress: true,
});

continue;
The retry is intentionally limited to one redirect. If the second attempt also fails with a region redirect, the service surfaces the error rather than looping.

Testing the Transfer

This kind of feature needs tests at two levels.
End-to-end tests cover the user flow:
  1. The UI exposes a "From URL" action.
  2. The modal opens.
  3. The filename is derived from the URL path.
  4. Invalid URL schemes are blocked before IPC.
  5. Custom headers are forwarded.
  6. A successful call creates the expected object entry in test state.
Unit tests cover the stream semantics:
  1. URL bytes reach the Upload body.
  2. The active upload entry exists before the HTTP request resolves.
  3. Cancellation works while the source request is still pending.
  4. Non-2xx source responses fail cleanly.
  5. Content type falls back from response header to filename lookup to application/octet-stream.
  6. Region redirects swap clients and retry once.
  7. A second redirect becomes a terminal failure.
This split is useful because UI tests cannot prove stream lifecycle behavior, and stream tests cannot prove the feature is wired correctly through the renderer and IPC boundary.

Open-Source Shape

The most reusable part of this design is the streaming service. It does not need to know much about the host application. It needs:
  1. An S3 client.
  2. A destination bucket and key.
  3. A source URL and optional headers.
  4. A progress emitter.
  5. A cancellation signal.
  6. A callback for resolving a new S3 client after region redirect.
That makes it a good candidate for an open-source reference implementation:
  1. A small Electron sample app that demonstrates the whole flow.
  2. A standalone transfer service with a minimal example UI.
  3. A test suite showing cancellation, redirects, and multipart upload behavior.

Takeaway

The core idea is simple: do not model this as "download then upload" unless you actually need a local file.
For this class of feature, the remote response stream can become the S3 upload body. Once the implementation is framed that way, the rest of the design becomes a set of practical lifecycle concerns: validation, redirect safety, progress counting, cancellation, region recovery, and tests.
# amazon-s3# data
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article