Migrating Microsoft OneDrive & Local Mac Device Data to Amazon S3

after 20 years of monthly $10 charges, I am now back on Microsoft's personal free-to-use service tier

Series: Information Technology: Managed Service Provider Flows (2 articles)

1
My Library of People & Pets I Care About
2
Migrating Microsoft OneDrive & Local Mac Device Data to Amazon S3 This article

Bryan Chasko

Published May 15, 2026

Last modified Aug 28, 2026

Microsoft personal accounts (hotmail, outlook.com, live.com) come with 1TB of OneDrive storage bundled into a $10/month Microsoft 365 Personal subscription. This is generous. and useful. & includes access to microsoft office suite & microsoft copilot. But, life happens & for whatever reason you need to end your microsoft monthly / yearly payment & wading through the files would take forever.

if you stop paying, Microsoft has a grace of a 30-day deletion countdown
your storage configuration is click-ops in a web portal, difficult to combine with other sources like usb drives, icloud, google photos, etc

This example took 402GB in OneDrive and 16.6GB of email in Outlook. There was no local storage with enough space to handle 402GB, & we wanted something we could apply an intelligent organization pipeline to to help us wade through & edit our files / emails. AWS S3 offers an affordable & accessible storage layer where I control the access patterns, lifecycle policies, encryption, and can wire AI tooling directly to the storage layer. So we needed to establish a way to stream data from a Microsoft personal account into an AWS S3 account.

what went wrong

open-claw style methods of working with the personal data ran into headaches:

method	date killed	what happened
IMAP basic auth	September 2024	username+password login returns `LOGIN failed` for all personal accounts
app passwords	late 2024	option removed from security settings without announcement
Graph API (unverified apps)	2024 tightening	`unauthorized_client: not enabled for consumers` unless publisher-verified
EWS (Exchange Web Services)	never supported	personal accounts were never on EWS — organizational only
SMTP basic auth	March 2026	final holdout killed for Exchange Online

what we chose: rclone with a verified client_id

although it felt akin to brute force, rclone succeeded where everything else failed for us because its developers went through Microsoft's publisher verification process. the rclone project ships a pre-registered, Microsoft-verified OAuth2 client_id embedded in its source code.

when you run rclone config and leave the client_id blank, you inherit this verified credential. Microsoft's consumer endpoint accepts it because rclone is a known, verified publisher.

the flow:

one-time browser OAuth consent (you sign in, grant rclone access)
rclone stores a refresh token locally
all subsequent operations are fully headless — the refresh token auto-renews indefinitely

estimated time for our use case for the transfer was 3 days, but this was the only path we found for personal OneDrive accounts as of May 2026. custom app registrations fail without going through Microsoft Partner Network verification.

architecture

key design decisions:

0 local disk usage — rclone streams chunks through RAM directly to S3
8 parallel transfer streams balance throughput (against Microsoft's per-connection throttling)
3 separate buckets with distinct lifecycle policies per workload type
0 destroy permissions: CloudFormation stack deletion not allowed on S3
ECS Fargate arm64 (Graviton) — no local machine dependency, runs headless for days

why this shape

Fargate over EC2: the task is ephemeral. There's no AMI to maintain, no OS patching, no idle instance cost. The IAM task role gives narrower scope than an EC2 instance profile — the container gets exactly the permissions it needs for the duration it runs, nothing persistent on a long-lived host.

arm64 (Graviton) over x86_64: Graviton is ~20% cheaper for the same compute. rclone cross-compiles cleanly; there's no compatibility cost for switching architectures.

Public subnet + IGW over private subnet + NAT gateway: a NAT gateway costs ~$32/month. This workload ran for ~12 hours total. Paying $32/month for a feature we'd use for half a day is the wrong call. The same instinct applies to the secure-access bastion — t4g.nano + EIP at ~$3/month instead of a NAT.

S3 gateway endpoint: free, in-region, routes S3 traffic off the public internet entirely. It doesn't count toward NAT egress even if you had a NAT. No reason not to use it.

Secrets Manager over SSM Parameter Store for rclone.conf: SecretString handles binary blobs cleanly. KMS encryption is automatic, not opt-in. The audit trail is cleaner. SSM Parameter Store would have worked, but the OAuth refresh token is sensitive enough to warrant a dedicated service.

implementation

aws infrastructure

CloudFormation template creates three S3 buckets with lifecycle rules, encryption, versioning, and owner-only access policies:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
AWSTemplateFormatVersion: "2010-09-09"
Description: >
  microsoft migration buckets — three private s3 buckets for email,
  onedrive, and photo archive. sse-s3, versioning, block-all-public-access,
  owner-only bucket policy, lifecycle rules per workload.

Parameters:
  Owner:
    Type: String
    Default: bryanchasko
  Project:
    Type: String
    Default: microsoft-migration

Resources:
  OneDriveArchiveBucket:
    Type: AWS::S3::Bucket
    DeletionPolicy: Retain
    Properties:
      BucketName: !Sub "${Owner}-onedrive-archive"
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: AES256
      VersioningConfiguration:
        Status: Enabled
      LifecycleConfiguration:
        Rules:
          - Id: ia-after-90d
            Status: Enabled
            Transitions:
              - TransitionInDays: 90
                StorageClass: STANDARD_IA
          - Id: glacier-after-1y
            Status: Enabled
            Transitions:
              - TransitionInDays: 365
                StorageClass: GLACIER
          - Id: noncurrent-expire-90d
            Status: Enabled
            NoncurrentVersionExpiration:
              NoncurrentDays: 90
          - Id: abort-incomplete-multipart-7d
            Status: Enabled
            AbortIncompleteMultipartUpload:
              DaysAfterInitiation: 7

deploy:

1
2
3
4
5
aws cloudformation deploy \
  --stack-name bryanchasko-migration-buckets \
  --template-file migration-buckets.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides Owner=bryanchasko

rclone installation & setup

1
2
3
4
5
6
7
8
9
# install (linux)
curl -sL https://rclone.org/install.sh | sudo bash

# configure onedrive remote (interactive — needs browser once)
rclone config create ms-onedrive onedrive drive_type=personal

# configure s3 remote (non-interactive)
rclone config create aws-archive s3 \
  provider=AWS env_auth=true region=us-west-2 no_check_bucket=true

the OneDrive config opens a browser for OAuth (account permissions) consent, enabling sign in with a Microsoft account to grant access. rclone stores a refresh token, & is done with the browser.

headless servers without a browser, i.e. for agents running playwright: run rclone authorize "onedrive" — then a human is helpful with a machine & web browser, to paste the resulting token into rclone config. SSH-tunnel port 53682 from a server to local machine is an alternative access method.

running the transfer on ECS Fargate (what actually worked)

our first attempt ran from a Mac laptop. it stalled at 7% after the machine slept. the successful approach: ECS Fargate arm64 container running rclone headless, with the OAuth token stored in AWS Secrets Manager.

the rclone command inside the container:

1
2
3
4
5
6
7
8
9
10
11
rclone copy ms-onedrive:/ aws-archive:bryanchasko-onedrive-archive/ \
  --transfers 8 \
  --checkers 16 \
  --s3-upload-concurrency 4 \
  --s3-chunk-size 64M \
  --retries 5 \
  --low-level-retries 10 \
  --exclude "Personal Vault/**" \
  --progress \
  --log-file /tmp/onedrive-migration.log \
  --log-level INFO

--transfers 8 enables parallel file streams — balancing throughput vs Microsoft throttling. --checkers 16 parallel hash-check workers for skip-if-exists logic. --s3-upload-concurrency 4 multipart upload parallelism per file on the S3 side. --s3-chunk-size 64M keeps large files flowing with resiliency. --retries 5 Microsoft occasionally 429s. --low-level-retries 10 retry individual HTTP requests within a transfer. --exclude "Personal Vault/**" encrypted vaults cannot be read programmatically.

this command is idempotent; a task restart, network drop — rerun the same command & rclone skips already-transferred files automatically.

why we chose Lambda for bootstrap and completion, not ECS

The bootstrap step is a one-shot, ~5-second job that seeds Secrets Manager with the rclone.conf before the Fargate task starts. Lambda is the right shape: no container image to build or maintain, no idle cost, invoked once per migration run.

The completion handler fires on an EventBridge rule when the Fargate task reaches STOPPED state. That's a native Lambda pattern — event-driven, no polling loop, nothing running between invocations.

Both functions run arm64 Graviton: same ~20% cost reduction, plus Python 3.12 cold-start characteristics are measurably better on arm64 than x86_64. Both are well within the Lambda free tier given total invocation count across the entire migration.

what we learned the hard way

issue	symptom	fix
2 GB Fargate memory ceiling	task exits silently at ~10 min	bump to 4 GB memory, 2 vCPU
Alpine + AWS CLI v2 glibc incompatibility	`dladdr1: symbol not found` on libpython	use `apk add aws-cli` instead of AWS's glibc installer
IAM policy pointed at wrong bucket names	403 AccessDenied on every S3 op	fix resource ARNs to match actual CFN-managed bucket names
Shell quoting of `--exclude "Personal Vault/**"` in ECS task def	exclusion silently dropped	pass as `RCLONE_EXCLUDE` env var (rclone reads `RCLONE_*` natively)
BUCKET_NAME baked into Terraform pointed at fabricated bucket name	task created a rogue bucket, deposited 142 objects there	bypass Terraform with direct `register-task-definition`; reconcile source as tech debt (#19)
Subagent execution environments don't see operator's AWS profile config	Stratia ghost couldn't run `aws s3 head-bucket --profile aerospaceug-admin`	rocm-aibox handles direct AWS CLI calls; ghosts handle code authoring + GitHub ops
Mac SSO temp credentials may expire mid-sync (8h max session)	long-running rclone hits `InvalidSessionToken`	rclone is idempotent; re-login + re-run is the recovery. Permanent fix: IAM Roles Anywhere with x509 cert if continuous sync becomes a requirement

results

1
2
3
4
Transferred:   371.836 GiB / 371.836 GiB, 100%, 15.381 MiB/s, ETA 0s
Transferred:        65447 / 65447, 100%
Elapsed time:  10h34m13.7s
Errors:                 0

verify

1
2
3
4
5
6
7
8
# bucket size
aws s3 ls s3://bryanchasko-onedrive-archive --recursive --summarize --human-readable \
  --profile aerospaceug-admin --region us-west-2 | tail -2
# Total Objects: 66307
#    Total Size: 401.9 GiB

# browse in console
# https://us-west-2.console.aws.amazon.com/s3/buckets/bryanchasko-onedrive-archive?region=us-west-2

demo: AI tagging your migrated photos

once your files are in S3, services like Amazon Rekognition can tag them automatically. here are two examples pulled directly from the migrated archive:

cats on the couch

Rekognition labels: Cat, Pet, Kitten, Couch, Furniture (confidence >93%)

S3 path: Pictures/Camera imports/2022_02_21/cats/IMG_0153.JPG

re:Invent 2017 — resort pool

Rekognition labels: Water, Lighting, Pool, Swimming Pool, Hot Tub, Tub (confidence >55%)

S3 path: Pictures/all media/Camera Roll/IMG_20171130_092045.jpg

re:Invent 2017 — vegas strip at night

Rekognition labels: City, Metropolis, Urban, Road, Cityscape, Intersection (confidence >55%)

S3 path: Pictures/all media/Camera Roll/IMG_20171129_233824.jpg

re:Invent 2017 — convention venue

Rekognition labels: Cinema, Lighting, Electronics, Screen, Person, Building (confidence >55%)

S3 path: Pictures/all media/Camera Roll/IMG_20171129_151614.jpg

these images were found by scanning 500 photos from re:Invent date windows using Amazon Rekognition's detect_labels API — no pre-training, no model hosting. the same pattern scales to all 66,000+ files in the archive for search, organization, or album generation.

note: two earlier images (vegas-strip-night.jpg and vegas-venetian-canal.jpg) were misidentified during manual review — they were actually Madison, WI and San Antonio, TX respectively. this batch scan corrected the record.

how this work runs: three bridge layers, named separately

The word "bastion" gets overloaded in cloud architecture. The system around this project has three distinct bridge layers that do different jobs, and naming them separately makes the rest of the architecture make sense.

The public ingress bridge is a small EC2 (secure-access-bastion, t4g.nano, ~$3/month) that holds the public IP for ci.bryanchasko.com. It runs WireGuard over UDP 51820 and TCP 80 for CI webhooks. rocm-aibox initiates the WireGuard tunnel outbound — no inbound port from the public internet to the LAN. This bastion is adjacent to the migration project, not part of it: it's what lets external services like GitHub webhooks reach rocm-aibox.

The data plane bridge is the Fargate task itself. It runs in a VPC with a public subnet + IGW for outbound HTTPS to graph.microsoft.com, plus an S3 gateway endpoint for in-region writes to S3. Inside the container, two credentials converge: an IAM task role gives AWS-side access (S3, Secrets Manager), and an OAuth refresh token from Secrets Manager gives OneDrive-side access. That container, while it's running, IS the bridge between AWS and OneDrive. It exists only while the task is alive. No NAT gateway (saves ~$32/month), no long-lived bridge host, narrowly-scoped IAM, single-purpose.

The control plane is rocm-aibox — the linux workstation that issues the AWS commands (register-task-definition, run-task, IAM updates, CloudFormation change sets). aws sso login opens a browser, exchanges a short-lived token, and the session expires on schedule. Nothing long-lived on disk. Single host means single audit trail. AI agents (poltergeists) SSH in and inherit the same SSO session — no credential distribution.

The control plane sees no file data. The data plane sees no AWS commands beyond its own task role. Separately scoped, separately credentialed.

For a one-operator migration, this shape is right-sized. You'd outgrow it once you have multiple operators (need EC2 + SSM Session Manager), continuous sync requirements (EventBridge schedules), or a compliance audit (centralized logging). The Mac backup applies the same control/data split — the Mac authenticates itself via SSO from its own browser, rocm-aibox SSHes in and drives rclone remotely; SSH bridges the shell, not credentials. Full architectural detail in docs/operations-host.md.

who pays for what

Public ingress bridge: ~$3/month (t4g.nano + EIP). Owned by separate stack.
Control plane: $0/month (a workstation already running 24/7 for other purposes).
Data plane: $0/month idle (Fargate billed per-task-second only when running). Total Fargate spend across the whole migration: ~$0.02.

security model

No long-lived credentials anywhere. The operator authenticates via AWS SSO — temp tokens, 8h max session. The Fargate task authenticates via IAM task role — scoped, ephemeral, attached to the container. The OneDrive OAuth refresh token lives in Secrets Manager and is rotated automatically by rclone, written back via a SIGTERM trap on container shutdown.

Least-privilege IAM. The task role is scoped to specific bucket ARNs and one Secrets Manager secret ARN. No wildcard S3. No wildcard secretsmanager. If the container is compromised, the blast radius is bounded to those two resources.

Personal Vault exclusion is cosmetic. Personal Vault is encrypted by Microsoft and not programmatically readable. The --exclude "Personal Vault/**" flag in rclone is belt-and-suspenders; the API would have rejected the listing regardless. Defense in depth.

No SSH on the public ingress bastion. Operator access to rocm-aibox is via SSM Session Manager — no open SSH port, no key management. Ingress traffic on the bastion is HTTPS:80 (CI webhooks) and WireGuard:51820/udp. That's the full surface.

S3 buckets are locked down by default.

control	setting
block-public-access	all four flags enabled
bucket policy	owner-only (no cross-account)
server-side encryption	SSE-S3 (AES256)
versioning	enabled
stack deletion protection	`DeletionPolicy: Retain` on every bucket

Stack tear-down cannot destroy data. That's a deliberate constraint.

wiping OneDrive after the migration

once you've verified the S3 transfer, you can delete the OneDrive contents. rclone delete runs against the remote the same way rclone copy does:

1
rclone delete ms-onedrive:/ --exclude "Personal Vault/**" --progress

The wipe ran on the same Fargate task family as the migration (revision 10, command swapped from rclone copy to rclone delete). Same image, same IAM role, same Secrets Manager OAuth token. 66,307 files / 401.881 GiB removed in 1h48m, exit code 1 from the expected Personal Vault listing rejection (encrypted, not API-readable — the rclone exclude is cosmetic; the API would reject regardless). One transient upload-chunk timeout retried successfully. S3 archive preserved unchanged at 66,307 objects / 401.9 GiB.

idempotency matters here. rclone copy and rclone delete are both idempotent. The migration's idempotency is what let us restart the Fargate task multiple times during the debugging saga — BUCKET_NAME bug, glibc/musl incompatibility, IAM scoping error, OOM at 2GB — without losing progress. Each restart resumed from where the last one left off. The same property covers the wipe: a mid-deletion crash is recoverable by rerunning the same command. rclone skips files already gone and continues from where it stopped.

This is a non-negotiable property for any multi-day data movement workload at home-internet scale.

what this costs (real numbers, not estimates)

These figures come from AWS Cost Explorer actuals and direct API queries against account 211125425201. Full breakdown in

one-time costs (migration project total: $0.64)

Item	Cost
Fargate ECS (arm64)	$0.019
Rekognition (500 calls)	$0.500
CloudWatch Logs ingest	$0.117
Lambda, ECR, SNS	$0.000
Total	$0.64

year 1 monthly ($8.43 avg)

Item	$/month
S3 Standard → Standard-IA (weighted)	$8.03
Secrets Manager	$0.40
Total	$8.43

S3 lifecycle: Standard months 1–3, Standard-IA months 4–12. Projected total storage: 531.5 GB (OneDrive archive + Mac backup).

year 2+ monthly ($2.53 avg, all in Glacier)

Item	$/month
S3 Glacier Flexible Retrieval	$2.13
Secrets Manager	$0.40
Total	$2.53

Microsoft 365 Personal — the subscription this replaces — was $9.99/month. Break-even on the migration is essentially immediate; savings compound after the Glacier transition.

See for the full service-by-service breakdown including Fargate task run detail and retrieval fee notes.

AI-ready archive

with former microsoft data migrated into S3, AI tooling can directly run embedding models against word docs, store vectors in a local database instance for semantic search (i.e. qdrant), ask natural language questions across years of email and files, use Amazon Bedrock or local LLMs to summarize, categorize, and tag. photo organization pipelines with vision models are also now possible.

Amazon S3 is easily accessible to any tool you point at your bucket, easily recognizable with unique bucket name & Amazon Resource Number.

sister project: the macbook backup

The same architectural pattern applies to a different leg. Mac filesystem → S3, using rclone running on the Mac directly — not in Fargate — because Fargate can't reach files sitting behind a home NAT.

The destination bucket pattern is identical: lifecycle, SSE-S3 encryption, owner-only policy, versioning. The control plane is the same: rocm-aibox SSHes into the Mac and drives the sync remotely. The auth principle is the same: each host authenticates independently to AWS via SSO, from its own browser. SSH bridges the shell, not credentials.

The Mac never forwards an AWS credential to rocm-aibox, and rocm-aibox never touches the Mac's filesystem directly. The separation is structural.

post-migration: revoke the token

after the transfer completes, revoke rclone's access to your OneDrive at https://account.live.com/consent/Manage. this invalidates the refresh token immediately. any copies of the token (chat logs, secrets manager, local rclone.conf) become worthless. do this before canceling your Microsoft 365 subscription.

as of May 2026, CloudFormation template, Terraform, Docker, Lambda, & rclone scripts are at

# amazon-s3 # ai-assisted-workflows # migration # secure-file-transfer

Series: Information Technology: Managed Service Provider Flows (2 articles)

1
My Library of People & Pets I Care About
2
Migrating Microsoft OneDrive & Local Mac Device Data to Amazon S3 This article

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

Migrating Microsoft OneDrive & Local Mac Device Data to Amazon S3

Series: Information Technology: Managed Service Provider Flows (2 articles)

Bryan Chasko

what went wrong

what we chose: rclone with a verified client_id

architecture

implementation

aws infrastructure

rclone installation & setup

running the transfer on ECS Fargate (what actually worked)

why we chose Lambda for bootstrap and completion, not ECS

what we learned the hard way

results

verify

demo: AI tagging your migrated photos

cats on the couch

re:Invent 2017 — resort pool

re:Invent 2017 — vegas strip at night

re:Invent 2017 — convention venue

how this work runs: three bridge layers, named separately

security model

wiping OneDrive after the migration

what this costs (real numbers, not estimates)

AI-ready archive

sister project: the macbook backup

Series: Information Technology: Managed Service Provider Flows (2 articles)