My Library of People & Pets I Care About

Myspace was a bigger online presence than Google in 2006. I kept my life there in terms of photos, birthdays, blogs. Enough other folks kept their life there for the company to sell for $580 million. 5 years later, the numbers had completely collapsed. I never intentionally quit the site, but - somewhere in their desperation - the company overhauled the site enough times that my space no longer had any of my content - content I had thought would be there forever

13 years later and I'd been riding the Facebook train - I'd recently been named an AWS Hero and had a pretty steady cadence of content to support that rise. This time, I blew up my online presence myself, driven a touch mad by first having tons of criticism from teammates for things I was doing on personal time - then things falling apart and ending with a job I'd been with for 15 years - and the end of a personal relationship. I didn't know where to begin unraveling all that - so I deleted all my posts and connections - I had no space or face online

And so it was that I found myself with no photos of myself nor the beloved pets of the first decades of my life. Fortunately, a family member had an old unsupported device, 4 usb drives, 5 cloud storage accounts, and two new windows devices on different Microsoft accounts. There were photos neither of us had seen in years, both unsure what did / didn't exist through their own digital chaos

I set out to index, organize, and make accessible 28,000 private photos. Using rClone command-line program, I previously proved pipelines to remotely copy files from physical devices & consumer photo clouds including Apple's iCloud 

Files ranged between grainy JPGs of ancestors, HEIC mobile phone files, PNG scans of genealogy documents, PDF, DOCx, DLSR extensions, and to a lesser extend MOV clips. From different devices, formats, operating systems, generations of storage were copied to Amazon S3. Placing the files on S3 meant I could build a web app for accessing them. I implemented lifecycle policies so that after the initial tagging-frenzy of the first 30 days I would move from S3 Standard to Standard-Infrequent Access, and on into Glacier Deep Archive after a year. In my previous rClone project, I used these lifecycle policies to cut my cloud storage bill in half, but on this project I would have happily paid a premium for access to long-lost images, so the accessibility & eleven 9s of durability were more relevant than cost controls

On a local bare-metal Linux server, I hosted a Server Message Block to facilitate the transfers from LAN connected devices (+ USB sticks plugged in). A 117-GB PhotoStick included duplicate photo copies of several vacations, and I chose to use the SMB process to use rclone's –checksum tag to compare hashes to avoid uploading duplicates like these. Alternatively, I could have zipped each device's files and used AWS Lambda to iterate through, but by going file-by-file before uploading to cloud I saved on thousands of read/write requests and compute - and I'd still be faced with extracting the files individually once on the cloud. We'd still be running rClone to process the transfer, so it made sense to do our best to get unique & clean files as our start on S3. The way I handled this, 4000 files are four thousand handshakes between our Server Message Block and S3, with graceful syncing and restarts. Transferring everything off a desktop took 18 hours, but everything went through without errors even when a lightning storm took down the power

First, I created a Python script to use path parsing and regular-expression to infer likely names, dates & locations from file names themselves. Next, I utilized OpenAI's open source model Contrastive Language-Image Pretraining (CLIP) to associate images with correlating text. Natural language supervision allows CLIP to predict the most relevant text snipped given an image. This allowed automagic labeling of 2,000 photos of sunsets to be grouped together, images with pets to be tagged as such, and images with human faces to be tagged as people

Our write model is that every tag, label, face-match and date lives in Amazon DynamoDB with a write model designed to record decisions into DynamoDB every click creating photo keys in scant milliseconds. In order to support gallery style views of the 28k+ photos, I need to render every photo's metadata at once and constant 28,000-row scans on every page load would be brutally slow and expensive

So I follow a Command Query Responsibility Segregation pattern with a write path and read path using different services shaped for different access patterns. Our pipeline flattens the info from Dynamo into a JSON format timeline manifest back on S3, with browsers like Chrome only fetching that one object to materialize the read view. I almost never query DynamoDB to display slow-changing data. I precompute the read view, cache it as a static object on Amazon CloudFront alongside the rest of our Progressive Web Application built with Vite - and as a result the web browser itself is able to power instant and free search

An Amazon Cognito User Pool is our identity provider using JSON Web Token (JWT) open standard to securely transmit information by handing our browser a signed JWT access token, which then gets sent to the Amazon Cognito Identity Pools via GetID and GetCredentialsForIdentity API calls, exchanging the JWT on successful login for temporary AWS IAM Access keys scoped for our application. The browser can then use the IAM creds to talk to Amazon S3 & DynamoDB for downloading & writing. This keeps access control out of our code and in IAM. This further enforces our static-site preference keeping very few moving parts, no API to scale, and clear security boundaries. We're able to produce a 1 time code for authorizing new devices, after which I add a passkey for biometric / passwordless entry

AWS CodeBuild runs our web app test and build processes whenever a GitHub webhook is triggered off of changes to the main branch. Vite turns syntax code like TypeScript into regular HTML that browsers understand. By removing extra code and splitting files up, our site loads quickly. Vite, like Webpack, generates a unique content hash during the build process and injects the hash directly into the file name, i.e., main.a8f9c2d1.js. Because the file name is unique and tied to its content, it can never change and is assigned a one-year immutable cache header. Our entry-point files include an index.html shell, a service worker script, manifest of progressive web app root files, and the JSON manifest with no-cache headers so they always revalidate. On deploy, I only invalidate the shell itself and the manifest paths - because the names change new builds of JavaScript bundles are brought in as new assets and I skip over invalidating the old assets as I no longer point to them. When HTML stop referencing the old paths, I drop out the old cached versions. The service worker itself needed cleanupOutdatedCaches, skipWaiting and clientsClaim so new workers take over immediately instead of waiting for tabs to close

Because all assets are protected by Amazon Cognito, Cognito User Pool ID, an app client ID, and our Identity Pool ID are presented at build time so Vite can bake them in as public client-side identifiers without ever sharing secrets with the client. The combination of caching-policy awareness, keeping our shell fresh with no-cache, targeted invalidation of cache elsewhere, and service workers that stream new assets mean zero-downtime serving new features

When ingesting files from the machines, rClone brought GEDCOM files (.ged) into S3 that were exports of an ancestry.com family tree. This provided a spine of names, documents, dates of birth, census records for 5 generations of family to shape our curation around

After building a login form and routes to leverage Cognito access management, I built a tagging interface with the ability to select from the now-known names or add new ones. Tapping on a face adds an Amazon Rekognition defined bounding box, and other buttons allow grouping into common other subjects like memes that I didn't want showing up in the photos grid. I added some basic editing features like crop & rotation as well as file information including the device and path the photo was copied from

I arbitrarily chose to tag at least 8 photos of individuals or pets before leaning on AI further, as well as providing a training set of what sunsets look like. I store this manual training alongside the CLIP labels as a "what I already know" cache in DynamoDB. A quick-label pass with Rekognition uses DetectLabels for broad classifications such as document or animal. IndexFaces adds embeddings into the collection I can the use to enable SearchFaces queries.

The process of routing documents, memes, sunsets, wildlife and people into different partitions and interfaces is aided by using Rekognition's DetectLabels to produce object labels at the frame level to group by location - DetectLabels was able to label the location of thousands of sunsets as being the same silhouette of the Tularosa Basin of New Mexico USA- improving focus on labeling faces within the user interface

With my small manually labeled training sets, I could then use Rekognition's SearchFaces against each confirmed face, to add a paginated "possible match" grid of photos that might be the same individual. This allowed me to then manually confirm hundreds of photos within a couple of minutes. Rekognition wants faces fifty pixels across, both eyes visible with consistent lighting. Chemical grain, a turned head or a low-resolution photo return zero bounding boxes for SearchFaces to match against. Photos 10 years apart in age showed roughly 18% less ability for Rekognition to provide matching

I then returned my focus to using my desktop where I had run the CLIP scene labeling; to implement more open-weight models. InsightFace's buffalo_l leverages my AMD Radeon graphics card well, producing 512 dimensional face embeddings through ONNX runtime. Because at this point I were working with unlabeled photos Rekognition couldn't handle, I bumped our similarity cosine down to .45 (instead of .6+) and welcomed in images with faces smaller than Rekognition's 40x50 pixel floor. On the local machine, I leveraged valkey for storing the embeddings as HNSW index to allow scripts to search 28k+vectors in less than a millisecond. I then use Amazon Titan Multimodal Embeddings to write 1024 dimensional image vectors in Amazon S3 Vectors. This allows us to grow the index through facial geometry when it's there and whole-image similarity when it isn't. Of 1110 photos confirmed by CLIP to have faces, buffalo_l extracted 860 faces as dimensional ArcFace vectors. ArcFace detected no people in 23 photos where faces were too small or obscured to use a bounding box. The rest of the images were pet images and ArcFace was only trained on human geometry

This was actually the second time I'd gone through this pipeline - I ran through it once before I had discovered the Ancestry.com files to establish a familial tree of relationships. In my family, names like John & Andrew are extremely common across the generations, and when you are mapping the geometry of family members the vectors can be extremely similar. So it was that on my first training pass an AI coding assistant made the call to approve merging two andrews into one - my father and my nephew - and unified their faces into one unified geometry… this was a mess that I cleared out and retrained once I had a better outline of who the people were. Once I was on track, family members who I'd manually tagged a fair amount of saw accuracy ranging from 67-80%, the accuracy growing as the labeled data grew. The remaining gap was blurred due to the family resemblance and some individuals only having a photo or two. Still, this was a reference set where Rekognition couldn't help and in 48 minutes I ended up with strong classifiers added with 61% accuracy from hundreds of photos, which eventually weaves with human decisions in DynamoDB to establish our ground truth of what someone's face looks like.

For pets, I leveraged Meta's DINOv2 vision transformer small-sized model. Given the incredible training data set Meta enjoys with Facebook and Instagram, Dinov2 has been trained on 1.9 million animal photos, and it showed with nearly a 95% accuracy even on pets I only had a few pictures of. Unfortunately, my AMD RX6700XT doesnt support the algebra libraryhipBLASLtneeded for matrix multiplication and I saw kernel overhead that led to this training being 130x slower than I had planned. Fortunately, my AMD Ryzen CPU had high enough clock speeds and optimized vector instructions that mapped out vector geometry of cats and dogs at 121 milliseconds per image - given 64 cores on that CPU, DINOv2 geometry could be performed for 11000 photos in under 30 seconds, or 22 minutes if I was to go one photo at a time. Scene labels and the names of individuals are now searchable and browsable

The new photo library in S3 is around 75 GB covering 28k+ photos and scanned documents. Especially as we have small thumbnail copies for the quick views like grid, within a year, most of the 75 GB will go into deep storage where a gigabyte is around $0.001 a month, landing at well under $5 USD a year, even cheaper if I don't use the app much

Rekognition is performed in one-time indexing runs, and so far those have cost me around $37 total across a week of development & scans. I used Rekognition a few times before I discovered CLIP and other methods so if I'd used my current architecture from the start, I suspect that would be much lower. For a library of this size you can roughly figure $10 per complete pass. Storing Face Vectors is less than $0.01 a month for under 1000 faces, which I'm well under. Now that we have those vectors, we've enabled SearchFacesByImage, which allows us to upload a single new photo and check who is in it for $0.001. Operationally, I'm unlikely to have a Rekognition bill larger than $0.10 going forward

DynamoDB writes at $1.25 per million writes and 75% less for reads. Because our gallery views use precomputed S3 manifest, this essentially means DynamoDB is free. Cognito joins it as our user pool will never grow past the free tier of 50k monthly users. Assuming we doa few dozen updates to the app a year, we may pay $2 for CodeBuild minutes by 2027. Our Route53 hosted zone clocks in at $0.50 a month. Aside from the initial Rekognition scans, and it's under $15 per year for the entire architecture

As I collect images of myself, I plan on using these to rebuild my Facebook page and move other "keeper" photos onto Amazon Photos unlimited photo storage - now an organized and well labeled photo library - and retire this with a pat on the virtual back

Thank you for reading - ^.^