I Built a Personal Memory Layer on AWS — Then Production Taught Me the Hard Parts
Recollect lets users find forgotten files by describing what they remember and ask grounded questions across documents, images, audio, and video. This is the story of how I built it on AWS and the production bugs that changed the architecture along the way.
We all have a digital memory problem.
A screenshot from three months ago. A PDF buried in Downloads. A lecture note you vaguely remember, but cannot name.
The information still exists. The difficult part is finding it again.
That was the idea behind Recollect:
Upload your digital files once, then later find them by describing what you remember.
Recollect supports documents, images, audio, and video. Users can search semantically, ask grounded questions, follow up conversationally, and open the original source behind an answer.
But the most useful lessons did not come from drawing the architecture.
They came when the architecture broke.
How Recollect works
There are two main flows: ingestion and retrieval.
When a user uploads a file, the backend does not proxy the bytes. It generates a presigned URL and the browser uploads directly to Amazon S3.
From there, an S3 event enters Amazon SQS, which buffers ingestion work. AWS Lambda coordinates processing, Amazon Bedrock Data Automation handles multimodal parsing, Titan Text Embeddings V2 generates embeddings, and S3 Vectors stores the semantic index.
DynamoDB stores application metadata and conversational state.
For retrieval, Amazon Bedrock Knowledge Bases powers semantic search. For grounded Q&A, Recollect retrieves relevant context and uses Amazon Nova Lite to generate an answer with citations.
Authentication is handled by Amazon Cognito, requests enter through API Gateway, the main backend runs on Java 21 Lambda with SnapStart, and the frontend is hosted on AWS Amplify.

Bug 1: Authentication worked — until the first real authenticated request
The frontend logged in successfully. API Gateway validated the JWT. Then my
/me endpoint returned HTTP 500.CloudWatch revealed:
ClassCastException:
LinkedHashMap cannot be cast to HttpApiV2JwtAuthorizerThe problem was not authentication itself. It was event deserialization.
My Lambda entry point originally used a typed
RequestHandler , which relied on Lambda's default event deserialization. The JWT authorizer claims arrived as a generic map instead of the typed model expected by the serverless Java container.I changed the entry point to a RequestStreamHandler and let the framework deserialize the raw API Gateway event itself.
The same JWT then worked correctly.
Lesson: testing business logic and testing the real AWS event path are not the same thing. I added a regression test that invokes the Lambda entry point using a raw HTTP API v2 event.
Bug 2: Serverless still has backpressure
My first ingestion design assumed that multiple uploaded files could simply start multiple Knowledge Base ingestion jobs.
In practice, active ingestion work for a data source was effectively serialized.
Instead of retrying aggressively, I redesigned around the constraint:
S3 upload
↓
SQS queue
↓
Ingestion worker
↓
Bedrock processingSQS became the backpressure boundary. It absorbs bursts and lets the downstream Bedrock workflow process work at a safe pace.
Lesson: managed services remove infrastructure management, but not distributed-systems concerns such as concurrency, retries, idempotency, backpressure, and recovery.
Bug 3: Vector search always finds something
This was the most important retrieval lesson.
If a user asks:
“What is my electricity bill?”
but no electricity bill exists, vector search still returns the nearest vectors.
“Nearest” does not mean “relevant”.
During early testing, Recollect could surface an unrelated document simply because it was mathematically the least unrelated option.
I created roughly 75 labeled queries covering clear matches, paraphrases, vague topics, and intentionally absent topics. Based on those results, I introduced a corpus-calibrated relevance threshold of 0.62.
Now, if nothing passes the relevance gate, Recollect returns:
“I couldn't find anything in your memories that answers that.”
and does not call the model.
Lesson: a grounded system is not trustworthy merely because it has citations. It also needs to know when not to answer.
Bug 4: “Find my assignment” is not “Explain my assignment”
Another issue appeared in the conversational Ask flow.
A request such as:
“find numerical method assignment”
was being treated as a normal Q&A request. Retrieval found the correct PDF, but the generation step had no sensible factual question to answer.
I separated file discovery from content Q&A.
For discovery requests, Recollect checks user-owned filename metadata and semantic candidates, returns the real document citation, and stores the selected document IDs as server-owned conversational context.
Then the follow-up:
“explain this assignment”
can be grounded to the file the user just found.
Lesson: natural-language products are not just “search” and “chat”. Users move between finding, opening, explaining, and following up on the same memory.
Bug 5: Deployment bugs changed how I shipped
One deployment issue looked like a backend bug: I changed code, deployed the stack, and the behavior did not change.
The real problem was a stale JAR. CDK had deployed an older backend package.
The fix was not another debugging session. The fix was removing the possibility of repeating the mistake.
I created a deployment wrapper that:
- performs a clean backend build,
- runs tests,
- runs
cdk diff,
- requires explicit confirmation,
- deploys only explicitly selected stacks using
--exclusively.
Later, I added GitHub Actions CI/CD. Pull requests and pushes validate the frontend, backend, and infrastructure independently.
Infrastructure deployment uses GitHub OIDC to obtain short-lived AWS credentials, followed by:
cdk diff
↓
production approval
↓
cdk deploy
↓
health checkEven this exposed real production details: the OIDC trust policy initially used the wrong GitHub subject format, then the deploy role proved too restrictive for
cdk diff . I fixed each permission narrowly instead of broadening the role.Lesson: safe deployment is part of system design.
Reliability became a feature
By the end, Recollect included several mechanisms that were not visible in the first architecture diagram:
- SQS backpressure
- a dead-letter queue
- idempotent ingestion
- scheduled reconciliation
- stale-state cleanup
- deterministic no-answer behavior
- bounded conversational recovery
- structured logging
- CloudWatch alarms
- SNS notifications
- OIDC-based deployment
None of these make the landing page look more impressive.
But they are the difference between:
“The demo works”
and:
“The system has a defined behavior when things go wrong.”
Where AWS helped the most
The biggest advantage of AWS was not any single service. It was how well managed services could be composed into a serverless product.
I did not need to operate EC2 instances, Redis, OpenSearch, a relational database cluster, or always-on application servers.
AWS CDK was especially valuable because infrastructure changes became code changes I could test, diff, review, and reproduce.
The AWS Agent Toolkit also made development much faster. I used it to inspect deployed resources, review CloudWatch behavior, check quotas, and validate AWS-specific assumptions while debugging newer services such as Bedrock and S3 Vectors.
The biggest lesson
When I started Recollect, I thought most of the work would be:
upload files
→ create embeddings
→ search vectors
→ call an LLMThat part came together relatively quickly.
The harder questions were:
Should I answer this query?
Who owns this result?
What happens when ingestion is busy?
Can retries duplicate work?
What happens when the model retrieves nothing?
Can a deployment silently ship stale code?
Can CI deploy without long-lived AWS credentials?Those questions changed Recollect from an AI prototype into a system I could reason about.
And that was the most valuable part of building it.
Recollect
Your digital life, remembered.
- Live application: https://main.d28nd6lc9fjyiv.amplifyapp.com
- Engineering walkthrough: https://main.d28nd6lc9fjyiv.amplifyapp.com/engineering
- Source code: https://github.com/vansh-005/first-commit
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article