AWS Builder Center
Built an AI Memory System. Then Spent Hours Debugging CORS.

Built an AI Memory System. Then Spent Hours Debugging CORS.

Building Remora for a hackathon taught us that the hardest bugs aren't always in the AI. Sometimes the model is fine, the backend is fine, and the entire problem is one missing CORS configuration.

I Built an AI Memory System. Then Spent Hours Debugging CORS.

I started building Remora around a simple question:
What if an AI application could actually use the information it has been given more intelligently?
Instead of treating every input as something the model has to process from scratch, I wanted to build a system that could extract useful information, turn it into structured memories, and retrieve relevant context when answering a query.
That sounds simple.
It wasn't.
One of those things turned out to be CORS.
But before getting there, I had to figure out what a useful AI memory pipeline should actually look like.

So, What Is Remora?

Remora is an AI memory layer designed to extract useful information from inputs, turn it into structured memories, and retrieve relevant context when answering a query.
The basic idea is:
Input → Extract information → Create memory → Store it → Retrieve relevant memories → Generate an answer
The input doesn't have to be just text.
A user might upload an image, a document, or provide information during a conversation.
For example, imagine I upload an image containing:
MacBook Pro 14 M3 — $1,599
Remora can extract that information and turn it into a memory.
A relevant query can then retrieve that memory and provide it as context to the model.
That is the core idea behind Remora:
Extract useful information, remember it within the system, and retrieve the right context when it matters.

The Interesting Part: Memory Isn't Just a Database

My first instinct was basically:
"I'll store information somewhere, add embeddings, search for similar things, and give the results to the LLM."
That works as a starting point.
But pretty quickly I realized that a useful memory system involves much more than vector search.
There are several questions the system has to deal with:
  • What information is worth remembering?
  • How should information from an image or document be represented?
  • How do I find the right memory for a vague user query?
  • What happens when several memories look similar?
  • How much retrieved context should be given to the model?
  • What happens when the retrieved memory is incomplete?
  • How do I prevent irrelevant memories from influencing an answer?
For example, if I have memories about ten different laptops and ask:
"What was the price of this product?"
The system shouldn't just return ten vaguely similar memories and hope the LLM figures it out.
The retrieval layer needs to provide useful context.
That made Remora less like a simple chatbot and more like a small information-retrieval system sitting underneath an AI application.

Building the Pipeline

I ended up separating the system into a few different pieces.
The frontend handles the user experience.
The backend exposes the API and coordinates the application logic.
A worker handles processing that doesn't need to block the user request.
Then there is the memory layer, where information is stored and retrieved.
The general flow looks something like this:
┌──────────────┐
│ User │
└──────┬───────┘
│
▼
┌─────────────────┐
│ Frontend │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Backend API │
└────────┬────────┘
│
┌───────────┴───────────┐
▼ ▼
┌─────────────┐ ┌─────────────┐
│ Memory │ │ Worker │
│ Layer │ │ Processing │
└─────────────┘ └─────────────┘
│
▼
┌─────────────┐
│ Retrieved │
│ Context │
└──────┬──────┘
│
▼
┌─────────────┐
│ LLM │
└─────────────┘
For the AWS side, I used AWS storage and vector capabilities as part of the memory architecture, while the application backend and worker run on an EC2 instance.
The frontend is deployed separately.
This separation made the project easier to develop, but it also created a completely new category of problems.

"It Works on My Laptop"

This is probably the most predictable sentence in software development.
During development, everything was running locally.
The frontend could talk to the backend.
The backend could access the required services.
Memories could be created.
Queries could retrieve them.
Then I deployed the application.
And suddenly, things started behaving differently.
This is where I learned that deploying an application isn't simply:
"Take the local project and put it on a server."
Now there are domains.
There is HTTPS.
There is Nginx.
There are environment variables.
There are separate frontend and backend origins.
There are browser security policies.
There are Docker containers.
There are services running in different places.
And every connection between those pieces becomes another thing that can go wrong.

The Mystery of the Missing Memories

One of the most confusing bugs I hit was when the application appeared to have no memories.
The UI was working.
The backend was running.
The memory existed.
The retrieval logic was working when I tested it directly.
But the application behaved as if nothing had been stored.
So I started looking in the obvious places.
Was the memory being created?
Yes.
Was it being stored?
Yes.
Could the backend retrieve it?
Yes.
Was the retrieval query correct?
Yes.
At that point I started questioning basically everything.
I went deeper into the retrieval pipeline.
I looked at API responses.
I checked the frontend.
I checked the deployment configuration.
And eventually I found the problem.
CORS.
The browser was blocking the request because the production frontend origin wasn't correctly allowed by the backend configuration.
That was it.
Not a broken embedding.
Not a broken vector search.
Not an LLM problem.
Not corrupted memory.
Just a browser refusing to let the frontend make the request.
I fixed the allowed origins, refreshed the application, and suddenly the memories were back.
After spending hours investigating the "AI memory system," the actual solution was a CORS configuration.
Classic.

The Lesson Was Bigger Than CORS

That incident actually changed how I debugged the rest of the project.
When something goes wrong in an AI application, my first instinct used to be to look at the AI component.
Is the prompt wrong?
Is retrieval broken?
Is the model behaving strangely?
Is the embedding incorrect?
But a production application has a lot more layers than that.
The failure could be anywhere between:
Browser
↓
Frontend
↓
Network
↓
Nginx
↓
Backend
↓
Worker
↓
Storage
↓
Vector Search
↓
LLM
If I only investigate the last three layers, I can waste a lot of time.
The CORS problem was a good reminder to debug from the outside in.
Where does the request actually stop?
That question is often more useful than:
Which complicated AI component is broken?

AWS Made the Project Feel Real

One thing I wanted from this project was to move beyond a local demo.
It's easy to make an AI prototype where everything runs on one machine.
It's a very different experience when the application actually has to survive outside your development environment.
I ended up using AWS as part of the production architecture, including storage and vector-based memory capabilities, with the backend and worker infrastructure running on EC2.
I also put Nginx in front of the backend and configured HTTPS for the public API.
Suddenly I had to think about things that don't show up in a simple prototype:
  • How should the frontend communicate with the backend?
  • Where should uploaded data live?
  • How should memory be persisted?
  • How should background processing work?
  • How do I handle failures?
  • How do I expose the API securely?
  • How do I configure CORS?
  • What happens when a worker stops?
  • How do I know whether a problem is in the application or infrastructure?
Those questions weren't really part of the original idea.
But they became a major part of actually building it.

What I Learned About Building AI Applications

Building Remora changed the way I think about AI applications.
The model is only one component.
A good AI application needs the infrastructure around the model to work properly.
For a memory system especially, the retrieval pipeline is just as important as the generation step.
If the system retrieves the wrong information, even a great model can produce a bad answer.
If the system retrieves the right information but the frontend can't reach the backend, the user still sees nothing.
And if everything works locally but the production configuration is wrong, none of it matters.
So the real system is something closer to:
Data + Retrieval + Infrastructure + Model + Application Logic + User Experience
not simply:
Prompt + LLM

What I Would Do Differently

If I were starting Remora again, I would spend more time thinking about observability from the beginning.
Not just logging errors, but being able to answer very quickly:
  • Did the request reach the backend?
  • Did memory retrieval run?
  • How many memories were found?
  • Which memory was selected?
  • Did the LLM receive the expected context?
  • Did the frontend actually receive the response?
AI systems can be difficult to debug because a final incorrect answer doesn't necessarily tell you where the problem happened.
A useful trace across the entire pipeline is much more valuable than staring at the final response and guessing.
I would also validate production configuration much earlier.
Because apparently spending hours debugging an AI system only to discover CORS is an excellent way to learn that lesson.

Where Remora Is Going

The current version is still a work in progress.
Right now, I'm focused on making the memory pipeline reliable — extracting useful information from inputs, storing it correctly, retrieving relevant memories, and providing that context to the model when it is needed.
There is still plenty to improve.
The bigger question I'm exploring is how much useful context an AI application can derive from the information it has access to, without simply throwing everything at the model.
For me, that's what makes the memory problem interesting.
It's not about storing everything.
It's about finding the right information at the right time.

Final Thoughts

Remora started with a simple question:
What if an AI application could remember?
The answer turned out to involve much more than a memory database.
It involved retrieval systems, multimodal processing, APIs, cloud infrastructure, background workers, deployment, debugging, and a surprisingly large amount of time looking at things that had absolutely nothing to do with AI.
And somewhere in the middle of all that, I learned one of the most useful lessons of the project:
When your AI application stops working, don't immediately blame the AI.
Check the boring stuff first.
Check the network.
Check the API.
Check the environment variables.
Check the logs.
And yes...
Check CORS.
Because sometimes your sophisticated AI memory architecture is working perfectly.
Your browser doesn't want to talk to it.

Try Remora

Remora is live, and the code is available to explore.
You can try the application, look through the implementation, and see how the different pieces come together.
I'm still improving Remora, especially the memory and retrieval side, so feedback is always welcome.
Thanks for reading — and hopefully your next production bug is something slightly more complicated than CORS. 😄
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article