Two models live inside your agent's memory. You probably haven't picked either of them.
Agents are stateless by nature but an effective agent needs to remember across sessions and conversations. Context awareness turns an agent from a mere chatbot to a true assistant. Agent memory helps agents to be context-aware. But, memory is often over-looked while building agents. This blog walk through model considerations for agentic memory and some recommendations for agentic memory
I have been building agentic applications for the last few months and something keeps catching me out. The agent that books your travel asks for your departure city every time. The coding assistant re-discovers the project conventions every session. The field-service agent does not remember it visited the same site last month, even though every other technician on the team did.
These are all the same bug. The agents are good at retrieval (RAG) and good at calling tools, and they have no memory.
Memory is the third leg of context, and most teams skip it. When teams do add it, they almost always take whatever default the framework ships, which means there is a model running on every conversation, deciding what gets stored, that nobody on the team chose. This post is about that model, a second one that quietly runs alongside it, and what happened when I measured both against the alternatives.
What context-aware agents actually need
An agent needs three sources of context to be useful for more than one turn:
- RAG: documents the agent looks up at request time. Manuals, policies, knowledge bases. Stateless lookups.
- Tools: live data the agent calls. CRM, ticketing, inventory, sensors. Also stateless.
- Memory: what happened in past conversations with this user, on this account, about this entity. Stateful.
The first two are easy to build. There are good libraries, the contract is clear, and the data sources already exist. Memory is harder, because memory is the part that has to change as the world changes. The customer's preferred vendor changes mid-year. The asset gets a new model number. The contact retires. A memory layer that never updates is a memory layer that lies to you, slowly, over time.
That is why memory is where the architectural choices matter most. Skip it and the agent is amnesiac. Build it wrong and the agent confidently states stale facts.
The four kinds of memory
"Memory" is one bucket in most developers' heads, and it should not be. A working memory layer keeps four distinct kinds of records, because each answers a different kind of question:
Semantic facts: durable facts about the world. "Acme's chiller is a Carrier 30RB-100." "The patient is on warfarin." Answers "What model is the Acme chiller?" or "What medications is the patient on?"
Episodic memory: discrete past events with a time and a place. "On Sept 12 we found a leak at Acme CH-01, recharged it, scheduled a 30-day follow-up." Answers "What did we do about the Acme leak last fall?" Disambiguating one event from many similar ones is the hard part, and the part most defaults get wrong.
Session summaries: a running condensation of one conversation. "This call covered a noisy condenser; replaced the fan motor; noise dropped." Answers "Summarize the Lakeside visit."
User preferences: durable preferences scoped to a user, account, or site. "Acme orders parts through Grainger." "This developer prefers terse code reviews." Answers "Which vendor do we use for Acme?"
If you are using AgentCore Memory, these correspond directly to four named strategies, each configurable independently. If you are using Mem0, these are organized around scope (session vs user vs org) and share one extraction pipeline. Either way, the same four ideas are present, and the right model choice can differ between them.
Two models live inside every memory layer
This is the part that is fascinating, as the memory layer is an agentic system in itself! Every memory layer, regardless of vendor, runs two distinct model stages, and most teams never pick either of them.
Stage 1: extraction and consolidation
Every time a conversation completes, an LLM reads the turns and decides: what is worth remembering, and how do I write it down? When something new contradicts something old, the same LLM decides do I add, update, or delete the existing record?
This runs on every conversation. Sometimes on every turn. It is high-frequency background work: exactly the wrong job for a slow frontier model and exactly the right job for a fast cheap one, provided quality holds. Two properties matter:
- Cheap enough to run continuously. A cents-per-thousand-calls model is the floor; otherwise the memory layer becomes the cost driver of the whole system.
- Controllable. You need to override the prompt and the output schema, because your domain has part numbers and account tiers and ID conventions that a generic prompt does not know about.
Stage 2: embedding and retrieval
On every turn, before the responder model runs, the agentic framework issues a retrieval call. It packages the user's text (and any observed image caption) as a query string, hands it to the memory layer, and stuffs the top-K returned records into the responder's prompt as context. The embedder is what turns that query string into a vector and ranks stored records by similarity.
Worth pausing on: the responder LLM does not decide to retrieve. The framework retrieves on every turn, deterministically, and the responder sees only whatever made it into the context block. If retrieval misses the right record, the responder cannot fix it.
Two patterns: framework-managed vs tool-exposed
This part is worth knowing before you commit to a memory layer, because the two layers I tested handle it differently.
AgentCore Memory is framework-managed. The canonical pattern is
RetrieveMemoryRecords called by your agent code, typically wired into hooks like AgentInitializedEvent or MessageAddedEvent in the Strands SDK integration. Retrieval happens at turn boundaries, before the responder runs. There is no officially-supported tool schema that exposes memory to the responder LLM as a callable tool. If the conversation shifts topic mid-turn (the user asks a follow-up that needs different context), the responder is stuck with what was retrieved at turn start.Mem0 supports both patterns. You can call
search() from framework code in the same retrieve-before-respond shape, or you can register memory operations as tools the responder LLM can call itself: through Mem0's official MCP server exposing add_memory / search_memories / get_memories / update_memory / delete_memory, or through the @function_tool decorator pattern shown in the OpenAI Agents SDK cookbook . The docs explicitly say: "AI agents decide when to save, search, or update memories."The trade-off is determinism vs flexibility. Hooks retrieve on every turn, predictably, even when not needed. Tools retrieve only when the LLM thinks to call them, which is great when the topic wanders and dead-weight when it does not. Narrow-topic agents (a single support flow, a coding agent inside one repo) fit AgentCore's framework-managed pattern fine. Conversational agents that wander (general assistants, multi-account triage) are where you actually need the responder to be able to issue a search mid-turn, and that is the Mem0 advantage. You can wrap
retrieve_memory_records as a custom tool on AgentCore yourself, but it is not a blessed pattern in AWS docs today.The right embedder depends on the modality:
- Text-only applications: any competent text embedder is fine.
- Multimodal applications, where queries can be photos, audio, or scanned documents, need an embedder that puts all those modalities into the same vector space. Otherwise you cannot ask "have I seen this exact part before?" with a photo.
These two stages are where the cost lives, where the quality lives, and where the architectural choices matter. The framework default picks both for you, silently. If you have not looked at what those defaults are, you should.
Why fast and cheap matters more than frontier here
Quick arithmetic. Suppose you run an agent for 100 users, each having 5 substantive conversations per day, 250 days a year. That is 125,000 extraction calls per year, just for Stage 1.
Bedrock list pricing as I write this:
| Model | Input ($/1M tok) | Output ($/1M tok) |
|---|---|---|
| Nova 2 Lite | $0.06 | $0.24 |
| GPT-4o-mini | $0.15 | $0.60 |
| Claude Haiku 4.5 | $1.00 | $5.00 |
| Claude Sonnet 4.6 | $3.00 | $15.00 |
A typical extraction call (~1.2k input tokens with retrieved similar memories + system prompt, ~400 output tokens for the decision + record bodies) costs:
| Model | Per-call | Annual at 125k calls |
|---|---|---|
| Nova 2 Lite | $0.000168 | $21 |
| GPT-4o-mini | $0.000420 | $52 |
| Haiku 4.5 | $0.003200 | $400 |
If you reach for Haiku because "Anthropic models are good," you have just signed up for $400/year on the memory layer of a 100-user deployment. Scale to 1,000 users, that is $4,000/year on extraction that Nova 2 Lite would do for $210.
This is the argument for picking a Stage 1 model deliberately: extraction is mostly mechanical work that runs continuously, and the right model is the cheapest one whose quality holds for the task. Frontier models on this job are like buying a Ferrari to deliver pizzas. You pay for capability you are not exercising.
The remaining question: does Nova 2 Lite's quality hold? I went and measured.
The schema is the lever, not the model
This is the finding that changed how I think about extraction.
I built a field-service technician scenario with the same data and the same question set, across three configurations of AgentCore Memory:
- Default built-in episodic strategy. AgentCore's default model and prompt, fixed prose-paragraph output format.
- Override. Same fixed format, but Nova 2 Lite as the extractor with my custom prompt instructions.
- Self-managed. Nova 2 Lite and my own output schema. I control what the record looks like.
The work-order corpus had three Northgate visits and two hammer-related episodes. They read alike in prose. The query: "What did I do at Northgate last November about the water hammer?"
The built-in and override configurations both store the episode as a paragraph with the identifiers buried inside the prose:
1
2
3
4
The technician visited Northgate Tower on November 18, 2025 and
addressed a water hammer condition on boiler BLR-03 by tightening
a loose pipe support strap on the 2-inch supply riser, restoring
normal operation per work order wo_003.
Self-managed lets me write it however I want. I put the keys at the top of the record:
1
2
3
4
site_id: site_northgate asset_id: blr_03 work_order_id: wo_003
Loose pipe support strap on 2-inch supply riser; tightened, hammer
resolved.
That single change, same model, different record format, moved disambiguation retrieval from ~50% to ~80% at recall@1. A +36-point lift, no model change.
The lesson is that the schema is the lever, not the model. A controllable extractor lets you encode entity scope into the record shape, and entity scope is what makes retrieval pick the right record when ten work orders share vocabulary. Picking Nova 2 Lite is necessary because it is cheap enough to run on every conversation and because it accepts overrides and self-managed configurations. But the win is unlocked by what you do with the override, not by the model on its own.
This same instinct works for semantic facts. The default extraction in my test sometimes recorded "3.0A FLA" when the conversation said "1/2 HP, 3.5A FLA". A generic prompt does not know to preserve units. One override line, "preserve part numbers and ratings with units", fixed it. Same model, sharper instructions, accurate facts.
For summaries and preferences, where the output shape is more constrained (one paragraph or one keyed value), the defaults perform fine. Nova matches them at lower cost; it does not pull ahead. The win is on episodic and semantic, where you can control the record shape.
When the query goes multimodal
The technician sends a photo of a corroded service valve and asks: "Have we worked on this exact valve before?"
The first instinct is to caption the photo with a vision-language model and search the caption against your text memory. That works for generic matches but fails when two sites have visually identical valves on different chillers. Captions describe categories, not individuals. "Brass service valve with light corrosion" describes ten thousand valves.
The fix is to keep the caption and index the raw photo through a multimodal embedder into a vector store. The technician's new photo embeds into the same vector space as past photos, and image-to-image similarity returns the exact past visit.
This is what Nova MME does. It embeds image, audio, video, text, and documents into one vector space, so a photo query and a text query land in the same coordinate system, and the same retrieval call finds whichever record matches.
I ran this on real Wikimedia Commons photos with deterministic augmentation: different shots of the same physical asset, not the same image file twice. That part matters; "the same JPEG retrieves itself" is not a re-identification result. Nova MME hit recall@1 = 0.89. The dual-layer setup (captioning + Nova MME side by side) beat captioning-only by +0.17 recall@1 and +0.28 judge points.
This is the kind of property captions cannot replace. A description of a valve is not the same signal as the valve.
Putting it together
Two stages, two requirements:
- Stage 1 wants a fast, cheap, controllable extraction LLM. Nova 2 Lite. Lowest-cost text model on Bedrock, accepts override prompts and self-managed schemas, fast enough to run on every turn.
- Stage 2 wants an embedder that spans modalities. Nova MME. Image, text, audio, video, documents in one vector space, so the same retrieval call works regardless of what the user sent.
The pairing is intentional, not a marketing alignment. Different problems, different models, same family, which means one billing surface, one region, one set of guardrails, one set of logs.
In my Mem0 comparison (Nova vs the OpenAI default), the cost arithmetic was the smaller story. The bigger one was quality:
| Nova 2 Lite + Nova MME | gpt-4o-mini + text-embedding-3-small | |
|---|---|---|
| Judge avg /2 | 1.11 | 0.82 |
| % perfect | 43% | 25% |
| Cost per WO | $0.00023 | $0.00044 |
| $ per correct answer | $0.00053 | $0.00176 |
Half the cost per work order. 1.7× more correct answers. A 3.3× better cost-per-correct-answer. Why does Nova win on quality? Because the records it writes are short, canonical, and single-fact-per-record: "the current preferred parts vendor for Acme Manufacturing is Ferguson". The OpenAI default produced longer, looser memory bodies that mixed multiple facts into one record, and the responder gets confused when two such records disagree. Both layers retrieve the right candidates; the Nova layer gives the responder cleaner material to work with.
The honest caveat: live extraction does not consolidate over time
Here is what no model fixed. Acme's preferred parts vendor changes mid-year, Grainger to Ferguson, introduced in a routine work order. The technician later asks: "What's Acme's current parts vendor?"
Every configuration I tested still has both records in the database. Mem0 chose ADD on the new value with both Nova and OpenAI as the decision-maker. Neither identified the contradiction. AgentCore added the new preference but did not remove the old fragment. Across 8 ledger contradictions in my 50-WO longitudinal test, 0 were cleanly consolidated by any of four configurations.
This is not a Nova problem. It is a property of live extraction itself. Live extraction sees one conversation at a time, and the prompt that decides ADD vs UPDATE only sees prior memories that happen to match the new conversation in vector space. When the new conversation is mostly about something else (a routine PM that mentions the vendor in passing), the contradicting memory simply is not in the prompt context, and the LLM defaults to ADD because that is the safe play.
The fix is a small nightly job, not a smarter live extractor. Group records by
(entity, topic), send conflicting groups to a heavier reasoning model, explicitly delete superseded fragments. About 50 lines of code. Roughly twenty cents a night for a 100-user fleet. Sketch:1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# nightly_reconcile.py: runs at 02:00 every night
all_memories = mem.get_all(filters={"user_id": "tech_maya"}, version="v2")["results"]
by_site_topic = defaultdict(list)
for m in all_memories:
site = m.get("metadata", {}).get("site_id")
topic = classify_topic(m["memory"]) # regex first, LLM fallback
if site and topic:
by_site_topic[(site, topic)].append(m)
for (site, topic), records in by_site_topic.items():
if len(records) <= 1:
continue
records.sort(key=lambda r: r["created_at"])
decision = call_nova_pro(reconcile_prompt(records))
for stale_id in decision["superseded_ids"]:
mem.delete(memory_id=stale_id)
classify_topic is regex-first because the topics that change are a known small set (vendors, contacts, thresholds) and regex is faster and more predictable than an LLM for known taxonomies. The LLM only fires on the actual reconciliation decision, where reasoning matters.The reason to call this caveat out is that most blog posts in this space pretend memory layers handle updates automatically. They do not. Pretending they do is how you ship the agent that confidently states the wrong vendor name three months later. Naming the failure mode honestly is what makes the rest of the recommendation defensible.
What to take away
Three things worth remembering:
- Two models live inside every memory layer, and the framework default picks both for you. The extraction LLM runs on every conversation; the embedder controls retrieval ranking and modality coverage. Both deserve a deliberate choice.
- The schema is the lever for extraction quality, not the model. A cheap controllable extractor (Nova 2 Lite) plus a record format you design yields larger recall lifts than swapping in a frontier extractor with the default schema would. Fast and cheap and controllable beats slow and capable here.
- Live extraction does not consolidate facts over time, not in any architecture I tested, not with any model. Plan for a nightly reconciliation job from day one. It is fifty lines of code and it is the difference between an agent that gets smarter as it accumulates context and an agent that gets more confidently wrong.
For the AWS-native stack, Nova 2 Lite + Nova MME is the cheapest credible memory-layer choice today, and it does not sacrifice quality to get there. In my measurements, it beats the OpenAI default on both cost-per-WO and correct-answer rate, and it matches frontier Anthropic models on extraction quality at a fraction of the cost. That is not a marketing claim; it is what the numbers said when I swapped models in and ran the same evals across both AgentCore Memory and Mem0.
Now go look at what model your memory layer is actually running.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article