AWS Builder Center

Hindsight: Why an AI Agent Should Distrust Its Own Memory

This article explores how I built Hindsight with FalkorDB to help incident-response agents detect stale runbooks, trace service dependencies, and find the right people to contact. It also shares what I learned about building agent memory that can question its own outdated advice.

At 3:12 AM, nine alerts fire.
The team finds a familiar runbook: restart  cache-a . It has helped resolve previous incidents, and its wording matches the current symptoms. Following it seems reasonable.
There is one problem.
 cache-a  was retired six days ago.
That is the scenario behind Hindsight, a project I built for Graph Hacks by WeMakeDevs and FalkorDB. It explores a question that I think matters for any agent that relies on memory:
How does an agent know when something it remembers is no longer safe to use?
Retrieving an old answer is useful. Recognizing that the world has changed since that answer was written is harder.
The example above is synthetic. The company, Parcelo, its employees, incidents, and documents are fictional. But the underlying problem is familiar: documentation and operational reality gradually drift apart.
A runbook can remain easy to find long after it stops being correct.

Starting with the relationships

Hindsight is an incident-response and organizational-memory prototype for engineering teams.
Its company graph connects services, dependencies, people, teams, documents, incidents, procedures, and architectural decisions. FalkorDB stores those relationships and runs the queries the application uses to investigate an outage.
In the demo, several alerting services share a dependency on  cache-b . Following their dependency paths helps identify a common root-cause candidate. Traversing those relationships in the other direction reveals which services could be affected.
That gives the investigation a concrete basis: which services depend on what, and how far the effects can spread.
The graph also contains the history behind the topology. An architectural decision, ADR-17, records the migration away from  cache-a . The old runbook, R-7, still contains a step targeting that retired service.
Connecting those records is where the project becomes interesting.

The runbook that should be refused

R-7 has a strong history of successful use. If I ranked procedures only by past outcomes, it would look attractive.
Hindsight checks another condition before selecting it.
A procedure has steps. Those steps target services. Changes affect those same services. The query compares the change timestamp with the procedure’s last validation timestamp.
The relevant relationship pattern is:
 Procedure → Step → Service ← Change 
If an affecting change happened after the procedure was validated, Hindsight treats the procedure as stale.
For R-7, the graph finds the later change associated with ADR-17. The procedure recall query returns that invalidation alongside the historical outcome score, and the responder skips R-7 in favor of R-12.
There is an important tradeoff here. A later change does not necessarily prove that every instruction in a runbook is wrong. It does mean the runbook’s assumptions need checking. The prototype takes a conservative approach and refuses procedures that fail this check.
That is the behavior I wanted to demonstrate: an agent should be able to question a successful memory.

Giving memory different jobs

Hindsight represents memory in three forms.
Episodic memory records incidents and actions: what happened, which entities were involved, and what the outcome was.
Semantic memory records facts and their replacements. When a dependency changes, the earlier belief remains in the history with a closed validity interval. A supersession link connects it to the newer fact.
Procedural memory stores runbooks, their steps, and their outcomes. Recall considers decayed success history as well as changes that may invalidate the procedure.
These structures answer different questions. “What happened before?” is different from “What is true now?” and both differ from “What should we try next?”
Keeping those distinctions explicit makes the reasoning easier to inspect.

The same graph helps answer company questions

An outage also raises questions about people.
Who owns this service? Who has worked on it before? Who made the migration decision? Which documents need updating?
In the synthetic scenario, Priya has the strongest expertise score for the cache layer, but she is on leave. The expertise query filters out unavailable people and recommends Marco, whose incident history provides relevant evidence.
The distinction matters. Being mentioned frequently in old documents does not make someone the right person to contact today.
Decision records connect to supporting documents and discussion messages. A staleness audit compares document verification times with dependency changes, helping identify documentation that needs another review.
This is the company-brain part of Hindsight: operational knowledge, organizational context, and technical history share the same graph.

Why I included a Graph OFF mode

I wanted a visible comparison, so Hindsight includes a simple TF-IDF keyword baseline.
In the planted scenario, that baseline retrieves the stale runbook and selects Priya without the graph’s availability check.
The comparison has a narrow purpose. It shows the information lost when retrieval does not follow dependency, temporal, and availability relationships.
It is not a benchmark against every retrieval system or a sophisticated RAG pipeline. The baseline is deliberately simple, and the scenario is synthetic.

What building this exposed

One practical lesson was that writing correct graph queries is only part of building a coherent application.
The backend, event stream, and UI also need to agree on which graph they are reading. Hindsight uses separate graphs for tenant data and visitor sessions. During testing, a session mismatch exposed a frustrating failure: the workflow could write an action into a visitor graph while a panel read the tenant’s original graph and appeared empty.
That is a useful reminder for any application with isolated sessions. A successful database write does not guarantee that the user sees the result.
Another lesson was to distinguish historical performance from present validity. A procedure can have an excellent record and still be inappropriate after one architectural change. Those signals deserve separate checks.
The agents themselves are deterministic, policy-driven workers. Their remediation actions are simulated. This makes the demo reproducible and keeps attention on the graph reasoning, while leaving production execution and broader failure recovery as future work.
I used AI assistance for implementation support, debugging, and documentation, and have been tracking that assistance for disclosure. Testing the deployed behavior remains essential; generated code and convincing explanations do not establish that a feature works.
Hindsight is still a hackathon prototype. Session consistency, approval behavior, and UI evidence rendering are areas that need careful end-to-end validation.
But its central idea is already concrete: useful agent memory needs a way to recognize when its own advice has expired.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article