AWS Builder Center
DynamoDB Vector Search: Building an Incident Copilot

DynamoDB Vector Search: Building an Incident Copilot

Building an incident retrieval agent on DynamoDB's new native vector search, with MCP tools, a LangGraph agent, and end-to-end tracing.

Introduction
On August 5, 2026, DynamoDB shipped native vector search. Two days later I started building on it.
The project is Incident Memory Copilot: an agent that takes a fresh alert description and surfaces what broke last time, along with the fix that worked. Embeddings live on the same DynamoDB item as the operational incident data — no second database, no sync pipeline.
But the honest motivation wasn't the vector search. I maintain an open-source OpenTelemetry instrumentation library for MCP servers called  opentel-mcp , and I wanted a realistic system to point it at. Demos I write for a library tend to confirm what I already believe about it. I wanted something that could contradict me.
It did. Building this surfaced two real gaps in my own library, one of which means a headline feature cannot fire at all under a common transport configuration. That finding is the most valuable thing in this article, and I didn't go looking for it.



The Problem

Every engineering team accumulates institutional memory about failures, and almost all of it evaporates. The OpAMP config bug that took a day to find lives in one person's head, a Slack thread nobody can search, and a commit message. Six months later someone hits the same class of failure and starts from zero.
Existing options are unsatisfying:
  • Runbooks require you to already know what you're looking for. They're organized by cause; alerts arrive as symptoms.
  • Full-text search over incident tickets fails on the case that matters most — two incidents with near-identical symptoms and completely different root causes share almost no vocabulary.
  • A dedicated vector database solves retrieval but adds a second system to operate, and your incident metadata now lives in one place while its embedding lives in another, with a sync pipeline in between that can silently drift.
That last point is what DynamoDB's launch changes.



The Solution

Store the incident and its embedding on the same item. Query it with vector similarity, filter it with the operational attributes already sitting there, and expose the whole thing to an agent through MCP tools.
An on-call engineer describes what they're seeing in plain language. The system embeds that description, runs a similarity search against past incidents, and — when it finds a good match — pulls the runbook and checks whether the service currently has open incidents. The agent then produces a diagnosis grounded in what actually happened before, not in what a model guesses might be happening.
Because MCP is the tool layer, the same tools work from a LangGraph agent, from Claude Desktop, or from anything else that speaks the protocol.



Vision

The intended journey is short by design:
  1. An alert fires. The engineer pastes the symptom into the copilot — no structured query, no knowing which service to look under.
  1. The copilot searches semantically across every past incident.
  1. If a strong match exists, it retrieves the runbook and confirms the service's current status.
  1. It returns a diagnosis: likely root cause, the fix that worked, current health.
  1. When the incident is resolved,  record_incident  writes it back — so the corpus improves with use.
The expected outcome isn't "the AI fixes it." It's cutting the distance between I have seen this before and here is what we did.



Key Features

  • Native DynamoDB vector search — embeddings and operational data on one item
  • Inline filtering by  service  and  status  inside the vector query
  • Four MCP tools over streamable HTTP
  • LangGraph agent with a hard tool-call ceiling
  • End-to-end distributed tracing across a Python → Node process boundary
  • Deliberate failure scenarios that produce real, reproducible traces
  • Swappable embedding provider behind a single interface



How I Built It

I ran this in five strictly scoped phases, one per working session, with a full test suite and a commit between each. No phase started before the previous one was green.

Phase 1 — Data layer

DynamoDB table plus vector index, embedding pipeline, seeded corpus.
The first thing I hit: the feature was 36 hours old, so every coding assistant's training data predated it. Left alone, they will confidently invent an API — plausible parameter names, a  CreateVectorIndex  call that doesn't exist. I made "read the actual AWS documentation and check the installed SDK's TypeScript types before writing anything" a hard gate on the session.
Second thing: AWS CLI v2.36.12 silently omits  VectorIndexes  from  describe-table  output. Not an error — the field just isn't in the older service model. The index existed and was  ACTIVE ; the CLI showed nothing. Verified via the SDK instead.
I seeded 8 real incidents from my own projects rather than synthetic data, deliberately including several with near-identical symptoms and different root causes.

Phase 2 — MCP server

Node 22 + TypeScript, four tools, streamable HTTP transport.
The transport choice was made in Phase 1 and it mattered: the agent is Python, the server is Node. Trace context has to cross that boundary via HTTP headers. Over stdio it doesn't, without custom plumbing.
I asked for verification by raw JSON-RPC round trip, not just unit tests. That caught a bug all 8 tests missed — see Development Challenges.

Phase 3 — Instrumentation

 opentel-mcp@0.7.0  from npm, OTLP export to Jaeger in Docker.
Before checking whether spans arrived, I checked that Jaeger's OTLP ports were actually bound with  ss -tlnp . This is not paranoia — "everything green, nothing listening" is incident #1 in my own seed data. Good thing, too: the standard ports were already held by an unrelated container, so host ports were remapped to 4319/4320.

Phase 4 — LangGraph agent

Python agent, OpenAI, verified trace propagation.
This was the session with real risk. If  traceparent  didn't cross Python → Node, I'd have two orphaned traces and no end-to-end story.

Phase 5 — Failure scenarios

Three reproducible demos: silent failure, agent thrash, cost visibility gap. This is where the library findings came from.



Technical Stack

Layer
Technology
Frontend
None — CLI entrypoint (main.py)
Backend
Node.js 22 + TypeScript (MCP server)
Database
Amazon DynamoDB with native vector search
Vector Store
Amazon DynamoDB (1024-dim, COSINE, inline filters)
Embeddings
OpenAI text-embedding-3-small (1024 dims)
AI / LLM
OpenAI gpt-4o-mini
Agent Framework
LangGraph (Python)
Tool Protocol
Model Context Protocol, streamable HTTP transport
Authentication
None — local development project
Hosting
Local (Node server + Python agent)
Infrastructure
Docker Compose (Jaeger)
Observability
opentel-mcp + OpenTelemetry SDK → Jaeger

AWS Services Used

AWS Service
Purpose
Amazon DynamoDB
Stores incident records and their embeddings on the same item; runs vector similarity search with inline attribute filtering
DynamoDB Vector Index
1024-dimension COSINE index with service and status as inline filter attributes
Amazon Bedrock (Titan Text Embeddings V2)
Originally the embedding provider — blocked by account verification and swapped for OpenAI. See Development Challenges
AWS IAM
Credential scoping for the DynamoDB client
DynamoDB is doing the interesting work here. The embedding provider is deliberately interchangeable and is not the story.



Architecture



Data flow

  1. The engineer's alert text enters the LangGraph agent.
  1. The agent opens a root span and calls  search_similar_incidents  over MCP. The active span's W3C  traceparent  is injected into the HTTP headers.
  1. The MCP server extracts that header into its active context, so its spans nest under the caller's rather than rooting a new trace.
  1. The tool embeds the query text, runs a DynamoDB vector search with optional  service  /  status  inline filters, and returns ranked incidents with distance scores.
  1. If the top score is strong, the agent calls  get_runbook  and  check_service_health .
  1. The agent produces a diagnosis. Spans from both processes land in one Jaeger trace.



AI Architecture

Embeddings —  text-embedding-3-small  at an explicit 1024 dimensions, matching the vector index exactly. The  dimensions  parameter matters: OpenAI supports 1024, 512, or 256, and the index dimension and embedding dimension must match or writes fail. I kept  EMBEDDING_DIMENSIONS  as a single constant used by both the table-creation script and the embedding client, so they cannot drift.
Retrieval — COSINE distance, lower is better. Only the  symptom  field is embedded. I tested embedding  symptom + rootCause  as an A/B and it made things worse — details in Development Challenges.
Tool calling — Four MCP tools. Distance scores are returned in the tool response, not stripped, because they become span attributes and are what reveal a low-confidence retrieval.
Agent (LangGraph) — A deliberately simple graph: one agent node bound to three read-only tools, one tool-execution node, looped until the model stops requesting tools or  MAX_TOOL_ROUNDS = 4  is hit. On hitting the cap, control routes to a  finalize  node that calls the model with no tools bound — so it is structurally unable to request another call. That's a hard stop, not a hint. A LangGraph  recursion_limit  sits underneath as defense in depth.
Prompt engineering — The system prompt tells the model explicitly that  score  is a distance where lower is better, and that a top score above roughly 0.5 is a weak match it should say so about rather than guess around. Without that, models confidently treat a poor match as an answer.
Guardrails — Hard iteration cap, capped  max_tokens , no unbounded loops anywhere that calls a model.



Core Components

Retrieval layer ( src/retrieval/ ) — Single implementation of search, fetch, health, and record. Both the MCP tools and the CLI verify script call into it, so there is exactly one code path to the database.
Embedding provider ( src/embeddings/ ) — An  EmbedText  interface with one implementation behind an index module that is the sole swap point. Switching providers is one file. This turned out to matter more than expected.
MCP server ( src/mcp/ ) — Stateless: a fresh server and transport per HTTP request. The POST handler extracts  traceparent  and runs the request under that context.
Agent ( agent/ ) — A fully separate Python project with its own venv and requirements. No Python dependencies mixed into the Node project.
Demos ( demos/ ) — Three runnable scenario scripts, each printing a trace ID for inspection in Jaeger.



Development Challenges

Challenge
Solution
DynamoDB vector search shipped 2 days earlier — no coding assistant knows the API and will hallucinate one
Made "read the live AWS docs and check installed SDK types" a hard gate before any code. Verified against the real SDK, never memory
AWS CLI v2.36.12 silently omits VectorIndexes from describe-table — no error, field just absent
Inspected vector index state via the SDK and console instead of the CLI
Bedrock returned ValidationException: Operation not allowed for every model. Access page retired, use-case form returned "account not authorized"
Root cause was new-account verification still in progress (up to 2 days) — surfaced only when CloudShell failed with a plain-English message. Swapped to OpenAI behind the provider interface, kept building
Vector index leaked the full 1024-float embedding into every tool response
The vector index's Projection: ALL projects all sibling attributes on the item, not just index-relevant ones. Fixed with an explicit ProjectionExpression whitelist. Caught only by a raw JSON-RPC round trip — all 8 unit tests passed
Enriching embeddings with rootCause degraded retrieval
A/B'd it rather than swapping. Score band narrowed instead of widening, and a previously-clean query flipped to the wrong top result. Kept symptom-only
traceparent not crossing the Python → Node boundary
NodeTracerProvider.register() already installs the AsyncLocalStorage context manager and W3C propagator globally — so no HTTP auto-instrumentation package was needed. Just propagation.extract() + context.with() in the POST handler
pnpm 10.30.1 forwards a literal -- into script argv instead of stripping it
pnpm run verify -- "text" silently searched for the string "--" and returned plausible results. Added a guard
httpx2 and mcp_types appeared as unrecognized transitive deps of mcp==2.0.0
Stopped and verified both against PyPI metadata before building. Both legitimate — real successor packages from the official orgs with coherent release histories



Engineering Decisions

DynamoDB vector search over OpenSearch Serverless. The old pattern was DynamoDB → zero-ETL → OpenSearch: two services, two APIs, and a sync pipeline to operate. Beyond the operational cost, OpenSearch Serverless carries a substantial always-on baseline. For a corpus this size that's absurd. One table, no pipeline.
Streamable HTTP over stdio. The single most consequential decision. Python agent, Node server — trace context propagates over HTTP headers and doesn't over stdio without custom work. Getting this wrong would have been discovered three days later as a broken waterfall.
No AWS SDK inside  opentel-mcp . The library is vendor-neutral OpenTelemetry instrumentation. Pulling  @aws-sdk/client-dynamodb  into it would make it heavier for every user who doesn't want AWS and couple a neutral library to one cloud. All AWS code lives in the application.
Real incidents as seed data. With synthetic rows you're squinting at cosine scores with no ground truth. With incidents you actually debugged, you know immediately whether a ranking is sensible.
Embedding provider behind an interface, before I needed it. Written in Phase 1 as good hygiene. When Bedrock blocked, the swap was one file and the table never changed — same 1024 dimensions, same COSINE index.



Security

IAM — DynamoDB access via IAM user credentials. Least privilege is not satisfied here: the user carries  AdministratorAccess , which is fine for a personal sandbox and wrong for anything real. A production version scopes to specific table ARNs and the specific actions used.
Secrets management — API keys in a gitignored  .env , loaded via  --env-file . Verified with  git check-ignore -v .env  before the first commit. Keys are never logged, and the codebase never echoes raw provider responses.
Input validation — Tool inputs are Zod-validated at the MCP boundary.  record_incident  is the only write path.
Data exposure — The embedding leak above was a genuine data-shape bug: internal vectors flowing into agent context. The fix is an explicit projection whitelist, and the lesson generalizes — assert on what a response doesn't contain, not just what it does.
Encryption — DynamoDB encrypts at rest by default; all API traffic is TLS.



Performance

Latency breakdown (from a real end-to-end trace, 16 spans):
  •  search_similar_incidents , end to end: ~2.9s
  •  get_runbook : ~219ms
  •  check_service_health : ~209ms
The 2.9s is not DynamoDB. Two things dominate it:
Embedding generation. The OpenAI call to embed the query text is the single largest component. Vector search itself is fast; getting the vector is not.
Stateless transport overhead. The server creates a fresh MCP server and transport per HTTP request, which means  initialize  +  tools/list  +  tools/call  on every single tool invocation — three round trips where you'd expect one. This is the real, rarely-discussed cost of stateless streamable HTTP, and it's clearly visible in the trace.
Optimizations not yet applied — caching embeddings for repeated queries; a session-reusing client to collapse the three round trips into one; parallel tool calls where the agent requests several independent ones.
Scalability — DynamoDB vector indexes are serverless and scale horizontally with no storage ceiling, so the data layer is not the constraint. The agent's sequential tool loop is.
Cold starts — Not applicable; nothing runs on Lambda in this build.



Cost Analysis

Service
Estimated Monthly Cost
DynamoDB (table storage, 8 items)
Effectively $0 — well within the always-free 25 GB allowance
DynamoDB vector index + queries
On-demand, pay-per-request. Verify current rates on the DynamoDB pricing page — vector search is new and I have not run it long enough for a meaningful Cost Explorer figure
OpenAI embeddings
~$0.000003 to seed all 8 incidents (143 tokens). Negligible at this scale
OpenAI agent inference
~$0.00053 per full diagnosis run (2,895 in / 159 out tokens across 3 model calls)
Jaeger
$0 — local Docker



What I Learned

Technical Learnings

Raw output inspection catches what unit tests structurally cannot. Eight passing tests missed a 1024-float vector leaking into every response, because tests assert on fields they care about. They check presence, never absence. One raw JSON-RPC round trip found it immediately.
A/B your retrieval changes, don't swap them. Adding  rootCause  to the embedding input felt obviously beneficial. It measurably degraded a query that previously worked. Root causes describe mechanisms, and mechanisms are exactly what differ between incidents that present identically — the wrong thing to embed for "what looks like this?"
Small corpora produce flat score distributions. With 8 heterogeneous incidents, every result landed in a 0.55–0.83 band. That isn't a ranking, it's noise with decimals. I initially blamed the embedding model; the data doesn't support that conclusion, and I wrote it up as unresolved rather than as a finding.

AI Learnings

Models need to be told what a distance score means. Without an explicit instruction that lower is better and that above ~0.5 is weak, the model treats a poor match as an answer.
Bounded loops need a structural stop, not a soft one. Telling a model to stop calling tools is a suggestion. Routing to a node with no tools bound is a guarantee.
Agents don't always retry when you want them to. I tried five alert phrasings hoping to provoke natural retry behaviour.  gpt-4o-mini  either trusted the top result or gave up after one search. Agent behaviour is much harder to induce on demand than it is to observe in the wild.

Architecture Learnings

Instrumentation lives on one side of a boundary, and that determines what it can see.  opentel-mcp  runs in the MCP server. The agent's token spend happens in the agent. No amount of server-side instrumentation reaches it.
Statelessness has an observable cost. Three round trips per tool call is invisible in code and obvious in a trace.

Things That Surprised Me

The demo found two bugs in the library it was built to demonstrate.
Finding 1 — thrash detection cannot fire on stateless HTTP transports.  opentel-mcp 's v0.6 agent-thrash detection counts repeated failures with matching fingerprints. But  ThrashDetector  is instantiated inside  instrumentMcpServer() , and a stateless per-request server calls that on every HTTP POST. The counter resets every call. I probed it directly with five back-to-back identical-fingerprint failures and got zero detections. This isn't "my demo didn't trigger it" — it's an entire transport class where a headline feature is silently inert. Filed as an issue against my own repo.
Finding 2 — cost attribution can't see agent-side spend. The library extracts token usage from MCP tool results, which covers tools that wrap LLM calls themselves. It does not cover the increasingly common case where the agent is the thing burning tokens. Measured concretely: trace-reported agent LLM cost $0, actual billed cost $0.000530. A 100% blind spot. Not a bug — a scope boundary — but one that needed documenting, with a downstream consequence: thrash detection's  wasted_cost_usd  will undercount, because retries are cheap server-side and expensive agent-side.
A third, smaller gap: the default pricing table covers 15+ chat models across five providers and no embedding models at all, so embedding spend is unpriceable even when usage data is present.
And the coincidence I couldn't have scripted: Jaeger's OTLP ports were already held by a running SigNoz ingester container — the same component whose "everything green, nothing listening" bug is incident #1 in the seed corpus. The system built to find similar past incidents was blocked at startup by the exact incident sitting inside its own database.



Future Improvements

  • Fix the thrash detector lifecycle in  opentel-mcp  so detector state outlives a single  instrumentMcpServer()  call, and document the current limitation in the README until it ships
  • Add embedding models to the default pricing table
  • Instrument the agent's LLM calls so token spend is visible in the same trace as tool execution
  • Session-reusing MCP client to collapse three round trips per tool call into one
  • Switch back to Bedrock Titan V2 once account verification clears — one file behind the provider interface
  • Grow the corpus to a few hundred incidents and re-run the retrieval experiments; the flat score distribution finding needs more data to resolve
  • Auth on the MCP endpoint and least-privilege IAM before any non-local deployment
  • Cache embeddings for repeated query text



Conclusion

I built an incident retrieval agent on DynamoDB's native vector search two days after it launched, with an MCP tool layer, a LangGraph agent, and end-to-end distributed tracing across a Python/Node boundary.
The DynamoDB piece delivered on its promise: embeddings and operational data on one item, inline filtering inside the vector query, no second database and no sync pipeline. For this class of problem that's a genuine simplification, not just a convenience.
But the thing I'll actually carry forward is that building a demo for my own library proved one of its features doesn't work under a configuration I use myself. That only happened because the demo was a real system with real constraints rather than a happy path I designed to succeed. If you maintain a library, the most useful thing you can do is build something with it that's allowed to disagree with you.
The repository has all five phases as separate commits, three reproducible failure scenarios, and both library findings documented honestly — including the one where the feature didn't fire.

Resources

  • Architecture diagram: see the Architecture section above (Mermaid, renders on GitHub)

Try It Yourself

Prerequisites

  • Node.js 22 (via nvm), pnpm via corepack
  • Python 3.12
  • Docker (for Jaeger)
  • An AWS account with DynamoDB access
  • An OpenAI API key

1. Clone and install

git clone https://github.com/Thirumalaiboobathi/incident-memory-copilot.git
cd incident-memory-copilot
pnpm install


2. Configure environment

cp .env.example .env
# Fill in: AWS_REGION, TABLE_NAME, OPENAI_API_KEY
git check-ignore -v .env # confirm it's ignored before committing anything


3. Create the table and vector index

pnpm run create-table

Verify the index is  ACTIVE  via the SDK or console — not the CLI, which may omit  VectorIndexes  on older versions.

4. Seed the corpus

pnpm run seed


5. Verify retrieval

pnpm run verify "data disappeared after a deploy"
pnpm run verify "health check passes but the feature doesn't work" status=open

Note: use  pnpm run verify "text"  with no  -- . pnpm forwards a literal  --  into argv.

6. Start Jaeger and the MCP server

docker compose up -d
ss -tlnp | grep -E '4319|4320|16686' # confirm ports actually bound
pnpm run mcp:start


7. Set up and run the agent

cd agent
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env # fill in OPENAI_API_KEY

python main.py "Collector started fine, logs look healthy, but nothing is arriving downstream"


8. Run the failure scenarios

agent/.venv/bin/python demos/scenario1_silent_failure.py
agent/.venv/bin/python demos/scenario2_agent_thrash.py
agent/.venv/bin/python demos/scenario3_cost_gap.py

Each prints a trace ID. Paste it into Jaeger's "Lookup by Trace ID" to inspect the spans.

Documentation

Available in the repository:
  •  README.md  — project overview, architecture, and setup
  •  CLAUDE.md  — architecture decisions, phase log, and the constraints that shaped the build, including why retrieval imperfection is intentional
  •  demos/README.md  — what each failure scenario demonstrates, and what did and didn't fire
  • Commit history — five phase commits ( a393ef6 ,  7d32129 ,  d89c227 ,  2a85485 ,  aa8893d ), each self-contained with tests passing
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article