The coaching agent that can't fake a citation (part 2 of 2)
This part builds the agents that read from that substrate, and the five mechanisms that stop them from inventing a citation.
Series: Agentic Graph for Media Industry (4 articles)
- 2The coaching agent that can't fake a citation (part 2 of 2) This article
This is part 2 of a two-part series. Part 1 turned a book into a citable graph on AWS: a repaired Markdown split by DoCO role, a chunk per book division, a Method_Model that keeps only what quotes the book word for word, and an MCP server that exposes 21 tools, 7 resources and 4 prompts against it. Two vocabulary points from part 1 carry into this one: a chunk is a division of the book (a chapter, a section) named by the table of contents, and a statement is a sentence the extraction model has written about the book, useful for search, never citable. This part builds the agents that read from that substrate, and the five mechanisms that stop them from inventing a citation.
How the agents are built
There are five families of agents, not five agents:
assess , coach , writer , one agent per persona the book describes, and three generic reviewers. On a book with three personas, that gives nine actual agents. All families come out of a single function, build_agent , that takes the book's name, a few booleans, and returns a Agent from Strands ready to run. The booleans are what distinguishes one family from another: enforce_grounding decides whether the verification chain applies, retrieval whether the agent receives tools, allow_work_writes whether it can modify your files.A second function,
agent_wiring , decides the composition without building an agent. It makes review possible, and I return to it at the end of this chapter.What a layer is
A Strands agent is built from four lists: a system prompt, tools, plugins and hooks . What I call a layer is an entry into one of those lists.
agent_wiring fills them and returns the argument dictionary that Agent will receive as is:Python
return AgentWiring(kwargs={"system_prompt": system_prompt,"tools": [*(AGENT_TOOLS if retrieval else []),*(WORK_WRITE_TOOLS if allow_work_writes else []),],"plugins": plugins,"hooks": [hook],"callback_handler": None,},skills=existing,)
Six layers are possible, and no agent has all six. The seventh row in the table is not a layer I place: it is the SDK default, and I name it because what follows depends on it.
Layer | Strands class | Present when |
|---|---|---|
Skill | AgentSkills | A SKILL.md has been rendered for this book |
Steering | Subclassed LLMSteeringHandler | By default, unless disabled |
Verification | GoalLoop | enforce_grounding=True |
Memory | MemoryManager | A reader identifier is supplied |
Tools | The book and document registries | retrieval=True |
Evidence ledger | Homegrown HookProvider | Always |
(SDK default) | SlidingWindowConversationManager | Always, because none is passed |
Steering: surveillance before a tool call
Steering is an SDK plugin. Each time the agent is about to call a tool, it submits the situation to a model with its own system prompt, which can wave it through or return an instruction the agent reads before acting. It is therefore preventive, where the output checks are corrective: they judge a finished answer.
Ours is deliberately narrow: it does not judge the quality of a finding, only whether it is sourced.
You keep an assessment honest about its sources. You judge only whether a claim is sourced, never whether it is good. […] Do not intervene on the substance of the assessment. The craft judgement belongs to the author and to the book, not to you. […] Never chooseinterrupt. This runs unattended, so there is no human to answer;guideis how you say something is wrong.
The last sentence is asked for in the prompt and enforced in the code, because a prompt is not a guarantee. The SDK offers three actions before a tool call :
Proceed , which waves through and is the nominal path, Guide , which cancels the call and reinjects an instruction, and Interrupt , which suspends for human intervention. A subclass demotes the third into the second:Python
class NonInteractiveSteering(LLMSteeringHandler):async def steer_before_tool(self, **kwargs: Any) -> Any:action = await super().steer_before_tool(**kwargs)if isinstance(action, Interrupt):return Guide(reason=action.reason)return action
This subclass is required for two reasons. First, there is no human behind an MCP tool call. Second, an active interrupt makes the
GoalLoop retry fail with a type error, because the invocation-end event carries a text string where the interrupt state expects a list of blocks.Memory: distilled facts, not preserved turns
There is no Strands session management here, and the execution path does not need one: an MCP tool builds the agent, invokes it once, clears its memory and returns a document. The day coaching becomes conversational, this is the first mechanism to wire in.
What survives across executions is something else: distilled facts, through the
MemoryManager plugin. The store deliberately omits add_messages (the method that would write the raw turns) to force the SDK to distil the conversation into facts with a model before writing. No memory search tool is exposed ( search_tool_config=False ), because an additional retrieval tool would compete with the citation discipline.The skill: a retrieval manual, not a copy of the book
The obvious design is to inline the method into the skill. I tried it: inlining every element cost 93,000 characters and prevented assessment from completing. A
references directory containing the same material was written, then never read.The
SKILL.md is therefore generated by render_skill , from the Method_Model, at the end of ingestion. It contains five things: a frontmatter, a tools table, an order of operations, a retrieval discipline, and a self-detection table. It contains no book content, only the names of the method steps.The frontmatter is the part that acts on the SDK:
YAML
---name: charpente-frdescription: How to interrogate <method> — the tools, the order, and the retrieval discipline.Activate when assessing, reviewing or coaching a work against this book.allowed-tools: book_search book_semantic_search book_passages book_passage book_chapters book_statusmetadata:book: charpente_frbook_id: charpente.fr---
allowed-tools names the six book tools, excluding book_ask and the tools for your document. Beware of what this field actually does: the documentation marks it as experimental and not enforced at runtime. The field is a human-readable declaration of intent, not a restriction. If you rely on it as an access control, you have the wrong mechanism.What enters the system prompt is a block listing the available skills with, for each, its name, its
description and the path of its SKILL.md . The body of the file is not there. The description therefore says when to activate the skill, and the agent will read the rest if it decides to use it. The block is refreshed before every invocation.The last part of the file is the most reusable. It lists concrete signals the agent can spot in its own draft ("I just wrote a sentence about the book without having called a tool") and the corrective action to take. These are review rules that a model knows how to apply to its own text, whereas an abstract instruction dilutes fast.
Signal in your draft | Corrective action |
|---|---|
A claim about the book with no retrieval behind it | retrieve it, or remove the claim |
A quotation you have not read in a passage | retrieve the passage, or drop the quotation marks |
A chapter title not obtained from book_chapters | list the chapters, use the real title |
Advice that would fit any book | it is generic, anchor it here or cut it |
"typically" or "generally" about the book | the book says it or it does not |
The five families, layer by layer
| assess | coach | writer | persona | generic |
|---|---|---|---|---|---|
Skill | the book's | the book's | none | none | none |
Steering | yes | yes | no | no | no |
GoalLoop | 2 attempts | 2 attempts | none | none | none |
Memory | if reader known | if reader known | if reader known | none | none |
Tools | book + document read | same | same + write | none | none |
Verification applied | yes | yes | no | no | no |
Returns | the assessment | the revisions | the prose | a reaction | a finding |
The
assess system prompt carries ten non-negotiable rules (not to be confused with the SOP: a SOP describes the steps to follow, these rules impose what the agent is not allowed to do). The coach has seven, the writer nine, the persona seven. Two of the ten assess rules are why a citation is a citation:You MUST NOT state what the book requires, recommends or forbids without having retrieved it in this conversation. Recall is not retrieval, and your training data is not this book.WHERE you put the book's words in quotation marks, they MUST be the book's words exactly, character for character. […] Do not shorten a sentence by ending it early at a comma. WHERE you drop anything from the middle, mark the gap with[...]. A tidied quotation is a misquotation.
coach receives a book-specific prompt, because the SOP generated for that book forms its opening. It does not receive your document a second time: the prompt carries a reference to the manifest, while the document and the assessment still enter the evidence ledger, since it is against the ledger that a citation is verified.When the SOP contains more than five steps (which is the case for any serious book), coaching is delegated to sub-agents. An orchestrator written in code (a loop, not a model) cuts the worklist into slices of five and instantiates a fresh
Agent per slice. Each sub-agent does its own retrievals, produces its revisions, and shares nothing with the others. Parallelism falls out naturally, three sub-agents run at once. On a book with 51 steps, 11 sub-agents produce 269,000 characters of coaching in ~20 minutes. A single agent, on the other hand, hits the model's output ceiling at 37,000 characters and does not finish. This is the sub-agent delegation pattern, tracked in the issue strands-agents/harness-sdk#911 . The SDK already provides Graph and Swarm for multi-agent. Graph runs the ready nodes of one wave in parallel but fixes its topology at construction. Swarm chains sequential handoffs. Here, the number of sub-agents equals the number of worklist slices. It is decided at runtime from the book, so the code orchestrator stays a fit.
Figure 4: sub-agent delegation. The orchestrator, a loop in code, cuts the worklist into slices of five steps and instantiates a fresh-context agent per slice. The agents run in parallel, share nothing, and their outputs are concatenated.
writer is the only agent allowed to write, and the only one whose verification is disabled. Writing a scene is not a claim about the book, and a citation guardrail would reject a good scene. Its prompt opens by naming the relationship ("You are the hand, not the head") and its second rule protects your structure:The heading is not yours. You rewrite what is under it […] do not renumber, retitle or re-level anything the author named.
Personas and generic reviewers: verifiable isolation
No persona agent definition exists in the code. Extraction produces the list of audiences the book describes, and a persona agent is a function call that fills a prompt template from one of those audiences. A book that describes no audience produces no persona. The first point of the prompt is what makes it useful:
React. Do not assess. "This violates the rule about X" is not something you would ever say.
Six further instructions follow: say where you dropped off and at which passage, say what it would take to say yes, stay in character, write in the language of the work, in the first person.
Three
generic reviewers sit next to the personas (structure, clarity, audience fit) with static prompts and no dependency on the Method_Model. They therefore work on a book that was never distilled.Both kinds of reviewer receive
retrieval=False , which has two effects: the tool list is empty, and build_agent does not bind the graph. The book is therefore not just kept quiet in a persona's context. There is no tool through which to reach it. No prompt instruction carries the load, and this is verifiable by reading six lines rather than rereading a prompt. These reviewers are also not subject to the check chain: a reaction or a craft observation claims nothing about the book, and submitting them to it would make them fail for not having cited a book they are forbidden to read.How to prove the agent really reads the book
Here is the measurement that made me build everything that follows. On an earlier version of the agent, one answer contained fourteen citations attributed to the book. Seven were absent from it. The agent had written them from memory, and the evidence ledger, the list of what it had actually retrieved, still claimed to hold all fourteen. The verifier, that is the code that confronts the citations in the answer with that ledger, therefore validated them. I call this measurement the seven-out-of-fourteen run.
The cause was not the model but the meeting of two mechanisms. Figure 5 shows them together: the ledger on the left, the check chain in the middle, the retry path on the right
Figure 5: how a citation is verified before the answer leaves the agent. Three columns in English: the evidence ledger that classifies each tool result and pins the message that carries it, the five checks chained from cheapest to most expensive, and the retry path that reruns the agent at most twice when a check fails.
The two mechanisms that collide
The first is the conversation manager. I pass none, and that is the point: every agent therefore inherits the SDK default, a
SlidingWindowConversationManager whose window is 40 messages. It intervenes through two paths. Routinely, at the end of every invocation, if the history exceeds the window, it calls reduce_context and trims the oldest messages. And when the model returns a ContextWindowOverflowException , the agent calls the same reduce_context passing it the exception. On that reactive path, the manager first tries to truncate the oldest tool results, and only cuts messages otherwise.The oldest tool results: these are precisely the passages the agent retrieved at the start of its work. The mechanism meant to save the invocation is therefore exactly the one that removes the evidence.
The second is the evidence ledger, an instance of our
GroundingHook class. Concretely, two Python lists in memory for the duration of a call: passages for what comes from the book, given for what comes from your document. It is registered as a hook provider with the agent and subscribes to two SDK events:Python
def register_hooks(self, registry: Any, **_kwargs: Any) -> None:from strands.hooks import AfterToolCallEvent, MessageAddedEventregistry.add_callback(AfterToolCallEvent, self._record)registry.add_callback(MessageAddedEvent, self._pin_evidence)
AfterToolCallEvent fires after every tool call: _record classifies the result into one of the two channels. The hook documentation is in the Strands Agents SDK guide .The collision is here. The
SlidingWindowConversationManager drops a tool result message from the conversation, but the ledger keeps its copy of the text. The ledger then vouches for a passage the model no longer sees. The model keeps writing, cites the book from memory, and the verifier accepts the citation because its ledger says it arrived through a tool.Mechanism 3: the fix, pin the evidence
_pin_evidence is the other subscription. On every message added, it looks at whether that message carries a tool result the ledger has recorded, and if so it pins it:Python
from strands.agent.conversation_manager.compression.pin_message import pin_messagepin_message([message], 0)
The
SlidingWindowConversationManager accepts a pin_first parameter, which protects the first messages in the window. It does not cover evidence that arrives mid-conversation, after the tool calls. pin_message fills that role, and it is the primitive that the SDK's own test suite uses to pin a message in the middle of the history. The import path goes through the submodule because the compression package does not expose a public alias. pin_message sets a flag in the message metadata, and reduce_context drops only unpinned messages. Trimming can therefore let any conversation turn go, and it keeps the pinned evidence.The general principle, if you take only one line from this chapter: the component that decides "this is evidence" must be the one that protects it.
Two channels, because reading your document is not reading the book
The ledger holds two separate lists:
passages for the graph tool results, given for what the agent read of your document. Both are citable, only one counts as evidence, passages answers retrieved_anything . Citing yourself can therefore never satisfy the check. A generic file-reading tool once figured among the evidence tools, and because the ledger indexes on the tool name, reading any file satisfied "the agent consulted the book". This separation closes the hole.Mechanism 4: a verifier that costs nothing until it must
Verification is carried by
GoalLoop , a plugin supplied by Strands: it submits the agent's answer to a goal, and if the goal is not met, it injects feedback and makes the agent try again, up to a cap on attempts.A goal can be expressed in two ways: a string, which the SDK has judged by a model, or a Python function (a callable) that renders the verdict itself. Here it is a function. No judge agent is built, no token is spent, and verification is therefore free. It chains five checks in order, from cheapest to most expensive, so that an answer that retrieved nothing costs no guardrail call.
- The answer is not empty.
- Something arrived through the evidence tools. Reading your document does not count.
- At least two chapter attributions are present.
- Every citation credited to the book appears word for word in the retrieved material, after typographic normalisation.
- The Amazon Bedrock Guardrails contextual-grounding check agrees.
Check 3 is deliberately coarse: it counts occurrences of the word "chapter" and of the real titles returned by
book_chapters , with a length filter that excludes three-letter titles liable to be found inside ordinary words.Check 4 is the one that decides that a citation is a citation. It judges nothing, it compares: each citation credited to the book must appear word for word in the retrieved material, after typographic normalisation. Two details matter. The comparison runs before any paid call, and it reads the untruncated material. Having run it against the truncated copy sent to the guardrail flagged fourteen citations out of nineteen as missing when they were in fact present.
The fifth check in depth: Amazon Bedrock Guardrails
The first four checks are our own code and cost nothing. The fifth asks a managed service for a second opinion: Amazon Bedrock Guardrails' contextual grounding check , which scores how well a text is supported by a provided source. The call is made with
ApplyGuardrail and takes three content blocks: the grounding source (qualified grounding_source ), the question (qualified query ), and the text to evaluate with no qualifier. The last block is the one that gets forgotten, because attached to a Converse call the guardrail would see the model's response for free.This check comes with two scoping caveats. The docs cover summarisation, paraphrasing and question-answering, but exclude conversational: an assessment triggered by a single tool call falls on the right side, coaching that has become a dialogue would no longer fit. And the guardrail merges all
grounding_source values before evaluating them, which erases the notion of "which division" by construction and mechanically prevents it from catching a chapter misattribution.Mechanism 5: citations the model does not compose
The four preceding checks work after the fact, on prose: the citation is a string the model has written that has to be found in the material, and its chapter attribution is read from the words around it. Check 4 catches the invented citation, but not the misattribution. A passage from chapter 3 credited to chapter 7 passes. Mechanism 5 changes the nature of the guarantee: the citation text no longer comes from the model, it becomes structurally incapable of it.
The flow runs in four steps:
- We send the Method_Model excerpts to Bedrock as a citable document. A
DocumentBlockfrom the Converse API acceptscitations: {"enabled": true}and a source whosecontentmember is a list of text blocks (DocumentContentBlock[]). Each block is an independent citable unit. Our excerpts are already verified word for word against the division they name and carry its identifier and chapter title.
- The model answers with
CitationsContentBlockattached to the passages it produces. Each citation carries two decisive fields:sourceContent[].text(a verbatim slice of the block sent, not a string composed by the model) andlocation.documentChunk.start(the index of the source block in the sent list).
- The citation resolves by index, no longer by text search. Since the source block is identified by an index and the quoted text is a slice of it, verification becomes an array lookup: read
documentChunk.start, retrieve the matching Method_Model excerpt, and read its chapter title. A wrong chapter is structurally impossible.
- An attribution filter decides which citations to submit. Verified agents legitimately put their own words in quotation marks (a need they articulate, a line they propose). A citation counts as a claim about the book only if an attribution cue ("according to", "the book", "chapter", the author's name, and about fifteen others) appears within the 120 characters that precede it. Without this filter, the agent is punished for doing its job.
This flow is not a homegrown construction. The Converse API citations are the feature Bedrock provides for verifiable attribution. The work specific to this project fits in one decision: send as citable blocks only excerpts already verified word for word.
Two artefacts survive the process. The delivered document carries a "Verification note" section that names in plain text every unverified citation, and the
results.json evaluation record tags rejected citations invented or not retrieved. The evidence ledger itself writes nothing. It lives in memory and dies with the call. A rejection is costly, it destroys a finished answer, so the log line is what distinguishes a justified rejection from an unjustified one.The retry budget, and a timeout that cannot help
The loop allows two attempts, not three. Each retry appends the previous answer and the feedback to the conversation, which grows on every turn. A measured run degraded exactly at the third: seven unverified citations out of twenty-nine, then one out of twenty-six, then a collapse to 1,252 characters.
max_attempts=2 is measured, not chosen.GoalLoop also accepts a timeout, but it checks it in AfterInvocationEvent , so after the model's work is finished and billed. A timeout at that point cannot cancel an in-flight invocation, only skip verification of an already-paid response. It is max_attempts that really bounds the loop. And GoalLoop returns its last attempt rather than its best one, so the final selection happens outside the loop over the preserved verdicts, with a rule that ranks "has at least one attribution" above every citation count. Otherwise a collapsed retry that asks for the document back (zero unverified citations) would beat a real assessment that has one.When the window loses anyway, nothing ships
Pinning protects the citations, but it can also make an overflow unrecoverable: if all the messages old enough to be dropped are pinned, the conversation manager has nothing left to cut and raises an exception. This is an accepted trade-off, and the failure is loud rather than silent. The agent raises
GroundingLostError with a message that says exactly why: "nothing was written : a document produced past this point quotes the book from the model's memory rather than from the passages, and reads exactly like one that does not".Measuring the agents: evaluations and optimiser
Two Strands packages carry this part and answer different questions. The evaluation SDK asks whether an answer is good. The Harness Optimizer asks whether a prompt is better.
Why evaluation cases come in pairs
Each evaluation case comes with a twin. The first carries a known violation of the method and tests that the agent finds it. The second is a compliant case and tests that the agent does not fabricate a finding. A suite made only of violations can never catch that second failure mode, which is the most costly one: an invented reproach on a correct text.
The optimiser proposes, a human applies
The optimiser wires up the real agent, the real evaluators and a
ContrastiveReflectionOptimizer that reads execution traces and proposes a rewrite of the prompts. The reward is the evaluation suite itself.Now comes the decision that matters most: the optimiser never writes into the sources. It writes a before, an after and an evaluation record into a timestamped directory.
Conclusion
The architecture turns a methodology book into a coaching agent, and five mechanisms defend the citation. A pinned evidence ledger prevents a context trim from letting the verifier vouch for a text the conversation dropped. One chunk per book division makes a citation name a place you can open. Word-for-word extraction or nothing keeps the unintelligible away from the model. Native Converse citations make a citation non-composable by the model and its attribution non-forgeable, because the quoted text is a slice of the block we supplied. And a two-stage output check runs an exact test for free before asking Amazon Bedrock Guardrails for a second opinion.
The failure that all five prevent is an unsourced answer that reads exactly like a sourced one. You cannot detect it on a read, and that is why it must be made impossible upstream. You get feedback on your own document with a citation you can open, and an auditable trail behind each one.
If you take only five engineering lessons from this architecture, take these. Pin the tool results of your evidence ledger, because a sliding window will otherwise let it vouch for text the conversation dropped. Register that ledger as a
HookProvider and not as a plugin, because the plugin registry indexes on a name it does not have. Bound a GoalLoop by attempts rather than by clock, since its timeout is evaluated after the model's work is billed. Let an optimiser propose rather than apply, because an evaluation suite that scores documents cannot catch a prompt quietly relaxed. And when an agent's output exceeds what the model can produce in one generation, delegate to sub-agents with a fresh context rather than raising the ceiling: the orchestration is code, the work is in the agent, and the parallelism falls out naturally.# agentic-ai# strands-agents# amazon-bedrock-agentcore# media-entertainment-industry# amazon-neptune
Series: Agentic Graph for Media Industry (4 articles)
- 2The coaching agent that can't fake a citation (part 2 of 2) This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article