SourceSignal AI: Building an Evidence-Gated Research Agent with Strands Decider 2B
SourceSignal AI is a research agent that uses Strands Decider 2B to decide when to retrieve more evidence, answer a question, investigate uncertainty, or abstain. This article explores its architecture, evaluation results, and deployment on Amazon Bedrock AgentCore.
SourceSignal AI: Building an Evidence-Gated Research Agent
Introduction
Large language models can produce fluent answers even when the available evidence is incomplete or contradictory. For research workflows, a confident but unsupported answer can be more harmful than no answer at all.
SourceSignal AI addresses this problem with an evidence-gated research workflow. Instead of treating every question as something that must be answered immediately, the agent evaluates whether the available evidence supports an answer, retrieves additional information when necessary, and abstains when the evidence remains insufficient.
The project was built for the Strands Decider 2B hackathon using the AWS User Group Madurai hackathon kit.
The Core Idea
SourceSignal AI separates evidence gathering from the decision to answer.
For each question, the agent follows a controlled workflow:
- Retrieve relevant records from its evidence corpus.
- Evaluate evidence sufficiency and potential contradictions using Strands Decider 2B.
- Retrieve more evidence when the first attempt is insufficient.
- Answer with source citations when the evidence supports the response.
- Abstain when the evidence remains insufficient after the permitted retrieval attempts.
The workflow uses the reusable
#policy pattern to make evidence-based decisions explicit and testable.Architecture and Technology
The implementation combines:
- Strands Agents for the agent workflow and tool integration.
- Strands Decider 2B for evidence-policy decisions.
- Amazon Bedrock for generative responses.
- Amazon Bedrock AgentCore Runtime for deploying and invoking the agent.
- A labelled evidence corpus with source identifiers and metadata.
- An evaluation suite containing 26 labelled cases, split into 13 tuning cases and 13 held-out test cases.
The agent includes retrieval tools for querying evidence and fetching individual source records. Retrieval is capped at two attempts to prevent unbounded searching.
Example: Answering with Evidence
During a live deployment test, we asked:
“How does UltraAttention-v2 reduce active KV-cache GPU memory usage?”
The agent retrieved source
SRC-101 , titled UltraAttention-v2 Architecture and Scaling Benchmark. The record states that the system uses structured block-sparse matrix kernels and reports a 68% reduction in active KV-cache GPU memory footprint compared with standard flash-attention baselines.The agent returned the answer with the citation
[SRC-101] and selected the answer action after one retrieval attempt.This example demonstrates the intended behavior: a supported claim is returned with traceable evidence rather than as an uncited assertion.
Example: Knowing When Not to Answer
We also tested a question about a return policy using
SRC-101 . That record concerns model architecture and performance, not product returns.The agent attempted retrieval twice and then returned
ABSTAIN (Insufficient Evidence) .This is an important distinction: abstention is not necessarily a failure. When a source does not support the question, refusing to invent an answer is the correct outcome.
Evaluation
The held-out evaluation contained 13 cases, producing 39 recorded decisions. The measured comparison was:
Metric | Strands Decider 2B workflow | LLM-only baseline |
|---|---|---|
Decision accuracy | 79.5% | 76.9% |
Median latency | 3.684 seconds | 22.355 seconds |
95th-percentile latency | 4.224 seconds | 49.805 seconds |
On this test split, the Decider workflow achieved a modest improvement in decision accuracy and approximately 6.1 times lower median latency than the baseline.
These results apply to the current evaluation suite; they should not be interpreted as a guarantee of the same performance on every research workload. The project also records confidence and evidence-policy decisions to support further evaluation.
Deployment on AWS
SourceSignal AI was successfully deployed to Amazon Bedrock AgentCore Runtime in the
us-east-1 region.The live invocation returned the agent response, policy decisions, model identifier, retrieval count, and latency. The runtime test confirmed that the deployed system could both abstain when evidence was insufficient and answer a supported question with a source citation.
Limitations and Future Work
The current implementation uses a labelled local evidence corpus, so retrieval quality depends on the coverage and quality of those records. The evaluation dataset is also relatively small.
Potential improvements include integrating larger trusted document collections, expanding contradiction-focused test cases, measuring citation precision, and evaluating the system against more diverse research questions.
Conclusion
SourceSignal AI demonstrates a practical approach to evidence-aware research agents: retrieve, evaluate, answer when supported, and abstain when necessary. Combining Strands Decider 2B with Strands Agents and Amazon Bedrock AgentCore makes the decision process observable and deployable.
The key principle is simple: a trustworthy research agent must know not only how to answer, but also when the available evidence is not enough.
Built with the AWS User Group Madurai Strands Decider hackathon kit.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article