AWS Builder Center

Builder Center

Build a Web Research Agent: Strands Agents and Self-Hosted SearXNG

Learn how to build a research agent whose search engine runs on your own machine — no search API key, no per-query cost.

Taimuri, Ali

By: Arjun Shakdher, Ali Taimuri
Most agent tutorials reach for a hosted search API, and for good reason: they are fast to wire up and the results come back clean. But search is the dependency an agent calls most. A single research question in the agent we are about to build runs 14 to 18 tool calls, and most of them are searches or page reads. That makes search the component whose cost, latency, and data handling you most often end up wanting control over.
In this tutorial we will build a conversational research agent using  Strands Agents  with  SearXNG  as the search layer — an open source meta-search engine you run as a container next to the agent. The queries never leave your machine for a third-party API, there is no key to manage, and the marginal search costs nothing.
By the end you will have an assistant that can:
  • Search the web through an instance you control, across engines you choose
  • Read the pages it finds and extract their text, respecting  robots.txt 
  • Write a cited Markdown report
  • Verify its own citations — every URL in the report is checked against what the tools actually returned, so a fabricated citation gets caught rather than published
That last one is the part worth stealing even if you use a different search provider.

What You'll Build

A single-agent system with three tools:
Tool
Does
 web_search 
Queries your SearXNG instance's JSON API
 fetch_page 
Retrieves one page and extracts readable text
 write_report 
Validates citations against a source ledger, then saves Markdown
The agent plans sub-questions, searches, reads, writes, and self-corrects when a citation does not check out. Amazon Bedrock provides the model.

Prerequisites

  • Python 3.12 or higher
  • A container runtime — this tutorial uses  Finch , but Docker works with the same commands
  • An AWS account with Amazon Bedrock model access in your Region, and credentials in your shell
  • No search API key. That is the point.

Step 1: Project Setup

Bash
mkdir searxng-research-agent
cd searxng-research-agent

python3.12 -m venv .venv
source .venv/bin/activate          # On Windows: .venv\Scripts\activate

cat > requirements.txt << 'EOF'
strands-agents>=1.30.0
httpx>=0.27
trafilatura>=2.0.0
EOF

pip install --upgrade pip
pip install -r requirements.txt
Three dependencies: the agent framework, an HTTP client, and  trafilatura  for pulling readable text out of HTML. No search SDK, because SearXNG speaks plain JSON over HTTP.

Step 2: Start the Search Layer

SearXNG keeps no index of its own. It forwards your query to other engines — Wikipedia, arXiv, PubMed, DuckDuckGo, GitHub and a couple of hundred more — parses what comes back, and merges it into one ranked list.
Create  settings.yml :
YAML
use_default_settings:
  engines:
    keep_only:
      - duckduckgo
      - brave
      - wikipedia
      - arxiv
      - pubmed
      - openalex
      - semantic scholar
      - stackoverflow
      - github
      - hackernews
      - mdn

server:
  secret_key: "set-via-SEARXNG_SECRET-env"
  bind_address: "0.0.0.0"
  port: 8080
  limiter: false
  public_instance: false

search:
  formats:
    - html
    - json
  safe_search: 0
  default_lang: "en"

outgoing:
  request_timeout: 6.0
  max_request_timeout: 12.0
Three settings in there decide whether this works at all:
  •  search.formats  must include  json . It does not by default, and without it the API returns 403.
  •  server.limiter: false . The limiter is bot protection: it inspects headers and request rhythm and rejects clients that do not look like browsers. An agent is not a browser. Turning it off is correct for an instance only your agent can reach, and wrong for anything exposed.
  •  use_default_settings.engines.keep_only  narrows the engine set. Without it your instance queries roughly 240 engines, most irrelevant to research and each one a site you are generating traffic against. This list goes from 242 to 11.
Now  compose.yaml :
YAML
services:
  searxng:
    image: docker.io/searxng/searxng:latest
    container_name: searxng-tutorial
    ports:
      - "127.0.0.1:8888:8080"
    volumes:
      - ./settings.yml:/etc/searxng/settings.yml:ro
    environment:
      SEARXNG_BASE_URL: http://localhost:8888/
      SEARXNG_SECRET: "${SEARXNG_SECRET:?generate one with openssl rand -hex 32}"
    restart: unless-stopped
Note the port mapping:  127.0.0.1:8888:8080 , not  8888:8080 . Your instance has no authentication of its own — no key, no token, no users — so the network is the only gate. A bare port mapping would publish an open search proxy on every interface your machine has.
Start it:
Bash
printf 'SEARXNG_SECRET=%s\n' "$(openssl rand -hex 32)" > .env
finch compose up -d

# Should return results, not 403
curl -s 'http://127.0.0.1:8888/search?q=searxng&format=json' | head -c 300
The secret goes in  .env  (add it to  .gitignore ) rather than into  compose.yaml , and the  :?  syntax makes compose refuse to start if it is unset — so no file you commit ever contains a usable secret.

Step 3: Building the Research Agent

Create  main.py . We will build it in four pieces.

The source ledger

This is the idea the whole agent is organized around:
Python
# Every URL the tools returned this session. An LLM judge is unreliable at
# verifying provenance; a set membership test is not.
SOURCE_LEDGER: set[str] = set()

def normalize(url: str) -> str:
    """Strip fragments and trailing slashes so citation matching is not brittle."""
    p = urlparse(url.strip())
    return urlunparse((p.scheme, p.netloc.lower(), p.path.rstrip("/") or "/", "", p.query, ""))
Every URL that search or fetch returns gets recorded. When the agent writes its report, every citation is checked against this set. A URL that no tool ever returned is a URL the model produced from somewhere else — and in a research report, "somewhere else" means fabricated.

The search tool

Python
@tool
def web_search(query: str, max_results: int = 8, time_range: str | None = None,
               engines: str | None = None) -> str:
    """Search the web through a self-hosted SearXNG instance.

    Returns ranked results as JSON: title, url, a short snippet, and the engine
    that supplied it. Snippets are short by design -- SearXNG returns what
    upstream engines show on a results page, roughly one or two sentences. Call
    fetch_page on a result when you need the actual content of it.

    Args:
        query: Natural language search query. Keep it specific; this is keyword
            search, not semantic search, so two narrow queries beat one broad one.
        max_results: Maximum results to return, 1-20. Defaults to 8.
        time_range: Restrict to recent content: 'day', 'week', 'month', 'year'.
        engines: Comma-separated engines to restrict to, e.g. 'arxiv,pubmed'.

    Returns:
        JSON with "results" and "engines_failed". An empty result list with
        entries in engines_failed means the engines refused this instance, not
        that the web holds no answer -- retry with different engines.
    """
    params = {"q": query, "format": "json", "language": "en"}
    if time_range:
        params["time_range"] = time_range
    if engines:
        params["engines"] = engines

    try:
        r = httpx.get(f"{SEARXNG_URL}/search", params=params, timeout=20,
                      headers={"User-Agent": USER_AGENT})
        r.raise_for_status()
        payload = r.json()
    except httpx.HTTPStatusError as e:
        hint = " (403 usually means limiter is on or json is not in search.formats)" \
            if e.response.status_code == 403 else ""
        return json.dumps({"error": f"search failed: {e}{hint}"})
    except httpx.HTTPError as e:
        return json.dumps({"error": f"search failed: {e}"})

    results = [
        {"title": i.get("title", ""), "url": i["url"],
         "snippet": (i.get("content") or "").strip(), "engine": i.get("engine", "")}
        for i in payload.get("results", [])[:max_results] if i.get("url")
    ]
    SOURCE_LEDGER.update(normalize(r["url"]) for r in results)

    # SearXNG reports refusals here rather than failing the request. Pass them
    # through: "blocked" and "nothing matches" need different next moves.
    failed = dict(payload.get("unresponsive_engines", []))
    return json.dumps({"query": query, "count": len(results),
                       "results": results, "engines_failed": failed})
Two things to notice. First, that docstring is the interface the model reads — it is written for the model, not for a maintainer, which is why it explains when to use the tool and what the parameters accept.
Second,  engines_failed . When an upstream engine CAPTCHAs or blocks your instance, SearXNG reports it in  unresponsive_engines  and returns an empty result list. If you drop that field, the model sees "0 results" and concludes the web has no answer — when in fact every engine refused it. Those two situations need opposite responses: one needs different engines, the other needs a different query.

The fetch tool

Python
@tool
def fetch_page(url: str, max_chars: int = 8000) -> str:
    """Retrieve one webpage and extract its readable text as Markdown.

    This is the step SearXNG does not do for you. Search gives you a one-line
    snippet; this gives you the body of the page. Call it on the two or three
    results most likely to answer the question before writing anything.

    The fetch is skipped when robots.txt disallows it, so expect a short error
    string rather than an exception for pages that cannot or should not be read.

    Args:
        url: Absolute http(s) URL, normally one returned by web_search.
        max_chars: Truncate extracted text to this many characters.

    Returns:
        JSON with "text" and "title", or {"error": ...} when the page was
        disallowed, unreachable, or held no extractable text.
    """
    if urlparse(url).scheme not in ("http", "https"):
        return json.dumps({"error": "url must be absolute http(s)", "url": url})

    rp = urllib.robotparser.RobotFileParser()
    origin = f"{urlparse(url).scheme}://{urlparse(url).netloc}"
    try:
        robots = httpx.get(f"{origin}/robots.txt", timeout=5, follow_redirects=True,
                           headers={"User-Agent": USER_AGENT})
        if robots.status_code < 400:
            rp.parse(robots.text.splitlines())
            if not rp.can_fetch(USER_AGENT, url):
                return json.dumps({"error": "disallowed by robots.txt", "url": url})
    except httpx.HTTPError:
        pass  # no reachable robots.txt means no stated restriction

    try:
        r = httpx.get(url, timeout=20, follow_redirects=True,
                      headers={"User-Agent": USER_AGENT})
        r.raise_for_status()
    except httpx.HTTPError as e:
        return json.dumps({"error": f"fetch failed: {e}", "url": url})

    text = trafilatura.extract(r.text, include_comments=False, include_tables=True) or ""
    if not text.strip():
        return json.dumps({"error": "no extractable text found", "url": url})

    SOURCE_LEDGER.add(normalize(url))
    meta = trafilatura.extract_metadata(r.text)
    return json.dumps({"url": url, "title": (getattr(meta, "title", "") or "").strip(),
                       "text": text[:max_chars], "chars": min(len(text), max_chars)})
This tool exists because SearXNG's snippets are short. A hosted search API often returns thousands of characters per result; SearXNG returns what an upstream results page shows, which in my measurements was a median of 164 characters from general web engines. That is enough to decide whether a page is worth reading and never enough to support a claim.
The  robots.txt  check and the honest user agent are not decoration. You are building something that reads other people's sites at machine speed; declining pages that ask not to be read is the baseline, and identifying your client means operators can tell you apart from an attack.

The report tool

Python
@tool
def write_report(title: str, markdown: str) -> str:
    """Save a finished research report, after checking its citations.

    Every http(s) URL in the Markdown is checked against the URLs that
    web_search and fetch_page actually returned this session. URLs that were
    never returned by a tool come back as unverified: fix them -- find a real
    source or remove the claim -- and call this tool again. An unverified URL in
    a research report is a fabricated citation.

    Args:
        title: Report title, used in the heading and the filename.
        markdown: Full report body in Markdown, with inline links to sources.

    Returns:
        JSON with "path", "citations", "verified", and any "unverified" URLs.
    """
    cited = {normalize(u) for u in re.findall(r"https?://[^\s\)\]\>\"']+", markdown)}
    unverified = sorted(cited - SOURCE_LEDGER)

    slug = re.sub(r"[^a-z0-9]+", "-", title.lower()).strip("-") or "report"
    path = OUTPUT_DIR / f"{slug}.md"
    path.parent.mkdir(parents=True, exist_ok=True)
    body = markdown if markdown.lstrip().startswith("#") else f"# {title}\n\n{markdown}"
    path.write_text(body, encoding="utf-8")

    return json.dumps({"path": str(path), "citations": len(cited),
                       "verified": len(cited) - len(unverified), "unverified": unverified})
One set difference, and the agent gets told which of its citations it invented. The feedback goes back into the loop, so the model fixes the claim rather than shipping it. No judge model, no extra API call, about thirty lines.

The prompt and the agent

Python
def create_research_agent() -> Agent:
    """Create the research agent."""
    model = BedrockModel(
        model_id=MODEL_ID,
        region_name=REGION,
        # The whole report is written as one tool call argument, so the output
        # cap has to fit a finished document rather than a chat reply.
        max_tokens=16000,
    )
    return Agent(
        model=model,
        tools=[web_search, fetch_page, write_report],
        system_prompt=SYSTEM_PROMPT,
        # A research episode runs 15-30 tool calls. The default sliding window
        # of 40 messages truncates episodes that long, sometimes between a tool
        # call and its result.
        conversation_manager=SlidingWindowConversationManager(window_size=200),
    )
Both comments there are scars.  max_tokens  is unset by default, which means the Bedrock default applies — and on my first full evaluation run that default ended seven of ten episodes in  MaxTokensReachedException , every one of them after the research was finished and paid for. The agent had searched, read eleven pages, and then failed on the last step. The conversation window is the same class of problem: 40 messages is fine for chat and too small for an episode that makes 18 tool calls.
The system prompt is where the method lives:
Plain text
1. Decompose the question into two to four specific sub-questions.
2. Run a web_search for each. Prefer several narrow queries over one broad one.
3. Fetch the most promising results. Read at least three pages before you write,
   from at least two different domains.
4. Write the report in Markdown: a two-sentence summary, sections per
   sub-question, then a "Limits" section stating what you could not establish.
5. Cite inline with Markdown links... Cite only URLs that web_search or
   fetch_page returned to you in this session.
6. Call write_report. If it reports unverified citations, fix them and call again.
The "Limits" section earns its place. It is what makes the agent say "I could not establish this" instead of filling the gap.

Step 4: Running Your Assistant

Bash
source .venv/bin/activate
export AWS_REGION=us-east-1
export SEARXNG_URL=http://127.0.0.1:8888

# Use a model your account can actually invoke -- resolve it rather than
# copying an ID from a blog post, including this one
aws bedrock list-inference-profiles --region us-east-1 \
  --query 'inferenceProfileSummaries[].inferenceProfileId' --output table
export BEDROCK_MODEL_ID='<id from that list>'

python main.py
A real session:
Plain text
SearXNG Web Research Assistant
==================================================
model: us.anthropic.claude-sonnet-5   search: http://127.0.0.1:8888
Ask a research question. Reports are saved to reports/.

You: How do open source meta-search engines handle rate limiting from upstream engines?

Tool #1: web_search
Tool #2: web_search
Tool #3: fetch_page
...
Tool #15: write_report

The report has been written and all 20 citations were verified.

[60 sources in the ledger this episode]
Fifteen tool calls, 20 citations, every one of them traceable to a page the agent actually retrieved. The report it saved opens like this:
Open-source meta-search engines such as SearXNG, Whoogle, and 4get sit between end users and commercial search engines, and because they aggregate many users' queries through a small number of outgoing IP addresses, they are routinely treated as bots themselves — triggering CAPTCHAs, 429 "Too Many Requests" responses, or outright IP bans from the upstream engines.
And its Limits section, unprompted:
Exact current numeric defaults for SearXNG's suspension timers vary across versions and even across SearXNG's own documentation pages, so the figures cited here should be treated as approximate/version-dependent rather than fixed constants.
That is the behavior the prompt and the citation check are for: noticing that the sources disagree and saying so, rather than picking a number.

What the Results Look Like

I ran a fixed set of ten research questions through this agent twice, on different days. Both runs completed all ten episodes, and citation validity was 1.00 in every single one — across both runs, every cited URL had been returned by a tool call in that same episode.
Measure
Run 1 (p50)
Run 2 (p50)
Episode wall time
79 s
92 s
Tool calls
14
18
Searches / pages read
4.5 / 7
6 / 8
Citations in report
9
10
Citation validity
1.00
1.00
Tokens per episode
140,000
170,000
Three things in there are worth more than the rest.
Reading dominates searching, 7 pages to 4.5 searches. That is the shape self-hosted search forces on you: 164-character snippets cannot support a claim, so the agent fetches. Budget for the extraction step — it is not optional.
Fetching fails about one time in six. Thirteen of 77 page fetches returned an error rather than text:  robots.txt  disallows, non-HTML responses, HTTP failures, pages with nothing extractable. Searching, by contrast, essentially never failed — one zero-result call out of 49. Your flaky dependency is the open web, not the search layer, so tell the agent to move to a different source instead of retrying.
Input tokens dominate output 21 to 1. Every search result and page stays in context so the model can cross-reference while writing: 132,000 input tokens against 6,200 output. Multiply by your own Bedrock rate; the ratio is the point, and it is what prompt caching aims at.

Which Engines Will Actually Answer You

Here is the thing nobody tells you about self-hosted meta-search: a fresh instance looks like automation to the engines it queries, because it is. Some answer happily, several return CAPTCHAs, some deny access outright — and which ones do what depends on your egress IP, so it is a measurement rather than a fact you can look up.
From a residential connection, on a fresh instance:
Engine
Results
Median snippet chars
semantic scholar
10
1,197
openalex
10
1,194
arxiv
10
1,190
pubmed
20
890
microsoft learn
10
346
brave
22
226
duckduckgo
10
207
mdn
10
150
github
30
124
stackoverflow
10
77
Refused outright: Google ( access denied ), Qwant ( access denied ). Returned nothing with no error at all: Wikipedia, Wikidata, PyPI.
The scholarly engines are the best search tools in the set — roughly an order of magnitude more text per result, faster, from sources that expect programmatic access. If your research domain has an open-access corpus, self-hosted search is not merely cheaper there, it is better. The general web engines are thin and interchangeable by comparison.
One warning worth repeating: the engines SearXNG queries have their own terms of service, and several restrict automated querying. Choosing your engine list is a decision with licence consequences. Prefer engines that publish an API or an open index, keep your volume proportionate, and identify your client honestly.

Troubleshooting

403 from the search API
Plain text
search failed: Client error '403 Forbidden' (403 usually means limiter is on
or json is not in search.formats)
Either  json  is missing from  search.formats  or  server.limiter  is still  true . Both are in  settings.yml ; restart the container after changing it.
Zero results, no error
Not a bug, and not an empty web. SearXNG suspends an engine after a 403 and then reports nothing — no error, no warning. Check  engines_failed  in the tool output, and try  engines="arxiv,pubmed"  to confirm the instance works at all.
Engine availability changes between runs
Suspensions are sticky. If you have been hammering an engine while debugging, it stays suspended for minutes to days, and any availability you measure afterwards is measuring your own history with it rather than reality. Restart the container for a clean reading.
 MaxTokensReachedException  after the research finished
Plain text
MaxTokensReachedException: Model stopped generating due to maximum token limit.
The report is written as one tool call argument, so it must fit inside a single model response. Set  max_tokens=16000  on  BedrockModel , as above.
An engine you listed is not enabled
 keep_only  filters; it cannot enable an engine upstream ships disabled. Mojeek and Startpage carry  inactive: true  because both front their results with a Proof-of-Work CAPTCHA, and neither  keep_only  nor  disabled: false  overrides that — it takes  inactive: false , which means deliberately overriding a flag upstream set as a warning.
 NoCredentialsError: Unable to locate credentials 
The agent reached Bedrock and your shell had no AWS credentials. Set your profile or region in the same shell you run  python main.py  from.

Next Steps

Host it. The agent is already shaped for  Amazon Bedrock AgentCore Runtime . Its container contract is short — port 8080, host  0.0.0.0 , an ARM64 image,  POST /invocations  and  GET /ping  — and the AgentCore SDK serves all of that, so the whole server becomes one decorated function:
Python
from bedrock_agentcore import BedrockAgentCoreApp

app = BedrockAgentCoreApp()

@app.entrypoint
def invoke(payload: dict) -> dict:
    question = (payload or {}).get("prompt", "").strip()
    SOURCE_LEDGER.clear()
    return {"report": str(create_research_agent()(question)),
            "sources": sorted(SOURCE_LEDGER)}
Returning the ledger alongside the report means whoever calls your agent can verify its citations without trusting it.
Put the search layer in a VPC. On a laptop, "my queries never leave my network" is a slogan. Running SearXNG on ECS Fargate in a private subnet, with an internal load balancer and an AgentCore runtime in VPC network mode, is what makes it demonstrable. Two details save time there: an agent in VPC mode needs Amazon S3 access from inside your VPC in order to start, and  RuntimeClientError  (HTTP 424) from  invoke-agent-runtime  means your container returned a 4xx or 5xx — look at the agent's logs, not the API.
Tighten the engine list to your domain. Measure which engines answer your instance, then keep the ones that do and that suit your subject matter.
Add caching. With a 21-to-1 input-to-output token ratio, prompt caching is where the money is.

Conclusion

You have built a research agent whose search engine is a container you run. The trade is real in both directions: you give up the full-page content, the SLA, and the compliance attestations a managed search API hands you, and you get network isolation, free marginal searches, control over which engines you query, and — in open-access domains like arXiv, PubMed, and OpenAlex — genuinely better results than a general web index would give you.
The piece to keep regardless of search provider is the source ledger. Citation validity held at 1.00 across twenty episodes not because the model is honest, but because a set difference told it when it was not.

Resources

# agents# amazon-bedrock-agentcore# amazon-bedrock# strands-agents# strands
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article