I Built a Self-Healing AI Agent on AWS Bedrock — It Thinks, Recovers, and Never Restarts From Zero

A phased Strands agent that classifies PDFs, extracts sections with Mistral Large, summarizes with Llama 3 70B, and survives any crash by checkpointing progress to S3 after every document — no duplicate compute, no human restart needed.

DHILIP S E

Published May 14, 2026

Last modified May 14, 2026

I Built a Self-Healing AI Agent on AWS Bedrock — It Thinks, Recovers, and Never Restarts From Zero

Tags: AWS Bedrock AgentCore Strands Agents Mistral Large Llama 3 Python AI Agents S3 CDK

My batch job died at 3 AM.

I had launched a pipeline to summarize 100 research papers. Eight hours of compute. Forty-seven papers done. Then — a network glitch. Process gone.

I restarted it. Forty-seven minutes of duplicate Bedrock calls. Forty-seven dollars of compute I had already paid for.

That afternoon I stopped trying to fix the script and started building something different — an agent that thinks before it reads, recovers from any crash, and never processes the same document twice.

This is exactly how I built it.

The Real Problem (It's Not Just Retries)

My first instinct was to add retry logic. My colleague suggested the same thing. But retries solve the wrong problem.

Retries handle transient errors within a single API call — a timeout, a rate limit, a blip. They cannot save a process that has been killed by the OS, lost its network entirely, or ran out of memory. When the process dies, every in-memory progress variable dies with it.

What I needed was a stateful, resumable agent — one that writes its progress to durable external storage after every single document, so that any crash at any point loses at most one document's work.

That's the foundation. Everything else builds on it.

What Makes This a Real Agent (Not Just a Script)

Before I show the code, I want to address something I've seen in a lot of "AI agent" tutorials.

Most of them are just a for-loop:

1
2
3
for doc in documents:

    summary = llm.summarize(doc)  # ← not an agent, just an API call

    save(summary)

That is a script. A real agent uses the LLM to make decisions — what to do next, which tools to use, how to handle unexpected situations.

Here is what my agent actually decides for every document:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
PDF arrives

    │

    ▼

LLM calls get_first_page → reads first 2,000 chars

    │

    ▼

LLM decides: "This is a research paper"

    │         (sees Abstract, Introduction, Results headers)

    ▼

LLM calls: extract_abstract  → 1,013 chars  ✅

           extract_methods   → empty        → "No methods section, skipping"

           extract_results   → 3,000 chars  ✅

    │

    ▼

Python → Llama 3 70B summarizes the extracted text

    │

    ▼

LLM calls: save_summary    → S3

           save_checkpoint → S3  ← the self-healing magic

For a survey paper, it picks completely different tools. For an unclear document, it falls back to full-text extraction. The LLM is making those decisions from the document content — not from if/else blocks I wrote.

Architecture

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
                        PDF in S3

                            │

                            ▼

        ┌───────────────────────────────────────────────┐

        │  PHASE 1 — CLASSIFY   (Mistral Large + 1 tool)│

        │  get_first_page → LLM reads first 2,000 chars │

        │  Output: research_paper | survey_paper | other │

        └───────────────────────────────────────────────┘

                 │               │               │

         research_paper    survey_paper        other

                 │               │               │

                 ▼               ▼               ▼

        ┌──────────────┐ ┌─────────────┐ ┌──────────────┐

        │PHASE 2 —     │ │PHASE 2 —    │ │PHASE 2 —     │

        │EXTRACT       │ │EXTRACT      │ │EXTRACT       │

        │              │ │             │ │              │

        │extract_      │ │extract_     │ │extract_      │

        │ abstract     │ │ abstract    │ │ full_text    │

        │extract_      │ │extract_     │ │(fallback)    │

        │ methods      │ │ contribs    │ │              │

        │extract_      │ │             │ │              │

        │ results      │ │             │ │              │

        └──────────────┘ └─────────────┘ └──────────────┘

                 │               │               │

                 └───────────────┴───────────────┘

                                 │

                                 ▼

        ┌───────────────────────────────────────────────┐

        │  PHASE 3 — SYNTHESIZE  (Python → Llama 3 70B) │

        │  summarize_text(extracted[:20000], max_words=150)│

        │  Large text bypasses LLM prompt entirely       │

        └───────────────────────────────────────────────┘

                                 │

                                 ▼

        ┌───────────────────────────────────────────────┐

        │  PHASE 4 — SAVE        (Mistral Large + 2 tools)│

        │  save_summary    → S3 output bucket            │

        │  save_checkpoint → S3 state bucket             │

        │  Python safety-net: always writes checkpoint   │

        └───────────────────────────────────────────────┘

                                 │

                          crash at any point?

                                 │

                                 ▼

        ┌───────────────────────────────────────────────┐

        │  WATCHDOG (demo/watchdog.py)                  │

        │  exit code ≠ 0 → wait 5s → restart --resume   │

        │  reads S3 checkpoint → skips completed docs    │

        │  → continues from where it stopped             │

        └───────────────────────────────────────────────┘

Two Models, Two Jobs

I use two Bedrock models with different roles:

Model	Role	Why
Mistral Large	Classification, tool orchestration	Strong reasoning; reliable tool calls
Meta Llama 3 70B	Summarization	Available in ap-south-1; cost-efficient at scale

One thing I learned the hard way: you cannot give Mistral Large 10 tools and expect it to pick the right ones. With a large tool set and a complex prompt, it describes what it would do instead of doing it. That's what pushed me toward the phased design — each phase gets 1–3 tools and a simple, unambiguous instruction.

Note for other regions: Llama 3 70B was chosen because Claude Haiku was not available in ap-south-1 (Mumbai) at the time. If you're deploying in us-east-1 or eu-west-1, Claude Haiku is ~10x cheaper for summarization and is the better choice.

Mistral gotcha: streaming=False is required for tool use on Bedrock. I spent an embarrassing amount of time on this error: "This model doesn't support tool use in streaming mode."

Tool Inventory

Ten tools across four phases:

Tool	Phase	What it does
`get_first_page`	Classify	Returns first 2,000 chars for LLM classification
`extract_abstract`	Extract	Regex pulls the Abstract section
`extract_methods`	Extract	Regex pulls Methods/Methodology
`extract_results`	Extract	Regex pulls Results/Experiments
`extract_contributions`	Extract	Regex pulls contribution lists (surveys)
`extract_full_text`	Fallback	Returns truncated full text
`summarize_text`	Synthesize	Calls Llama 3 70B via Bedrock InvokeModel
`save_summary`	Save	Writes Markdown to output S3 bucket
`save_checkpoint`	Save	Writes JSON progress to state S3 bucket
`record_error`	Error	Logs failure, lets pipeline continue

The Code

Phase 1 — LLM Classifies the Document

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
_CLASSIFY_PROMPT = """You are a document classifier.



Call get_first_page immediately. Read the result, then output EXACTLY ONE label:



  research_paper  — has Abstract, Introduction, Methods, Results

  survey_paper    — title or abstract contains "survey", "review", "overview"

  other           — unclear or non-standard structure



Output only the label. Nothing else."""





def _classify_document(bucket: str, key: str) -> str:

    agent = _make_agent([get_first_page], _CLASSIFY_PROMPT)

    response = agent(

        f"Classify this document.\n"

        f"bucket='{bucket}'\nkey='{key}'\n\n"

        f"Call get_first_page now, then output the label."

    )

    text = str(response).lower()



    if "research_paper" in text:

        return "research_paper"

    elif "survey" in text:

        return "survey_paper"

    else:

        return "other"

One tool. One output format. That's it. Any more complexity and the model starts narrating instead of acting.

Phase 2 — Tools Chosen From Phase 1 Result

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
def _extract_sections(bucket: str, key: str, doc_type: str) -> str:

    if doc_type == "research_paper":

        agent = _make_agent(

            [extract_abstract, extract_methods, extract_results],

            "Call ALL THREE tools. If any returns empty, note it and continue.",

        )

    elif doc_type == "survey_paper":

        agent = _make_agent(

            [extract_abstract, extract_contributions],

            "Call both tools.",

        )

    else:

        agent = _make_agent([extract_full_text], "Call extract_full_text now.")



    return str(agent(prompt)).strip()

The LLM's Phase 1 classification directly determines which tools it gets in Phase 2. That's the actual agent decision — not a human-written routing table.

Phase 3 — Summarize Without Touching the LLM Prompt

1
2
# Large text goes directly to Bedrock — never through the agent prompt

summary = summarize_text(extracted[:20000], max_words=150)

I tried passing extracted PDF text through the agent prompt. Two things happened: Greek and math symbols caused UnicodeEncodeError on Windows, and 40,000-character prompts hit token limits. Calling Bedrock directly from Python fixed both.

Phase 4 — Save With a Safety Net

1
2
3
4
5
6
7
# LLM saves via tools

agent = _make_agent([save_summary, save_checkpoint], _SAVE_PROMPT)

agent(f"Save summary to '{output_key}', then checkpoint task '{task_id}'.")



# Python safety-net — runs regardless of what the LLM did

# LLMs skip the last tool call more often than you'd expect

save_checkpoint(task_id, global_idx, doc_key)

I added the Python safety-net after noticing that a meaningful number of documents were being processed correctly but not checkpointed — the LLM was occasionally skipping the final tool call. The safety-net made checkpoint writes unconditional.

The Self-Healing Mechanism

Layer 1: S3 Checkpoint After Every Document

1
2
3
4
5
6
{

  "task_id": "run-001",

  "completed_idx": 47,

  "completed_keys": ["papers/2605.06227.pdf", "papers/2605.06228.pdf"],

  "errors": []

}

On startup, the agent loads this file and skips anything already completed:

1
2
3
4
checkpoint     = load_checkpoint(task_id)

completed_keys = set(checkpoint.get("completed_keys", []))

all_docs       = list_documents(input_bucket, "papers/")

remaining      = [k for k in all_docs if k not in completed_keys]

Crash at document 48 → checkpoint has 47 safe → resume skips 1–47 automatically.

Layer 2: Watchdog Auto-Restart

1
2
3
4
5
6
7
8
9
10
11
12
13
14
while attempt <= max_retries:

    result = subprocess.run([

        "python", "-m", "demo.run",

        "--task", task_id, "--resume", ...

    ])



    if result.returncode == 0:

        return 0        # done



    if result.returncode == 130:

        return 130      # Ctrl+C — user quit intentionally



    time.sleep(retry_delay)  # any crash → wait → restart

    attempt += 1

No human needed. The watchdog catches any non-zero exit, waits 5 seconds, and restarts with --resume. The agent reads the checkpoint and picks up exactly where it stopped.

Proven live:

1
2
3
4
5
6
7
8
Session 1: starts → processes 1 doc → process killed overnight

Checkpoint: {"completed_keys": ["papers/2605.06227.pdf"]}



Session 2: resumes →

  already_done=1, remaining=99

  [1/99] papers/2605.06228.pdf ...

  ...

  Done: 98/100 summarized, 2 errors (throttled, auto-skipped)

What Real LLM Reasoning Looks Like

Here is actual output from the agent processing a fairness-in-ML paper:

1
2
3
4
5
6
7
8
9
10
11
12
13
[Phase 1]

Tool: get_first_page → "Abstract  1. Introduction  2. Problem Formulation..."

Classification: research_paper



[Phase 2]

Tool: extract_abstract → 1,013 chars ✅

Tool: extract_methods  → (empty)

LLM: "No Methods section found — skipping and continuing."

Tool: extract_results  → 3,000 chars ✅



[Phase 4]

Tool: save_summary     → 803 bytes written to S3

Tool: save_checkpoint  → progress saved

The model read the first page, identified the document type, tried three tools, handled the missing section by noting it and moving on, and saved everything. No human wrote those decisions into if/else blocks.

Why Not Use AWS AgentCore?

Fair question — AWS offers AgentCore , a managed runtime with built-in memory, auto-recovery, and 8-hour execution windows. So why build DIY?

	This approach	AgentCore
State storage	S3 JSON — full control	Managed memory
Recovery	Subprocess watchdog	Managed 8-hour auto-recovery
Cost	S3 + Bedrock only	Additional runtime fees
Portability	Runs anywhere Python runs	AWS-specific
Transparency	Every component is visible	Managed abstraction

I chose DIY for two reasons: regional availability (AgentCore wasn't fully available in ap-south-1 when I built this) and transparency (I wanted every checkpoint, restart, and recovery decision to be readable in the source code, not hidden inside a managed service).

If you're in us-east-1 and want zero-maintenance recovery, AgentCore is the better choice. The checkpoint pattern here transfers directly to AgentCore Memory.

Results

Run against 100 cs.AI papers from arXiv:

Metric	Result
Total documents	100
Successfully summarized	98
Errors (throttled, auto-skipped)	2
Crash-resume validated	✅ Skipped 1 completed doc after overnight crash
S3 checkpoint integrity	✅ JSON intact across restart
Total Bedrock cost	~$1.80
Cost per document	~$0.018

1
2
3
4
Mistral Large   (reasoning + tool calls)   ~$0.80

Llama 3 70B     (summarization)            ~$1.00

──────────────────────────────────────────────────

Total                                      ~$1.80

Five Things I Learned Building This

1. Phase isolation is not optional.
Give Mistral Large 10 tools and it describes what it would do. Give it 2 tools and a simple instruction — it executes every time. The phased design was the single biggest reliability improvement.

2. Large text and LLM prompts don't mix.
PDF text has Greek letters, math symbols, ligatures. Putting that into a Strands agent prompt on Windows causes charmap encoding errors. Keep raw text out of the LLM — let Python handle it and call Bedrock directly.

3. LLMs skip tool calls more than you'd think.
Particularly the last tool in a sequence. Always add a Python fallback for anything critical — checkpointing, in particular, should never depend entirely on LLM behavior.

4. Two failure modes need two recovery mechanisms.
Per-document failures (empty sections, bad PDFs) → handled by LLM instructions and record_error. Process-level failures (crashes, OOM, network loss) → handled by the external watchdog. Don't conflate them.

5. Checkpoints must live outside the process.
I started with an in-memory dict. When the process died, so did every entry. The lesson: if it's not on S3 (or another durable store), it doesn't exist after a crash.

Run It Yourself

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
git clone https://github.com/DHILIP-S-E/self-healing-agent

cd self-healing-agent

pip install -r requirements.txt



# Deploy 3 S3 buckets via CDK

cd infrastructure && cdk deploy



# Seed 100 arXiv PDFs

python -m demo.seed_documents --count 100



# Run the reasoning agent

python -m demo.run \

  --task run-001 \

  --bucket YOUR_INPUT_BUCKET \

  --output-bucket YOUR_OUTPUT_BUCKET \

  --mode strands



# Run with auto-restart watchdog

python -m demo.watchdog \

  --task run-001 \

  --bucket YOUR_INPUT_BUCKET \

  --output-bucket YOUR_OUTPUT_BUCKET \

  --mode strands --max-retries 10

Kill the process mid-run. Run it again. Watch it skip everything already done.

What's Next

Three things on the roadmap:

Parallel agents — fan out with a shared S3 progress map and document-level locking
More document types — textbooks, code papers, technical reports with tailored extraction
CloudWatch dashboard — live progress bar reading checkpoint state during runs

The code is at github.com/DHILIP-S-E/self-healing-agent . MIT license. PRs welcome.

If you've built something similar or found a better pattern for long-running agent reliability — I'd love to hear about it in the comments.

# amazon-bedrock-agentcore # strands-agents # data # aws-community-builders

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

I Built a Self-Healing AI Agent on AWS Bedrock — It Thinks, Recovers, and Never Restarts From Zero

DHILIP S E

I Built a Self-Healing AI Agent on AWS Bedrock — It Thinks, Recovers, and Never Restarts From Zero

The Real Problem (It's Not Just Retries)

What Makes This a Real Agent (Not Just a Script)

Architecture

Two Models, Two Jobs

Tool Inventory

The Code

Phase 1 — LLM Classifies the Document

Phase 2 — Tools Chosen From Phase 1 Result

Phase 3 — Summarize Without Touching the LLM Prompt

Phase 4 — Save With a Safety Net

The Self-Healing Mechanism

Layer 1: S3 Checkpoint After Every Document

Layer 2: Watchdog Auto-Restart

What Real LLM Reasoning Looks Like

Why Not Use AWS AgentCore?

Results

Five Things I Learned Building This

Run It Yourself

What's Next