Durable, Learning, Trustworthy: Building a Autonomous Ops Agent on AWS (Part 1 — The Engine)

A fault-tolerant AI agent system that autonomously diagnoses AWS infrastructure issues, learns from past experience via episodic memory, and pauses for human approval before destructive actions — all powered by AWS-native primitives.

Series: Building a Autonomous Ops Agent: Fault-Tolerant Agentic AI with AWS Lambda Durable Functions, Amazon Bedrock, and Episodic Memory (2 articles)

1
Durable, Learning, Trustworthy: Building a Autonomous Ops Agent on AWS (Part 1 — The Engine) This article
2
Durable, Learning, Trustworthy: Building a Autonomous Ops Agent on AWS (Part 2 — The Guardrails)

Ajaykumar K V

AI Engineer | AWS Community Builder

Published Jun 30, 2026

Last modified Jul 9, 2026

This is Part 1 of a two-part series. Part 1 covers the system architecture, AWS Lambda Durable Functions, the multi-agent pipeline, Amazon Bedrock Nova Pro, the read-only tool catalog, and Amazon Bedrock AgentCore Episodic Memory.

Part 2 → covers the human-in-the-loop approval gate, the judge and escalation system, safe ingestion, the real-time frontend, observability, cost anatomy, and lessons learned from production bugs.

Built with AWS Lambda Durable Functions, Amazon Bedrock Nova Pro, and Amazon Bedrock AgentCore Episodic Memory.

Table of Contents — Part 1

Introduction — The Problem This Solves
System Architecture — The Complete Picture
AWS Lambda Durable Functions — The Orchestration Backbone
The Six-Step Orchestration Pipeline
The Multi-Agent Architecture — Four Specialists, One Synthesizer
Amazon Bedrock Nova Pro and the Strands Agents SDK
The Read-Only Tool Catalog — Secure Agent Actions
Amazon Bedrock AgentCore Episodic Memory — The Learning Loop

1. Introduction — The Problem This Solves

It is 2am and a CloudWatch alarm fires. A Lambda function is failing 95% of its invocations. An on-call engineer wakes up, opens the console, scrolls through logs, runs aws iam simulate-principal-policy from memory, finds the root cause, decides whether to act, and goes back to bed.

This is undifferentiated work. The investigation is mechanical. The diagnosis is pattern-matching against past incidents. The decision to act is the only part that genuinely benefits from a human brain.

"AI ops" tools today wrap an LLM around a chat interface and call it a day. That works for prototypes but collapses in production for three concrete reasons:

Reliability — A naive Lambda that calls a model is not fault tolerant. If the function times out at minute 14, you have lost everything. The work cannot resume from where it stopped.
Memory — A stateless agent is dumb every time. It cannot learn that a particular DynamoDB table always shows latency spikes during deploys, or that one specific Lambda's "high error rate" is actually a missing IAM permission that has been escalated three times this month.
Trust — An autonomous agent that can mutate AWS state will eventually do the wrong thing. There must be a way to pause for human authorization without keeping a Lambda running for hours.

This article describes a Autonomous Ops Agent agent that addresses all three. It runs on AWS Lambda Durable Functions for fault-tolerant orchestration, uses Amazon Bedrock Nova Pro for diagnosis and judgment, and remembers every past task in Amazon Bedrock AgentCore Episodic Memory. When a plan would change AWS state, it pauses on a durable callback for human approval — and during that wait, it costs nothing.

By the end of this two-part series, you will understand exactly how to build the same pattern.

2. System Architecture — The Complete Picture

Every component in the system was selected for a specific reason. The architecture is a deliberately layered design where each layer has one responsibility and clear boundaries with the next.

Diagram 1 — AWS Architecture

Diagram 2 — Layered Architecture

The Five Layers

Frontend Layer — A React + Vite single-page application that renders the dashboard, submission form, live task timeline, approval panel, and memory library. It uses TanStack Query for live polling, so the UI stays in sync with the backend without WebSockets.

API + Ingestion Layer — Amazon API Gateway exposes a small REST surface. Two Lambdas sit behind it: an API Lambda that serves the frontend, and an Intermediary Lambda that consumes from an SQS queue, deduplicates incoming tasks, and starts a durable execution.

Durable Orchestration Layer — The heart of the system. The orchestrator is a single AWS Lambda Durable Function. It runs a six-step pipeline (plus a parallel investigator block and an optional approval gate) with automatic checkpointing. If anything crashes mid-flight, it resumes from the last completed step on a new invocation — no work is lost.

AI Layer — Amazon Bedrock Nova Pro is the model. The Strands Agents SDK wraps it with structured output, tool calling, and the ReAct loop. Six different agent roles share the same model with different system prompts, tool subsets, and iteration budgets.

Storage + Memory Layer — DynamoDB holds task state with a 30-day TTL. S3 stores a full-fidelity audit log of every agent decision. Amazon Bedrock AgentCore Episodic Memory holds the semantic episode store that lets the agent learn across runs. CloudWatch captures structured JSON logs from every Lambda.

Long-Running Operations

This is the part that matters most. A typical task runs through:

Ingestion (SQS → Intermediary Lambda → Orchestrator)
Memory retrieval (~300 ms)
Three parallel investigator agents (~10–18s wall-clock, running concurrently)
The synthesizer agent (~10–45s with up to 12 tool calls)
Possibly a human approval pause — minutes, hours, or up to 24 hours
Judge evaluation (~2–5s)
Memory write + finalisation

Steps 1–4 and 6–7 run in roughly 30–60 seconds of compute. Step 5 is what makes this architecture different. With a naïve Lambda, holding execution open for 24 hours would either be impossible (the 15-minute Lambda timeout) or burn $50+ per task in idle compute. With Durable Functions, the Lambda exits cleanly during the wait. Storage costs apply, but compute does not.

That single property — pausing without paying — is what makes the human-in-the-loop pattern viable for a real production agent.

3. AWS Lambda Durable Functions — The Orchestration Backbone

Durable Functions extend AWS Lambda with built-in state, fault tolerance, and the ability to suspend execution for up to one year. They are the architectural backbone of this entire system. Every feature listed below is in active use.

What Durable Functions Are

A durable function is a Lambda function declared with @durable_execution that has access to a DurableContext. The SDK records every operation the function performs as a checkpoint in an AWS-managed durable execution store. If the Lambda crashes, runs out of time, or is suspended on a wait, the next invocation replays from the last checkpoint — no work is repeated.

1
2
3
4
5
6
7
8
9
10
11
12
13
from aws_durable_execution_sdk_python.context import DurableContext, durable_step
from aws_durable_execution_sdk_python.execution import durable_execution

@durable_step
def register_task(step_context: StepContext, task_id: str) -> dict[str, Any]:
    # ... DynamoDB write
    return {"taskId": task_id, "startedAt": started_at}

@durable_execution
def lambda_handler(event: dict, context: DurableContext) -> dict:
    task = TaskMessage.model_validate(event)
    context.step(register_task(task.taskId))   # checkpoint
    # ... more steps

context.step(...) is the basic primitive. Its result is persisted to the durable store, and on replay the SDK skips re-executing the underlying function and returns the cached result instead.

Features Used in This Project

The orchestrator uses six distinct durable execution primitives. Each one solves a real problem.

1. @durable_step — Checkpointed steps

Every long or side-effectful operation is wrapped in a step. DynamoDB writes, agent invocations, S3 puts, AgentCore reads, S3 audit log writes — all six pipeline stages are steps. If the Lambda dies after step 4, the replay starts at step 5 with all prior results intact.

2. context.parallel(...) — Structured concurrency

The three investigator agents (logs, metrics, IAM) run in parallel, each on its own thread inside the SDK's executor:

1
2
3
4
5
6
7
8
9
parallel_result = context.parallel(
    functions=[
        lambda c: c.step(logs_analysis_step(parallel_task_dump)),
        lambda c: c.step(metrics_analysis_step(parallel_task_dump)),
        lambda c: c.step(iam_analysis_step(parallel_task_dump)),
    ],
    name="parallel_investigations",
    config=ParallelConfig(completion_config=CompletionConfig.all_completed()),
)

CompletionConfig.all_completed() means the block waits for every branch to finish, even if some fail. Failed branches contribute an {"error": ...} finding to the synthesizer instead of aborting the whole run. Each branch's result is checkpointed individually.

3. context.map(...) — Bounded fan-out

When a task targets a list of resources, the orchestrator fans out one investigator per resource using context.map() with a concurrency cap:

1
2
3
4
5
6
7
8
9
map_result = context.map(
    inputs=affected,
    func=_resource_branch,
    name="per_resource_diagnosis",
    config=MapConfig(
        max_concurrency=5,
        completion_config=CompletionConfig.all_completed(),
    ),
)

Without a concurrency cap, a 50-resource task would burn through Bedrock quota in seconds. Five concurrent branches keeps wall-clock low while staying within rate limits.

4. context.wait_for_callback(...) — Suspend for external events (the killer feature)

When the agent's plan would mutate AWS state, the orchestrator creates a durable callback, persists the callback ID + status to DynamoDB, and the Lambda exits. From the official AWS documentation :

"When you create a callback, the SDK checkpoints the operation and returns a unique callback ID. After you send the callback ID to an external system, the SDK suspends the durable function and terminates the current invocation. The function does not incur compute charges while suspended."

This is what makes the 24-hour approval timeout viable:

1
2
3
4
5
raw = context.wait_for_callback(
    submitter=submit,
    name="approval-gate",
    config=WaitForCallbackConfig(timeout=Duration(seconds=86400)),  # 24h
)

When the operator clicks Approve, the API Lambda calls SendDurableExecutionCallbackSuccess, the durable backend wakes up a brand new orchestrator invocation, and execution resumes from exactly the point it was suspended. The full approval gate lifecycle is covered in depth in Part 2.

5. context.invoke(...) — Chained durable invocation

Escalation is a separate durable function. The orchestrator invokes it as a child execution when the judge score falls below 0.5:

1
2
3
4
5
context.invoke(
    _qualified_escalation_arn(),   # MUST be qualified (:$LATEST)
    {"taskId": task.taskId, "goal": task.goal, "judgeScore": verdict.score},
    name="escalate-low-score",
)

Durable functions must be invoked with a qualified ARN (:$LATEST or a numeric version). Calling with the bare function ARN raises InvalidParameterValueException and — because this invoke happens inside the orchestrator's own durable execution — fails the entire execution. This was a real production bug covered in Part 2's lessons section.

6. Exception-based replay safety

The SDK guarantees that on replay, every previously-checkpointed step returns the same result without re-executing the side-effectful body. I rely on this in record_approval — the API Lambda writes the approval decision to DynamoDB, and the orchestrator's own record_approval step writes it again. Both writes are idempotent, and on replay only one actually runs.

How Long-Running Operations Are Supported

Three separate features compose to support arbitrary-length workflows:

Lambda's per-invocation 15-minute limit is bypassed — a workflow that takes 24 hours uses 24 hours of wall-clock time but only seconds of Lambda compute spread across many short invocations.
Wait operations cost nothing — wait, waitForCallback, and waitForCondition all checkpoint and exit. The AWS best practices guide is explicit: "Durable functions can pause for seconds, hours, or days without keeping a Lambda invocation running."
State is durable for up to one year — execution state survives Lambda redeploys, account-level limits, and regional incidents that would kill an in-memory workflow.

This is what lets the system pause for a human and resume with full context preserved.

4. The Six-Step Orchestration Pipeline

The orchestrator is a single durable function that runs six checkpointed steps, plus an optional approval gate and parallel investigator block. Here is what each step actually does.

Diagram 3 — Full Task Lifecycle Sequence Diagram

Step 1 — `register_task`

The first step transitions the task from PENDING to RUNNING in DynamoDB and records the start timestamp.

1
2
3
4
5
6
7
8
9
10
@durable_step
def register_task(step_context: StepContext, task_id: str) -> dict[str, Any]:
    started_at = datetime.now(timezone.utc).isoformat()
    _table().update_item(
        Key={"taskId": task_id},
        UpdateExpression="SET #status = :status, startedAt = :startedAt",
        ExpressionAttributeNames={"#status": "status"},
        ExpressionAttributeValues={":status": "RUNNING", ":startedAt": started_at},
    )
    return {"taskId": task_id, "startedAt": started_at}

Step 2 — `retrieve_memories`

Before the agent does any work, the orchestrator does a semantic search against AgentCore Episodic Memory for similar past tasks. The query is built from the task type and goal, and the top-K (5) episodes are returned.

1
2
3
4
5
response = _agentcore.retrieve_memory_records(
    memoryId=_memory_id(),
    namespace=actor_namespace(_memory_strategy_id(), task_type),
    searchCriteria={"searchQuery": query, "topK": MEMORY_TOP_K},
)

If the call fails, we degrade gracefully and return an empty MemoryResult. Memory failure must not abort a task. The retrieved episodes get injected into the synthesizer's prompt as PAST_EPISODES — that is the learning loop.

Step 2.5 — `parallel_investigations`

Three specialist agents fan out in parallel:

Logs Investigator — has access to cloudwatch_logs_filter_events only.
Metrics Investigator — has access to cloudwatch_get_metric_statistics only.
IAM Investigator — has access to iam_simulate_principal_policy, iam_list_attached_role_policies, and lambda_get_function (so it can resolve a Lambda's execution role).

Each runs in its own thread, each writes its own checkpoint, and the merged findings dict is passed to the synthesizer. Narrower tool surface means tighter prompts and cleaner reasoning.

Step 2.6 — `per_resource_diagnosis` (conditional)

If the task includes a list of affected resources, a context.map(...) block runs one investigator per resource with a concurrency cap of 5. Skipped cleanly when no resources were specified.

Step 3 — `invoke_agent` (the synthesizer)

The synthesizer is the main agent. It receives the goal, the retrieved memories, and the merged investigator findings. It has access to all 10 read-only tools and an iteration budget of 12 tool calls. It produces a structured AgentResponse with diagnosis, actions, reasoning, confidence, and the critical destructive flag.

Step 3.5 — Approval Gate (conditional)

If the synthesizer's plan is destructive AND the task did not pre-authorize destructive actions, the orchestrator suspends on a durable callback. The full lifecycle is covered in Part 2 Section 9.

Step 4 — `evaluate_outcome` (the judge)

A separate Bedrock invocation, with no tools, scores the agent's plan on a four-criteria rubric: Goal Achievement (40%), Action Correctness (25%), Efficiency (15%), Safety (20%). The judge is a different agent than the synthesizer to prevent self-evaluation bias.

Step 5 — `write_memory`

The orchestrator builds an EpisodeRecord from the task, agent response, and judge verdict, then writes it to AgentCore Memory under a session namespace. This closes the learning loop — every task contributes to the agent's accumulated knowledge.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
_agentcore.batch_create_memory_records(
    memoryId=_memory_id(),
    records=[{
        "requestIdentifier": episode_id,
        "namespaces": [session_namespace(strategy_id, task.taskType, task.taskId)],
        "content": {"text": episode.model_dump_json()},
        "timestamp": datetime.now(timezone.utc),
        "metadata": {
            "taskType": {"stringValue": task.taskType},
            "judgeScore": {"numberValue": verdict.score},
            "success": {"stringValue": str(verdict.success)},
        },
    }],
)

Step 6 — `finalize_task`

The terminal step writes the final status (SUCCEEDED, FAILED, or ESCALATED) to DynamoDB and persists the full audit log to S3 at audits/{taskId}.json. The audit log is the source of truth — the UI reads from it to render the timeline, retrieved episodes, agent reasoning, and judge verdict. If the score is below 0.5, the orchestrator additionally invokes the escalation child durable function as a side effect.

5. The Multi-Agent Architecture — Four Specialists, One Synthesizer

A single agent with all 10 tools and a 4000-token system prompt produces noisy, inconsistent output. We learned this the hard way. The fix was to split the work across six role-specific agents, each with a focused tool subset and a dedicated system prompt.

Why Specialized Agents Beat One Big Agent

When you give an agent ten tools, it has to reason about which one to use at every step. Tool selection becomes a search problem layered on top of the actual diagnostic problem. Splitting into specialists fixes both issues. The Logs Investigator only knows about cloudwatch_logs_filter_events. Its system prompt is small and laser-focused on filter pattern syntax. It cannot pivot to "let me check IAM" because it has no IAM tools. It does one thing well, returns its findings, and exits.

The Six Roles

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
ROLE_TOOLS: dict[Role, list] = {
    "logs_investigator": [cloudwatch_logs_filter_events],
    "metrics_investigator": [cloudwatch_get_metric_statistics],
    "iam_investigator": [
        iam_simulate_principal_policy,
        iam_list_attached_role_policies,
        lambda_get_function,   # to resolve a Lambda's execution role
    ],
    "resource_investigator": [
        dynamodb_describe_table, dynamodb_describe_continuous_backups,
        lambda_get_function, ec2_describe_instances, rds_describe_db_instances,
    ],
    "synthesizer": ALL_TOOLS,
    "judge": [],
}

The synthesizer is the only role with the full tool catalog. Its system prompt explicitly tells it to keep the goal primary — investigator findings are hints, not directives. The judge has no tools at all — it is a pure evaluation agent whose only job is to score the synthesizer's output against the rubric.

The IterationLimitHook

Strands does not expose a max_iterations constructor argument. The documented pattern is to count tool calls and short-circuit the ReAct loop once a budget is exhausted. We implement this with a hook that fires on BeforeToolCallEvent:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
class IterationLimitHook:
    def before_tool(self, event: BeforeToolCallEvent) -> None:
        tool_name = event.tool_use.get("name") or ""

        # Never count or cancel the synthetic structured-output tool.
        if tool_name in self._exempt or _is_structured_output_tool(tool_name):
            return

        if self._count >= self._budget:
            event.cancel_tool = (
                f"Iteration limit ({self._budget} tool calls) reached. "
                f"Stop calling tools and produce your final answer now."
            )
            return

        self._count += 1

Two non-obvious details, both verified against the Strands 1.41 source:

Cancellation must happen on BeforeToolCallEvent, not AfterToolCallEvent. The latter only allows writes to {"result", "retry"}; assigning cancel_message there raises AttributeError and aborts the whole invocation.
The structured-output tool must be exempt. Strands implements structured_output_model= as a dynamically-registered tool whose name equals the Pydantic model's class name (e.g. AgentResponse). Cancelling it leaves the agent with no structured output, forcing the runner's fallback path.

Investigator Findings Flow

After the parallel block completes, findings are merged into a dict keyed by domain and passed to the synthesizer:

1
2
findings: dict[str, Any] = _merge_parallel_findings(parallel_result)
# findings = {"logs": {...}, "metrics": {...}, "iam": {...}}

This dict is injected as SUPPORTING_INVESTIGATOR_FINDINGS (context only — address the GOAL above first). The framing is intentional: in early runs, the synthesizer would pivot away from the stated goal when investigators returned alarming-looking findings. Adding this framing — plus the explicit "goal is primary" rule in the synthesizer system prompt — fixed it.

Reconstructing the Tool Trace

After every agent invocation, we walk agent.messages and pair assistant-side toolUse blocks with user-side toolResult blocks by toolUseId. This gives us a chronological trace of every tool call, which we attach to the response and surface in the UI's "Tool Calls" panel.

6. Amazon Bedrock Nova Pro and the Strands Agents SDK

Amazon Bedrock Nova Pro is the model behind every agent in the system. The Strands Agents SDK is the wrapper that connects the model to tools, manages the ReAct loop, and enforces structured output.

Why Nova Pro

Three properties matter most for an ops agent:

Strong tool calling — the model must produce well-formed tool calls every time.
Reliable structured output — when we ask for an AgentResponse Pydantic schema, we need the model to actually conform.
Reasonable latency — synthesizer turns take 10–45 seconds with up to 12 tool calls; anything slower would make the UI feel unusable.

Nova Pro hits all three. It is also a first-party AWS model, so there is no cross-account billing, no third-party rate limiting, and IAM is the only access-control surface.

`BedrockModel(streaming=False)`

Streaming is disabled deliberately. The orchestrator does not need bedrock:InvokeModelWithResponseStream, which keeps the IAM policy smaller and prevents token-by-token responses from cluttering CloudWatch logs.

1
2
3
4
5
model = BedrockModel(
    model_id=model_id,
    temperature=0.3,
    streaming=False,
)

The temperature of 0.3 is a calibration choice: low enough that the agent reliably picks the right tool, high enough that it occasionally explores when stuck.

Structured Output via Pydantic

The biggest reliability win in this codebase is structured_output_model=. Strands enforces the schema at the model boundary — the agent cannot return free-form JSON that needs regex parsing:

1
2
3
4
5
6
7
8
9
agent = Agent(
    model=model,
    tools=ROLE_TOOLS[role],
    system_prompt=system_prompt,
    structured_output_model=AgentResponse,  # enforced by Strands
)

result = agent(prompt)
parsed = result.structured_output     # already a validated AgentResponse

The AgentResponse Pydantic model declares every field the orchestrator needs:

1
2
3
4
5
6
7
8
9
class AgentResponse(BaseModel):
    diagnosis: str = ""
    actions: list[str] = Field(default_factory=list)
    detailedActions: list[ActionDetail] = Field(default_factory=list)
    reasoning: str = ""
    confidence: float = 0.0
    destructive: bool = False
    toolCalls: list[ToolCall] = Field(default_factory=list)
    findings: dict[str, Any] = Field(default_factory=dict)

We retired all regex-based JSON parsing in Phase 2 of the project. Schema enforcement at the model boundary is significantly more reliable.

Graceful Degradation on Token Overflow

Real-world LLMs occasionally hit context-window limits, especially when the synthesizer prompt grows with retrieved episodes. We catch MaxTokensReachedException specifically and retry once with a stripped-down prompt:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
def run_synthesizer(task, memories, findings, *, model_id, region_name=None):
    def _attempt(mem: MemoryResult) -> AgentResponse:
        # ... build agent and prompt, invoke

    try:
        return _attempt(memories)
    except Exception as exc:
        if "MaxTokens" in type(exc).__name__:
            logger.warning("Token limit hit; retrying without episodic memory.")
            try:
                return _attempt(MemoryResult())   # empty memories on retry
            except Exception as exc2:
                return AgentResponse()           # final fallback
        return AgentResponse()

Try with full context, retry without optional context, then degrade gracefully — this pattern is essential for production AI agents.

7. The Read-Only Tool Catalog — Secure Agent Actions

The agent can only call read-only AWS APIs. This is a hard architectural boundary combining IAM policies, tool design, and a deliberate separation between diagnosis and execution.

Design Principle: Diagnosis Is Not Execution

The agent's job is to investigate, plan, and propose. It does not execute changes. When a destructive action is needed, the orchestrator pauses for human approval and a human (or a post-approval automation) executes the change.

Blast radius is bounded — an LLM that hallucinates a destructive call cannot harm anything because it has no destructive tools.
Audit trails are clean — every state change in the account has a human signature, not an "agent did it" entry.
The agent's design space is smaller — it does not need to handle rollback, partial failure, or transactional semantics.

The 10 Tools

Every tool is decorated with @strands.tool, wraps exactly one boto3 read-only API, and returns a JSON-serializable dict. Errors are converted to {"error": str, "code": str} instead of raising, so the LLM can reason about the failure.

1
2
3
4
5
6
7
8
9
10
11
12
ALL_TOOLS = [
    cloudwatch_get_metric_statistics,
    cloudwatch_logs_filter_events,
    dynamodb_describe_table,
    dynamodb_describe_continuous_backups,
    lambda_list_functions,
    lambda_get_function,
    iam_simulate_principal_policy,
    iam_list_attached_role_policies,
    ec2_describe_instances,
    rds_describe_db_instances,
]

The tool docstring is the LLM's instruction manual. Bad docstrings produce malformed tool calls. We learned this when an early run wasted six tool calls on InvalidParameterException because the agent tried "error" OR { $.level = "ERROR" } (mixing text and JSON selectors in a CloudWatch filter). Adding the explicit "do not combine" rule to the docstring fixed it permanently:

1
2
3
4
5
6
7
8
9
10
11
12
13
@tool
def cloudwatch_logs_filter_events(...) -> dict:
    """Search a CloudWatch Logs group for events matching a filter.

    Filter patterns — valid forms:
      - Plain substring: ``ERROR`` or ``"Task timed out"``
      - Match ANY of several terms: ``?ERROR ?Exception ?Traceback``
      - JSON logs: a SINGLE selector ``{ $.level = "ERROR" }``
      - Empty string ``""`` matches everything.

    Do NOT combine a JSON ``{ ... }`` selector with text terms via OR.
    Do NOT pass a bare/unbalanced quote — both fail with InvalidParameterException.
    """

`_serialize()` — Coercing boto3 Responses to JSON

boto3 responses contain datetime objects, sets, and the noisy ResponseMetadata envelope. The LLM needs none of that:

1
2
3
4
5
6
def _serialize(value: Any) -> Any:
    if isinstance(value, _dt.datetime):
        return value.isoformat()
    if isinstance(value, dict):
        return {k: _serialize(v) for k, v in value.items() if k != "ResponseMetadata"}
    # ... lists, sets, bytes handled similarly

Two-Layer Defense: IAM Allow + Explicit Deny

The orchestrator's IAM role has an explicit allow on the 14 read-only actions, but more importantly it has an explicit deny on common destructive actions as defense-in-depth:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
orchestrator_fn.add_to_role_policy(
    iam.PolicyStatement(
        sid="DenyAllDestructive",
        effect=iam.Effect.DENY,
        actions=[
            "ec2:TerminateInstances", "ec2:StopInstances",
            "rds:DeleteDBInstance", "rds:ModifyDBInstance",
            "dynamodb:DeleteTable", "dynamodb:UpdateTable",
            "lambda:DeleteFunction", "lambda:UpdateFunctionCode",
            "iam:Delete*", "iam:Put*", "iam:Update*", "iam:Attach*",
            "logs:DeleteLogGroup",
        ],
        resources=["*"],
    )
)

IAM evaluates explicit deny before any allow. If a future tool is accidentally added that calls a destructive API, this policy stops it cold. The agent literally cannot harm the account, even if compromised.

8. Amazon Bedrock AgentCore Episodic Memory — The Learning Loop

Without memory, an AI agent is dumb every time. Every task starts from scratch, every diagnosis ignores prior incidents, and every wrong answer gets repeated. Episodic memory turns the agent from a stateless function into something that genuinely improves with experience.

What "Episodic" Means

Episodic memory is different from RAG. RAG searches a static knowledge base. Episodic memory stores agent experiences — what the agent tried, what worked, what failed, what it concluded — and retrieves them by semantic similarity to a new task.

Each episode is a structured record:

1
2
3
4
5
6
7
8
9
10
class EpisodeRecord(BaseModel):
    goal: str
    taskType: str
    strategyUsed: str          # the agent's diagnosis, summarized
    actions: list[str]         # what the agent recommended
    outcome: TaskOutcome       # success/failure + summary
    judgeScore: float          # the rubric score
    reflection: str            # what worked + what failed + what to do next time
    tags: list[str]
    createdAt: str

The most important field is strategyUsed. It appears verbatim in future tasks' PAST_EPISODES context block — this is literally how the agent learns.

Building a Meaningful `strategyUsed`

Early runs wrote strategyUsed: "unknown" whenever the agent's diagnosis was empty. Future tasks would then retrieve "Strategy: unknown" — useless context. The fix builds the field from the richest available signal in priority order:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
def build_episode(task, agent_response, verdict) -> EpisodeRecord:
    strategy = (agent_response.diagnosis or "").strip()
    if not strategy and agent_response.detailedActions:
        strategy = agent_response.detailedActions[0].title.strip()
    if not strategy and agent_response.actions:
        strategy = agent_response.actions[0].strip()
    if not strategy:
        reasoning_excerpt = (agent_response.reasoning or "").strip()[:200]
        if reasoning_excerpt:
            strategy = reasoning_excerpt
    if not strategy:
        outcome_word = "succeeded" if verdict.success else "failed"
        strategy = f"{task.taskType.lower().replace('_', '-')} task {outcome_word}."
    # ... build EpisodeRecord

Retrieval Before Every Task

Step 2 of the orchestrator does a semantic search against the actor namespace for similar past tasks. The top-K episodes are injected into the synthesizer's prompt:

1
2
3
4
5
6
7
PAST_EPISODES (top 2 by score — learn from these):
- Goal: An alarm has fired for the DynamoDB table 'self-healing-tasks-mvp'...
  Strategy: The high read latency on the DynamoDB table is likely due to
  a sudden increase in read traffic that the on-demand capacity has not
  yet scaled to handle efficiently...
  Outcome: SUCCESS (score: 0.86)
  Reflection: What worked: The diagnosis correctly identifies potential...

This is the agent learning. When the same goal arrives again, the agent already has a high-confidence template for the answer. A real example from the system: the same DynamoDB latency task moved from a 0.60 score (no memory) to 0.95 (memory retrieved, strategy reused with full confidence).

Bounded Memory Injection

Past episodes can grow large. Without bounds, a task with five long episodes can blow the context window. The fix is to trim deterministically before injection — top-2 by judge score, with reflections truncated to 200 characters each:

1
2
3
4
5
6
7
8
9
def _trim_episodes(episodes, max_episodes=2, max_reflection_chars=200):
    sorted_eps = sorted(episodes, key=lambda e: getattr(e, "judgeScore", 0.0), reverse=True)
    trimmed = []
    for ep in sorted_eps[:max_episodes]:
        reflection = (getattr(ep, "reflection", "") or "").strip()
        if len(reflection) > max_reflection_chars:
            reflection = reflection[:max_reflection_chars].rstrip() + "…"
        trimmed.append({...})
    return trimmed

Two Namespaces

Writes go to a session namespace (uniquely attributable to a task), but reads search across the actor namespace (so any INFRA_REMEDIATION task retrieves all past INFRA_REMEDIATION episodes regardless of which session wrote them).

The Memory Library UI

The frontend's Memory Library page calls GET /memory to surface accumulated episodes for browsing. This page is a window into the agent's accumulated knowledge — useful for ops engineers who want to understand what the agent has learned and why it makes certain decisions.

What's Next: Part 2

Part 1 covered the foundation: architecture, the durable orchestration model, the multi-agent pipeline, Bedrock integration, the read-only tool catalog, and episodic memory.

Continue to Part 2 →, which covers:

Section 9 — The Human-in-the-Loop Approval Gate (the durable callback lifecycle, cost reality, what the operator sees)
Section 10 — The Judge, Scoring, and Escalation (the four-criteria rubric, approval-gate scoring rule, the escalation child function)
Section 11 — Safe Ingestion (SQS, deduplication, the qualified ARN requirement)
Section 12 — The Execution Timeline (translating the durable event stream to a live UI)
Section 13 — Infrastructure as Code (CDK stack, DurableConfig, SQS visibility timeout)
Section 14 — The Frontend (TanStack Query polling, ApprovalPanel, ExecutionTimeline, ScoreGauge)
Section 15 — Observability, Reliability, and Cost (structured logs, S3 audit log, cost breakdown, chaos engineering)
Section 16 — Key Design Decisions and Lessons Learned (7 real production bugs and the patterns that fixed them)
Section 17 — Conclusion and What's Next

# agents # agentic-ai # aws-lambda # ai-engineering # amazon-bedrock-agentcore

Series: Building a Autonomous Ops Agent: Fault-Tolerant Agentic AI with AWS Lambda Durable Functions, Amazon Bedrock, and Episodic Memory (2 articles)

1
Durable, Learning, Trustworthy: Building a Autonomous Ops Agent on AWS (Part 1 — The Engine) This article
2
Durable, Learning, Trustworthy: Building a Autonomous Ops Agent on AWS (Part 2 — The Guardrails)

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

Durable, Learning, Trustworthy: Building a Autonomous Ops Agent on AWS (Part 1 — The Engine)

Series: Building a Autonomous Ops Agent: Fault-Tolerant Agentic AI with AWS Lambda Durable Functions, Amazon Bedrock, and Episodic Memory (2 articles)

Ajaykumar K V

Table of Contents — Part 1

1. Introduction — The Problem This Solves

2. System Architecture — The Complete Picture

The Five Layers

Long-Running Operations

3. AWS Lambda Durable Functions — The Orchestration Backbone

What Durable Functions Are

Features Used in This Project

How Long-Running Operations Are Supported

4. The Six-Step Orchestration Pipeline

Step 1 — register_task

Step 2 — retrieve_memories

Step 2.5 — parallel_investigations

Step 2.6 — per_resource_diagnosis (conditional)

Step 3 — invoke_agent (the synthesizer)

Step 3.5 — Approval Gate (conditional)

Step 4 — evaluate_outcome (the judge)

Step 5 — write_memory

Step 6 — finalize_task

5. The Multi-Agent Architecture — Four Specialists, One Synthesizer

Why Specialized Agents Beat One Big Agent

The Six Roles

The IterationLimitHook

Investigator Findings Flow

Reconstructing the Tool Trace

6. Amazon Bedrock Nova Pro and the Strands Agents SDK

Why Nova Pro

BedrockModel(streaming=False)

Structured Output via Pydantic

Graceful Degradation on Token Overflow

7. The Read-Only Tool Catalog — Secure Agent Actions

Design Principle: Diagnosis Is Not Execution

The 10 Tools

_serialize() — Coercing boto3 Responses to JSON

Two-Layer Defense: IAM Allow + Explicit Deny

8. Amazon Bedrock AgentCore Episodic Memory — The Learning Loop

What "Episodic" Means

Building a Meaningful strategyUsed

Retrieval Before Every Task

Bounded Memory Injection

Two Namespaces

The Memory Library UI

What's Next: Part 2

Series: Building a Autonomous Ops Agent: Fault-Tolerant Agentic AI with AWS Lambda Durable Functions, Amazon Bedrock, and Episodic Memory (2 articles)

Step 1 — `register_task`

Step 2 — `retrieve_memories`

Step 2.5 — `parallel_investigations`

Step 2.6 — `per_resource_diagnosis` (conditional)

Step 3 — `invoke_agent` (the synthesizer)

Step 4 — `evaluate_outcome` (the judge)

Step 5 — `write_memory`

Step 6 — `finalize_task`

`BedrockModel(streaming=False)`

`_serialize()` — Coercing boto3 Responses to JSON

Building a Meaningful `strategyUsed`