Durable, Learning, Trustworthy: Building a Autonomous Ops Agent on AWS (Part 2 — The Guardrails)

A fault-tolerant AI agent system that autonomously diagnoses AWS infrastructure issues, learns from past experience via episodic memory, and pauses for human approval before destructive actions — all powered by AWS-native primitives.

Series: Building a Autonomous Ops Agent: Fault-Tolerant Agentic AI with AWS Lambda Durable Functions, Amazon Bedrock, and Episodic Memory (2 articles)

1
Durable, Learning, Trustworthy: Building a Autonomous Ops Agent on AWS (Part 1 — The Engine)
2
Durable, Learning, Trustworthy: Building a Autonomous Ops Agent on AWS (Part 2 — The Guardrails) This article

Ajaykumar K V

AI Engineer | AWS Community Builder

Published Jun 30, 2026

Last modified Jul 9, 2026

This is Part 2 of a two-part series. Read Part 1 for the system architecture, AWS Lambda Durable Functions, the multi-agent pipeline, Amazon Bedrock Nova Pro , the read-only tool catalog, and Amazon Bedrock AgentCore Episodic Memory.

Part 2 covers the human-in-the-loop approval gate, the judge and escalation system, safe ingestion, the real-time frontend, observability, cost anatomy, and production lessons learned.

Table of Contents — Part 2

The Human-in-the-Loop Approval Gate
The Judge, Scoring, and Escalation
Safe Ingestion — SQS, Deduplication, and the Intermediary Lambda
The Execution Timeline — Translating Durable Events to a Live UI
Infrastructure as Code — The CDK Stack
The Frontend — Real-Time Task Execution Visibility
Observability, Reliability, and Cost
Key Design Decisions and Lessons Learned
Conclusion — What's Next

9. The Human-in-the-Loop Approval Gate

This is where Durable Functions earns its place in the architecture. When the agent proposes a destructive action, the orchestrator pauses for human authorization. During that pause, the Lambda is not running and the cost is zero.

Diagram 2 — Approval Gate Sequence

When the Gate Fires

The gate condition is simple:

1
2
3
4
needs_approval = (
    plan.destructive
    and not task.payload.constraints.allowDestructiveActions
)

The agent sets plan.destructive = True whenever any action in its plan would mutate AWS state — based on what the action does, not on whether the agent can execute it. If the operator pre-authorized destructive actions on the submission form, the gate is skipped.

The Callback Lifecycle

When the gate fires, the orchestrator calls wait_for_callback(...). The submitter function persists the callback ID and proposed actions to DynamoDB so the API Lambda can look them up:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
def submit(callback_id: str, ctx: WaitForCallbackContext) -> None:
    _table().update_item(
        Key={"taskId": task.taskId},
        UpdateExpression=(
            "SET #status = :status, callbackId = :cid, "
            "pendingApprovalAt = :pa, proposedActions = :actions"
        ),
        ExpressionAttributeNames={"#status": "status"},
        ExpressionAttributeValues={
            ":status": "PENDING_APPROVAL",
            ":cid": callback_id,
            ":pa": pending_at,
            ":actions": actions,
        },
    )

raw = context.wait_for_callback(
    submitter=submit,
    name="approval-gate",
    config=WaitForCallbackConfig(timeout=Duration(seconds=86400)),  # 24h
)

The instant wait_for_callback is invoked:

The callback ID is generated and persisted.
The submitter runs (writing PENDING_APPROVAL to DynamoDB).
The Lambda terminates with InvocationCompleted.

From this point, zero compute is billed until the callback arrives.

Resuming from the Callback

When the operator clicks Approve in the UI, the API Lambda calls SendDurableExecutionCallbackSuccess:

1
2
3
4
5
6
7
8
_lambda.send_durable_execution_callback_success(
    CallbackId=callback_id,
    Result=json.dumps({
        "approved": True,
        "comment": comment,
        "decidedAt": decided_at,
    }).encode("utf-8"),
)

The durable backend immediately spins up a new orchestrator invocation. The SDK replays all completed checkpoints from the stored history (no re-execution of steps), and execution resumes at the line after wait_for_callback.

Reject and Timeout

Rejection uses SendDurableExecutionCallbackFailure with ErrorType="REJECTED". Inside the orchestrator, this surfaces as CallbackError:

1
2
3
4
5
6
7
8
try:
    decision = _approval_gate(context, task, plan)
    approval_comment = decision.comment
    context.step(record_approval(task.taskId, approval_comment))
except CallbackError as exc:
    comment = str(exc) or "Approval timeout"
    context.step(finalize_rejected(task.taskId, comment, audit_rejected))
    return {"taskId": task.taskId, "status": "REJECTED", "approvalComment": comment}

The 24-hour timeout works the same way — the SDK raises CallbackError after the timeout elapses.

What the Operator Sees

The frontend polls GET /tasks/{id} every 2.5 seconds. When the status flips to PENDING_APPROVAL, the ApprovalPanel renders with the proposed actions verbatim, an optional comment field, and Approve/Reject buttons.

Cost Reality

A real task in the system ran for 2 minutes 48 seconds wall-clock, of which 2 minutes 18 seconds was the operator reviewing and clicking Approve. The Lambda billed compute time was approximately 30 seconds. The 138 seconds of waiting cost exactly $0.

With a naïve polling approach (a Lambda that loops checking DynamoDB every 30 seconds), 138 seconds of waiting at 1024 MB would cost ~$0.0024 per task. Across 10,000 tasks/month, that is $24/month for nothing. The Durable Functions approach is $0. Across any approval period up to 24 hours, the cost is still $0.

10. The Judge, Scoring, and Escalation

The judge is a separate Bedrock invocation that scores the agent's plan. It has no tools, just the same model with a different system prompt and structured_output_model=JudgeVerdict.

Why a Separate Judge

If the synthesizer scored its own work, it would be biased. We invoke a fresh Bedrock model with a clean context window and ask it to evaluate the synthesizer's output against an explicit rubric. This is closer to LLM-as-judge research patterns — and it works well because the judge sees only the final output, not the agent's noisy reasoning trace.

The Four-Criteria Rubric

1
2
3
4
5
6
7
goalAchievement (40%)   — did the agent accomplish the stated goal?
actionCorrectness (25%) — were proposed actions appropriate and safe?
efficiency (15%)        — was the approach efficient?
safety (20%)            — any unintended side effects?

score = 0.4 * goalAchievement + 0.25 * actionCorrectness
      + 0.15 * efficiency + 0.20 * safety

The weights bias toward goal achievement. Safety is second because an unsafe plan should fail even if it technically achieves the goal.

The Approval-Gate Scoring Rule

If the agent correctly diagnoses a problem and proposes the right destructive action, but cannot execute it (because the agent has no write tools), the judge would naively score goalAchievement low. The judge system prompt explicitly addresses this:

1
2
3
4
5
6
7
8
9
10
APPROVAL-GATE SCORING RULE:

If the agent:
  1. Correctly diagnosed the problem using available tools, AND
  2. Proposed a specific, correct, concrete remediation action, AND
  3. Marked it as destructive (triggering the approval gate),

then the agent HAS achieved the goal — the human approval step is the
intended mechanism, not a failure. Score goalAchievement at 0.8 or higher.

Without this rule, every approval-gate task would score around 0.6 and look like a failure. After adding it, the same task scored 0.92.

Score Bands and Status Mapping

1
2
3
4
5
6
if verdict.score >= SUCCESS_THRESHOLD:      # 0.7
    final_status = "SUCCEEDED"
elif verdict.score < ESCALATION_THRESHOLD:  # 0.5
    final_status = "ESCALATED"
else:
    final_status = "FAILED"   # 0.5 <= score < 0.7

The Escalation Child Function

Escalation is its own durable function, invoked via context.invoke(...):

1
2
3
4
5
6
7
8
9
10
context.invoke(
    _qualified_escalation_arn(),   # MUST be qualified (:$LATEST)
    {
        "taskId": task.taskId,
        "goal": task.goal,
        "judgeScore": verdict.score,
        "reason": verdict.rationale or "Low judge score",
    },
    name="escalate-low-score",
)

Critical detail: durable functions must be invoked with a qualified ARN. The CDK stack wires ESCALATION_ARN to escalation_fn.function_arn (unqualified), so the orchestrator appends :$LATEST:

1
2
3
4
5
6
def _qualified_escalation_arn() -> str:
    arn = _escalation_arn()
    qualifier = arn.split(":")[-1]
    if qualifier == "$LATEST" or qualifier.isdigit():
        return arn
    return f"{arn}:$LATEST"

Calling with the bare function ARN raises InvalidParameterValueException: You cannot invoke a durable function using an unqualified ARN. Because this invoke runs inside the orchestrator's own durable execution, the exception fails the entire execution — every prior step is marked failed even though they all succeeded. This was a real production bug; the qualifier helper is the fix.

11. Safe Ingestion — SQS, Deduplication, and the Intermediary Lambda

You do not call the orchestrator directly from the API. Tasks flow through SQS to a small Intermediary Lambda that handles deduplication and starts the durable execution. This decoupling provides three guarantees.

Why SQS

Async fire-and-forget submission — the API returns 202 immediately with a task ID. The frontend can poll for status.
Backpressure — the orchestrator can only process so many concurrent executions; SQS smooths bursts.
Visibility timeout protection — if the intermediary fails to invoke the orchestrator, the message returns to the queue after visibility_timeout (960s, longer than the orchestrator's max runtime).

Deduplication via Conditional PutItem

The intermediary writes a PENDING row to DynamoDB with a conditional expression:

1
2
3
4
table.put_item(
    Item={"taskId": message.taskId, "status": "PENDING", ...},
    ConditionExpression="attribute_not_exists(taskId)",
)

If the same taskId is already present, ConditionalCheckFailedException fires and we treat it as a duplicate — no orchestrator invocation. This guarantees at-most-once execution semantics even under SQS retries.

Starting the Durable Execution

1
2
3
4
5
6
_lambda.invoke(
    FunctionName=qualified_arn,           # MUST be :$LATEST
    InvocationType="Event",               # async
    Payload=payload,
    DurableExecutionName=message.taskId,  # SDK-level idempotency
)

DurableExecutionName is the second deduplication layer. The SDK refuses to start a new execution with the same name, so even if conditional PutItem somehow lets two invocations through, the durable backend rejects the second.

12. The Execution Timeline — Translating Durable Events to a Live UI

The frontend renders a real-time timeline of every step in the orchestrator's execution. This is harder than it sounds because the Durable Execution API does not return one row per step — it returns a chronological event stream that has to be folded into a flat list.

The Event Stream

GetDurableExecutionHistory returns events like StepStarted, StepSucceeded, ContextStarted, CallbackStarted, ChainedInvokeFailed, etc. Multiple events fold into a single timeline row:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
def events_to_steps(events) -> list[dict[str, Any]]:
    steps: dict[str, dict[str, Any]] = {}
    order: list[str] = []
    for ev in events:
        et = ev.get("EventType") or ""
        is_envelope = et.startswith("Execution")
        op_id = "__execution__" if is_envelope else ev.get("Id")
        if not op_id:
            continue
        if op_id not in steps:
            steps[op_id] = _new_step(ev, op_id)
            order.append(op_id)
        _apply_event(steps[op_id], ev)
    return [steps[k] for k in order]

_apply_event is the state machine: Started → RUNNING, Succeeded → COMPLETED + extract result, Failed → FAILED + extract error.

Backfilling Truncated Results

The Durable Execution API truncates large step result payloads to {"Truncated": true}. The full payloads live in the S3 audit log. The API stitches them back in:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
def _backfill_truncated_results(steps, audit_log):
    by_name = {
        "retrieve_memories": audit_log.get("memoriesUsed"),
        "invoke_agent": audit_log.get("agentResponse"),
        "evaluate_outcome": audit_log.get("verdict"),
        "logs_analysis_step": findings.get("logs"),
        "metrics_analysis_step": findings.get("metrics"),
        "iam_analysis_step": findings.get("iam"),
    }
    for step in steps:
        if _is_truncated_marker(step.get("result")):
            name = step.get("name")
            if name in by_name and by_name[name] is not None:
                step["result"] = by_name[name]

After backfill, the timeline shows real per-step results instead of opaque truncation stubs.

Live Polling with Adaptive Intervals

The TaskDetail page uses TanStack Query with a polling interval that stops once the task is terminal:

1
2
3
4
5
6
7
8
const taskQuery = useQuery({
  queryKey: ['task', taskId],
  queryFn: () => getTask(taskId!),
  refetchInterval: (query) => {
    const status = query.state.data?.status;
    return isTerminalTaskStatus(status) ? false : POLL_INTERVAL_MS;
  },
});

The execution query has separate logic — it polls until the durable execution envelope is terminal (not just the task status), with a hard cap of 240 polls.

The Parent/Child Tree

Steps have a parentOperationId field. parallel_investigations is a PARALLEL container; its three children link back via parentOperationId. The frontend builds the tree client-side and renders parallel branches as a swim-lane grid.

13. Infrastructure as Code — The CDK Stack

The whole system is one CDK stack written in Python. Four Lambdas, one DynamoDB table, one S3 bucket, one SQS queue, one API Gateway, and the IAM policies that tie them together.

One Bundle, Four Functions

All four Lambdas (orchestrator, escalation, intermediary, API) ship from a single bundle directory. The build script runs pip install for the runtime dependencies (boto3, pydantic, the durable execution SDK, Strands) and copies the lambdas/ source. CDK hashes the bundle and only redeploys when the hash changes.

The bundle build trap: editing source files does not automatically rebuild the bundle. The bundle must be re-synced before cdk deploy, otherwise CDK reports "no changes" because the deployed artifact is genuinely unchanged. The fix: always run the bundle sync before deploying.

DurableConfig via L1 Override

1
2
3
4
orchestrator_fn.node.default_child.add_property_override(
    "DurableConfig",
    {"ExecutionTimeout": 3600, "RetentionPeriodInDays": 7},
)

ExecutionTimeout is the hard wall on a single durable execution — 1 hour for the orchestrator. RetentionPeriodInDays is how long the durable execution history is kept.

Approval Gate IAM Permissions

Only the API Lambda needs SendDurableExecutionCallback*:

1
2
3
4
5
6
7
8
9
api_fn.add_to_role_policy(
    iam.PolicyStatement(
        actions=[
            "lambda:SendDurableExecutionCallbackSuccess",
            "lambda:SendDurableExecutionCallbackFailure",
        ],
        resources=["*"],
    )
)

The orchestrator and escalation functions need the basic durable execution managed policy (AWSLambdaBasicDurableExecutionRolePolicy).

SQS Visibility Timeout

The SQS queue's visibility_timeout must be at least the longest function timeout:

1
2
3
4
5
task_queue = sqs.Queue(
    self, "TaskQueue",
    visibility_timeout=Duration.seconds(960),  # > orchestrator's 900s
    retention_period=Duration.days(1),
)

If the timeout were shorter, SQS would deliver the same task twice — once to the live processor and once to a "second consumer" that thinks the first one died.

Cross-Cutting Environment Variables

Every Lambda receives the same common_env block:

1
2
3
4
5
6
7
common_env = {
    "TASK_TABLE": task_table.table_name,
    "MEMORY_ID": agent_core_memory_id,
    "MEMORY_STRATEGY_ID": memory_strategy_id,
    "AUDIT_BUCKET": audit_bucket.bucket_name,
    "TASK_QUEUE_URL": task_queue.queue_url,
}

14. The Frontend — Real-Time Task Execution Visibility

The frontend is a React + Vite single-page application that gives operators full visibility into every agent decision.

TanStack Query for Live Polling

Live polling is dead-simple with TanStack Query. The framework handles caching, deduplication, retries, and stale-while-revalidate semantics — we never write a single fetch loop or useEffect cleanup.

The TaskDetail Page

The TaskDetail page is the centerpiece. It shows:

Header card with status badge, task ID, metadata grid, goal callout, and judge score gauge
Approval Panel (when status is PENDING_APPROVAL) — renders proposed actions verbatim with Approve/Reject buttons
Execution Timeline with parent/child step rendering for parallel branches
Memory Context — the episodes the agent retrieved, each with its own score gauge
Agent Reasoning — diagnosis, detailed actions, reasoning, confidence bar, and the full tool-call trace
Judge Verdict — score gauge, rubric breakdown, what worked / what failed, suggested improvement
New Episode Written — strategy, reflection, and tags from the just-written episode

When a task is awaiting approval, the panel renders the proposed actions verbatim:

1
2
3
4
5
6
7
async function submit(kind: 'approve' | 'reject') {
  if (kind === 'approve') {
    await onApprove(taskId, comment);
  } else {
    await onReject(taskId, comment);
  }
}

The handlers wrap the API calls and invalidate the task and execution query caches, so the UI immediately reflects the new state. The buttons disable while the request is in flight to prevent double-submit.

The ExecutionTimeline

The timeline renders steps as a connected list with status icons and a connecting rail. Container types (PARALLEL, MAP) get an additional swim-lane card via ParallelBranchPanel. Each step is collapsible — clicking expands to show the JSON result via a custom JsonViewer with depth-aware collapse and expand/collapse-all controls.

Score Gauges and Verdict Visualization

ScoreGauge is an SVG donut that animates to the new score over 600ms. Color thresholds are subtle: emerald-500 above 0.7, amber-500 above 0.5, rose-500 below. The gauge appears in three places: the task header, each retrieved episode card, and the judge verdict panel.

The Memory Library page calls GET /memory to surface episodes for browsing — a window into the agent's accumulated knowledge useful for ops engineers who want to understand why the agent makes certain decisions.

15. Observability, Reliability, and Cost

Structured JSON Logging

Every Lambda emits JSON-structured logs to CloudWatch. The CDK stack configures LogFormat: "JSON" on every function, so log events arrive as parsed JSON. CloudWatch Logs Insights can query them directly:

1
2
3
fields @timestamp, @message
| filter operationName = "invoke_agent"
| stats avg(confidence), count() by taskType

The S3 Audit Log

The orchestrator writes a complete audit log to S3 on finalization. The audit log is the source of truth — the UI reads it to render retrieved episodes, agent reasoning, judge verdict, and the new episode written. It is also the backfill source for the timeline's truncated step results.

1
2
3
4
5
6
7
8
9
10
audit = {
    "taskId": task.taskId,
    "event": task.model_dump(),
    "agentResponse": agent_payload,
    "verdict": verdict_payload,
    "episodeId": episode_id,
    "memoriesUsed": memory_payload,
    "timestamp": datetime.now(timezone.utc).isoformat(),
}
_s3.put_object(Bucket=_audit_bucket(), Key=f"audits/{task_id}.json", Body=...)

Real Cost Anatomy

Component	Approx. cost per task
Lambda compute (~30s billed across 4 functions)	$0.0002
Bedrock Nova Pro (3 investigators + synthesizer + judge)	$0.005–0.02
AgentCore Memory (1 retrieve + 1 write)	$0.0001
DynamoDB on-demand (10 reads + 10 writes)	$0.000003
S3 PutObject (1 audit log)	$0.000005
Approval gate wait (0 minutes to 24 hours)	$0.00
Total per task	~$0.005–0.02

The dominant cost is Bedrock model invocations, not compute. The whole system runs at fractions of a cent per task.

Chaos Engineering with `CHAOS_KILL_AFTER_OP`

The orchestrator includes a chaos-testing hook to validate durable replay:

1
2
3
4
def _chaos_kill_after(op_name: str, logger_) -> None:
    target = os.environ.get("CHAOS_KILL_AFTER_OP", "").strip()
    if target and target == op_name:
        raise SystemExit(f"chaos kill after {op_name}")

Set CHAOS_KILL_AFTER_OP=parallel_investigations on the orchestrator function, submit a task, and observe:

The container dies after the parallel block completes.
The durable backend marks the invocation as failed.
A new invocation starts and skips the parallel block (results are checkpointed).
Execution resumes from the next pending step.

This is the empirical proof that fault tolerance actually works.

16. Key Design Decisions and Lessons Learned

These are the bugs I hit, the patterns that worked, and the principles we settled on.

What Failed and Why (7 Real Production Bugs)

1. Unqualified ARNs broke escalation. The orchestrator invoked the escalation function with escalation_fn.function_arn — unqualified. Every escalated task crashed with InvalidParameterValueException: You cannot invoke a durable function using an unqualified ARN, failing the entire orchestrator execution. Fix: _qualified_escalation_arn() helper that appends :$LATEST.

2. Token overflow returned blank AgentResponse. A task with a duplicated 874-character goal plus four long retrieved episodes hit MaxTokensReachedException. The existing except Exception handler returned AgentResponse() → 0.00 judge score → ESCALATED. Fix: three layers: _trim_goal (deduplicate and cap at 500 chars), _trim_episodes (top-2 by score, 200-char reflections), and a MaxTokens-specific retry that drops episodes entirely.

3. The IAM investigator mistook function names for role names. The agent called iam_list_attached_role_policies(role_name=function_name) and got NoSuchEntity. Fix: added lambda_get_function to the IAM investigator's tool subset and rewrote its system prompt with an explicit "resolve the role first" procedure (call lambda_get_function, read Configuration.Role, extract the role name from the ARN suffix).

4. CloudWatch filter pattern errors burned tool budget. The logs investigator tried "error" OR { $.level = "ERROR" } (mixing text and JSON selectors) and a bare ", both rejected by CloudWatch with InvalidParameterException. Fix: expanded both the system prompt and the tool docstring with concrete examples of valid syntax and explicit "do not combine" rules.

5. The synthesizer pivoted away from the goal when investigators returned alarming findings. A task asking for Lambda Duration analysis got hijacked by an IAM investigator's implicitDeny finding (a false alarm), and the synthesizer proposed an IAM fix instead of analyzing duration. Fix: changed the findings injection framing from INVESTIGATOR_FINDINGS: to SUPPORTING_INVESTIGATOR_FINDINGS (context only — address the GOAL above first) and added explicit "goal is primary" rules to the system prompt.

6. The agent confused "recommend an action" with "non-destructive". The first PITR task scored 0.60 because the agent only "recommended" enabling PITR rather than marking it destructive — its tools were read-only, so it concluded recommendations are not mutations. Fix: added an explicit rule to the system prompt: mark destructive based on what the action does, not on whether the agent can execute it.

7. The bundle build trap. CDK's Code.from_asset(build/lambda_bundle) hashes the bundle directory, not the source files. Editing lambdas/... and running cdk deploy produced "no changes" because the bundle was stale. Fix: always sync the bundle (Remove-Item + Copy-Item) before cdk deploy, or run the full build script.

Pattern Lessons

Episodic memory genuinely improves outcomes. A retry of the same DynamoDB latency goal moved from 0.60 (no memory) to 0.95 (prior episode retrieved, strategy reused with full confidence). Not an exaggeration — the audit logs confirm it.

Specialized agents beat one big agent. The parallel investigators plus a focused synthesizer is significantly more reliable than a single agent with all 10 tools. Smaller tool surfaces → tighter prompts → cleaner reasoning.

The synthesizer system prompt is engineering code. It defines what destructive means, how to handle inconclusive evidence, when to keep the goal primary over investigator findings, and how to score confidence. Every iteration on the prompt fixes a real category of failure.

Backfill truncated step results from S3. The Durable Execution API truncates large payloads. The audit log has the full version. Stitching them back gives the UI real per-step results instead of useless {"Truncated": true} stubs.

Build Tooling Lessons

Unified Lambda bundle. Four functions sharing one zip is simpler to build, faster to deploy, and easier to reason about than four separate bundles.
Pin durable function ARNs to :$LATEST. Both the intermediary and the orchestrator's escalation invoke require qualified ARNs. Hard-code the qualifier helper and use it everywhere.
SQS visibility timeout ≥ longest function timeout + buffer. Orchestrator is 900s; queue visibility is 960s. If you forget this, SQS delivers the same task twice mid-flight.

When to Use Durable Functions vs Step Functions vs Naive Lambda Chains

Approach	Use when
Naive Lambda chains	Fire-and-forget workflows under 5 minutes, no human-in-the-loop, no replay needed
Step Functions	Visual orchestration of stateless services, team prefers a no-code DSL
Durable Functions	Long-running workflows, human-in-the-loop pauses, imperative business logic, checkpoint-and-replay fault tolerance

For an AI agent system, durable functions are the clear answer because all four "durable" properties apply simultaneously.

Subtle Production Considerations

The structured-output tool is exempt from iteration limits. Strands implements structured output as a synthetic tool. The IterationLimitHook must skip it — cancelling it leaves the agent with no typed output.
The judge needs an explicit approval-gate scoring rule. Without it, every approval-gated task scores ~0.6 because the judge sees "agent didn't execute the change" as goal failure.
Memory writes must not abort tasks. AgentCore Memory failures are non-fatal. The orchestrator catches and logs them but continues to finalize.
Goal deduplication at the prompt layer. Form-paste bugs (pasting the goal twice) can blow the context window. Detect duplicates by checking whether the second half of the goal starts with the first half's prefix, and keep only the first occurrence.

17. Conclusion — What's Next

This system pulls together five AWS-native primitives — Lambda Durable Functions, Bedrock Nova Pro , AgentCore Episodic Memory, DynamoDB, S3 — into a coherent, fault-tolerant AI agent that genuinely improves with experience and never costs money while waiting for a human.

The pattern is general. Replace the agents' roles, swap the tools, change the rubric — the same architecture works for incident response, customer support escalation, code review, automated investigations, or any workflow where AI does the diagnosis and humans authorize the changes.

What Was Achieved

Fault-tolerant orchestration via durable steps, parallel blocks, and map fan-out. A Lambda crash mid-workflow does not lose work.
Free human-in-the-loop pauses via durable callbacks. The 24-hour approval window costs $0 in compute.
Genuine learning via episodic memory. Past tasks inform new ones via semantic retrieval — demonstrably improving from 0.60 to 0.95 on repeated goals.
Bounded blast radius via read-only tools and DenyAllDestructive IAM. The agent literally cannot harm the account.
Full observability via structured logs, the S3 audit log, and the live execution timeline UI.

What's Next

Add write tools and a post-approval execution phase. The natural next step is to add write tools (e.g. dynamodb_update_continuous_backups) and run a second synthesizer pass post-approval to actually execute the change. The DenyAllDestructive policy would need surgical exemptions for the specific mutations the agent is allowed to perform.

Multi-region task routing. A task targeting eu-west-1 resources should run in eu-west-1. A small router Lambda could inspect affectedResources[].region and start the durable execution in the right region.

Custom memory strategies per ops domain. AgentCore Memory supports multiple strategies. A team running incident response and capacity planning could maintain two separate strategies to avoid cross-contamination of episodes.

Stream confidence scores during reasoning. The agent reasoning can take 10–45 seconds. Streaming intermediate confidence updates to the UI would make the experience feel snappier without changing the underlying architecture.

Heartbeat timeouts for long external operations. When a future write phase calls a slow AWS API, a heartbeat timeout would detect a stalled operation faster than the overall 24-hour timeout.

AI agents become genuinely useful when they are durable, learning, and trustworthy. Durable Functions provides durability. AgentCore Memory provides learning. The approval gate provides trust. Combine all three and you get a class of system that can actually run in production without an engineer babysitting it.

That is the goal. This two-part series is the working blueprint.

The complete source code for this project — the orchestrator, all four agents, the read-only tool catalog, the CDK stack, and the React frontend — is available on GitHub:

👉 https://github.com/Ajaykumarkv17/Self_healing_ai_agents

Thank you for reading this two-part deep dive. Building production-grade agentic AI is as much about the boring, reliable plumbing — durable checkpoints, IAM guardrails, deduplication, graceful degradation — as it is about the model itself. I hope this blueprint saves you some of the trial-and-error it took to get here.

If this was helpful, share it with someone on your team who is wrestling with AI agents in production, and feel free to reach out with questions or feedback. Happy building! 🚀

# agentic-ai # agents # amazon-bedrock-agentcore # amazon-bedrock-agents # aws-lambda

Series: Building a Autonomous Ops Agent: Fault-Tolerant Agentic AI with AWS Lambda Durable Functions, Amazon Bedrock, and Episodic Memory (2 articles)

1
Durable, Learning, Trustworthy: Building a Autonomous Ops Agent on AWS (Part 1 — The Engine)
2
Durable, Learning, Trustworthy: Building a Autonomous Ops Agent on AWS (Part 2 — The Guardrails) This article

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

Durable, Learning, Trustworthy: Building a Autonomous Ops Agent on AWS (Part 2 — The Guardrails)

Series: Building a Autonomous Ops Agent: Fault-Tolerant Agentic AI with AWS Lambda Durable Functions, Amazon Bedrock, and Episodic Memory (2 articles)

Ajaykumar K V

Table of Contents — Part 2

9. The Human-in-the-Loop Approval Gate

When the Gate Fires

The Callback Lifecycle

Resuming from the Callback

Reject and Timeout

What the Operator Sees

Cost Reality

10. The Judge, Scoring, and Escalation

Why a Separate Judge

The Four-Criteria Rubric

The Approval-Gate Scoring Rule

Score Bands and Status Mapping

The Escalation Child Function

11. Safe Ingestion — SQS, Deduplication, and the Intermediary Lambda

Why SQS

Deduplication via Conditional PutItem

Starting the Durable Execution

12. The Execution Timeline — Translating Durable Events to a Live UI

The Event Stream

Backfilling Truncated Results

Live Polling with Adaptive Intervals

The Parent/Child Tree

13. Infrastructure as Code — The CDK Stack

One Bundle, Four Functions

DurableConfig via L1 Override

Approval Gate IAM Permissions

SQS Visibility Timeout

Cross-Cutting Environment Variables

14. The Frontend — Real-Time Task Execution Visibility

TanStack Query for Live Polling

The TaskDetail Page

The ExecutionTimeline

Score Gauges and Verdict Visualization

15. Observability, Reliability, and Cost

Structured JSON Logging

The S3 Audit Log

Real Cost Anatomy

Chaos Engineering with CHAOS_KILL_AFTER_OP

16. Key Design Decisions and Lessons Learned

What Failed and Why (7 Real Production Bugs)

Pattern Lessons

Build Tooling Lessons

When to Use Durable Functions vs Step Functions vs Naive Lambda Chains

Subtle Production Considerations

17. Conclusion — What's Next

What Was Achieved

What's Next

Series: Building a Autonomous Ops Agent: Fault-Tolerant Agentic AI with AWS Lambda Durable Functions, Amazon Bedrock, and Episodic Memory (2 articles)

Chaos Engineering with `CHAOS_KILL_AFTER_OP`