AIdeas: AI Runtime Assurance
A lightweight flight recorder that sits alongside AI agents on AWS, capturing every action and using Amazon Bedrock when available to evaluate security policies in real time, with rule-based fallback. Produces structured session reports with severity scores and plain-English explanations.
App Category
Workplace Efficiency - Security and compliance tooling for AI agent operations.
My Vision
Every organization deploying AI agents faces the same question: "What did the agent actually do?"
Current security tools see infrastructure events - "Lambda function accessed S3" - but they lack agent context. They don't know why the AI agent made that call, whether it was part of a legitimate workflow, or whether it's the first step in a data exfiltration pattern.
AI Runtime Assurance is a lightweight flight recorder that sits alongside your AI agent on AWS. It captures every tool call, API request, and file operation. When Amazon Bedrock is available, it evaluates each action against natural-language security policies, assigning confidence-scored severity levels and plain-English explanations; otherwise the system falls back to rule-based analysis. The result is an audit-ready session report that gives security teams a structured view of agent behavior without reconstructing events manually.
The goal is to transform agent oversight from "trust and hope" to "verify and explain."
Why This Matters
The Trust Gap Is Real
Organizations want to deploy AI agents for document processing, customer interactions, and workflow automation. But security teams have no visibility into what these agents actually do at runtime. Traditional tools like CloudTrail and GuardDuty see the infrastructure layer - not the agent layer.
This creates a deadlock: engineering teams want to ship agents, security teams want evidence of what agents do, and neither side has the tools to bridge the gap.
Who Benefits
- Security Operations teams who need to audit AI agent behavior without reading code
- Compliance teams who need evidence trails for regulatory reporting
- DevOps engineers who deploy AI agents and need to prove they're operating within policy
The Scale Problem
Manual security reviews don't scale. When you have dozens of agents executing hundreds of actions per hour, you need automated policy evaluation with structured output - not another dashboard to watch. AI Runtime Assurance produces session reports with severity counts, flagged events, and actionable recommendations automatically.
How I Built This
Architecture
AI Runtime Assurance is a Python-based sidecar proxy built with FastAPI. It runs alongside your AI agent and intercepts three types of operations:
1
2
3
4
5
6
7
8
9
10
11
┌──────────────┐ ┌─────────────────────────────────────┐
│ AI Agent │───\>│ AI Runtime Assurance Proxy │
│ (any agent) │ │ │
└──────────────┘ │ /tools/http → HTTP capture │
│ /tools/aws → AWS API capture │
│ /tools/file → File op capture │
│ │
│ Policy Engine (rules + Bedrock) │
│ Evidence Store (JSONL + S3) │
│ Report Generator │
└─────────────────────────────────────┘The agent routes its tool calls through the proxy instead of calling APIs directly. The proxy executes the request, logs the event with full metadata, evaluates it against security policies, and returns the result to the agent transparently.
Development with Kiro
I used Kiro's spec-driven development workflow to design the system before writing code. The
.kiro/specs/ai-runtime-assurance/ directory contains three artifacts that Kiro generated and I iterated on:- requirements.md - 4 functional requirements with 24 acceptance criteria across 4 categories (AI analysis, policy system, reporting, cost optimization)
- design.md - system architecture, module boundaries, data flow
- tasks.md - 12 implementation tasks with dependency tracking
This approach kept scope tight. When I was tempted to add features (adaptive learning, Lambda deployment, real-time dashboards), the spec reminded me what was in scope and what wasn't.
Amazon Bedrock Integration
The
BedrockPolicyEngine (~600 lines) is the core intelligence:- Auto-discovery: On startup, it probes a prioritized list of candidate models via the Converse API (Nova Lite, Nova Micro, Titan, Claude Haiku) and uses the first available one.
- Dual API support: If the standard Converse API is unavailable (e.g. account-level authorization pending), the engine automatically falls back to Bedrock's OpenAI-compatible API using API keys. This broadens the set of Bedrock-hosted models available through the same analysis flow, including providers like Mistral, DeepSeek, and OpenAI GPT-OSS.
- Semantic analysis: Each agent action is evaluated against natural-language security policies. Bedrock understands intent - distinguishing between "agent accessed S3 for a config file" (LOW) and "agent read /etc/passwd then called an external API" (HIGH).
- Context window: The engine tracks the last 5 session events and includes them in the analysis prompt, enabling detection of multi-step attack patterns like data exfiltration (read sensitive file → send to external domain).
- Cost guardrails: Token usage is tracked per month with circuit breakers at 90% of Free Tier limits (45K input / 9K output tokens). Intelligent caching prevents redundant analysis of similar events.
- Graceful degradation: Three tiers of analysis: (1) Bedrock Converse API, (2) Bedrock OpenAI-compatible API, (3) rule-based fallback. The system always produces useful output.
The Fallback Engine
The rule-based fallback isn't a toy - it correctly classifies:
- External domain access → HIGH severity
- Sensitive file paths (
/etc/,/.aws/,/root/) → HIGH severity - Internal/allowed domains (
*.amazonaws.com) → LOW severity - Non-sensitive operations → LOW severity by default
This means the system always produces useful output, even without Bedrock access.
AWS Free Tier Compliance
Every component runs within Free Tier limits:
- Bedrock: 50K input + 10K output tokens/month (demo uses <5% of limits)
- EC2: t2.micro or local execution
- S3: Report storage well under 5GB
- No Lambda, no NAT, no paid services
Demo
The demo runs in-process using FastAPI's TestClient - no server subprocess needed. The intended entry point is:
1
DEMO_MODE=1 python demo_showcase.pyWhat the demo captures
The demo simulates six realistic agent scenarios in a single session:
- External API call (httpbin.org) - Flagged as HIGH severity - external domain detected
- Internal AWS call (s3.amazonaws.com) - Classified as LOW - allowed domain
- File write + read (temp directory) - Logged as file operations for the session report
- Sensitive file access (/etc/hosts) - Flagged as HIGH - sensitive path detected
- AWS STS GetCallerIdentity - Classified as LOW - standard AWS operation
- Rapid-fire HTTP requests (5 quick calls) - Logged as repeated external activity in the same session
Demo run covering HTTP, AWS, and file-operation scenarios in a single monitored session.
Session report output
Each successful demo run produces a structured report with:
- total monitored actions
- severity counts
- flagged events with explanations
- recommendations for follow-up review
Example session report generated from the monitored event stream.
Evidence files produced
A successful demo run generates:
data/events.jsonl- Chronological event log (11 events, one per line)data/reports/sess_*.json- Structured session report with severity counts, flagged events, and recommendations- Session statistics with events/second, unique domains, and AWS services used
Key endpoints
| Endpoint | Purpose |
|---|---|
GET /health | Server health + storage status |
POST /session/start | Begin monitored session |
POST /tools/http | Execute + log HTTP request |
POST /tools/aws | Execute + log AWS API call |
POST /tools/file | Execute + log file operation |
POST /session/{id}/end | End session + generate report |
GET /reports/{id} | Retrieve session report |
GET /reports/{id}/statistics | Session analytics |
What I Learned
1. Bedrock model access requires account-level authorization beyond IAM
Having
AmazonBedrockFullAccess attached to your IAM user is necessary but not sufficient. Bedrock models require a separate account-level authorization step before any model - including Amazon's own Nova and Titan - can be invoked via the Converse API. I discovered this after the IAM policy was in place but every model returned authorizationStatus: NOT_AUTHORIZED. The solution was building multi-path resilience: the engine tries the Converse API first, then Bedrock's OpenAI-compatible API (which uses API keys and bypasses the account-level gate), and finally the rule-based fallback. This layered approach means the system works regardless of which Bedrock access path is available.2. The Converse API is genuinely model-agnostic
Bedrock's Converse API uses the same request/response shape across all providers (Amazon Nova, Anthropic Claude, Meta Llama). This let me write a single
_call_bedrock() method and an auto-discovery loop that probes candidates until one works. No model-specific code paths. This is how vendor lock-in should be avoided.3. Kiro specs prevented scope creep
The biggest risk in a competition with a fixed deadline is gold-plating. Kiro's spec-driven workflow gave me a written contract with myself about what was in scope. When I wanted to add real-time WebSocket streaming or a React dashboard, the spec said "not in scope" - and I listened. The result shipped.
4. FastAPI TestClient eliminates demo fragility
Every previous demo attempt involved starting a server subprocess, waiting for it to bind, running a client, and hoping nothing timed out. TestClient runs the entire app in-process, which removes the extra server orchestration and most of the port-binding fragility. If I were starting over, I'd build the TestClient demo first and add server deployment later.
5. Security monitoring is a content problem, not a technology problem
The hardest part wasn't building the proxy or integrating Bedrock. It was defining what counts as suspicious. A file read is normal. A file read of
/etc/passwd followed by an HTTP POST to an external domain is a data exfiltration pattern. The difference is context - and that's exactly what AI analysis provides that rule engines can't match at scale.Built with Kiro and Amazon Bedrock on AWS Free Tier for the 10,000 AIdeas Competition.
GitHub: tsyrulb/Agent-Runtime-Assurance-Wrapper
GitHub: tsyrulb/Agent-Runtime-Assurance-Wrapper
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article