Your WAF Is Blocking the Robots You Actually Want
A practical look at agent readiness — and the security blind spot hiding in plain sight - all with AWS edge services
Your WAF Is Blocking the Robots You Actually Want
A practical look at agent readiness — and the security blind spot hiding in plain sight
There's a quiet shift happening on the web. The first visitor to many sites is no longer a human — it's an AI agent probing for context: what does this site do, can I call its API, how do I authenticate?
Most sites aren't ready for this. Including, as I recently discovered, a demo site I built specifically to showcase modern web security.
I came across ora.run , a tool that scores websites on "agent readiness" — how well a site can be discovered, understood, and used by AI agents. On a whim, I ran it against my demo. The score came back: 14 out of 100.
That number was humbling enough to take seriously.
What followed was a few hours of iterative improvements, WAF log analysis, and a few surprises. I ended at 49/100 — far from perfect, but a meaningful jump. This post is about what I changed, what actually moved the needle, and what I hadn't anticipated.
How ora.run Scores Your Site
ora.run evaluates five dimensions, each worth a different number of points:
| Category | Max |
|---|---|
| Discovery | 20 |
| Identity | 20 |
| Auth & Access | 30 |
| Agent Integration | 20 |
| User Experience | 10 |
The Auth & Access category is worth the most — 30 points — and rewards sites that have real authentication infrastructure in place. Everything else requires deliberate work.
One feature of ora.run I found particularly useful: after scanning, an AI agent actually attempts to integrate with your site and leaves a review. It read:
"Attempted to integrate with the platform as an AI agent and immediately hit a wall — there's no public API endpoint, no OpenAPI documentation, and no developer portal to even discover how to get started."
That's not a vague scoring algorithm — that's a concrete failure description. It's also a useful reframe: instead of asking "what score do I want?", ask "what does an agent actually need to use my site?"
First: What Agents Are Actually Looking For
Before changing anything, I wanted to understand exactly what ora's crawler was probing. The WAF logs told the story clearly. Here's what it requested:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
/llms.txt
/llms-full.txt
/.well-known/llms.txt
/robots.txt
/openapi.json
/swagger.json
/sitemap.xml
/.well-known/agent.json
/.well-known/agent-card.json
/.well-known/mcp.json
/.well-known/agent-skills/index.json
/api
/api-reference
/docs
/v1
/mcp
/guides/llms.txt
/developers/llms.txt
/developer.md
/api.md
/agent.mdThis is the emerging convention for agent-readable infrastructure — a layer of well-known paths that agents check before they do anything else. Most of these are analogous to things we understand:
robots.txt for crawlers, /.well-known/ for protocol discovery, openapi.json for API consumers.The Easy Wins: Tier 1 Static Files
Most of these files are static — no compute required, just content to serve. I created and deployed them all:
/llms.txt — A plain-text summary of your site written for LLMs. Think robots.txt but for AI agents: your site's purpose, data model, key endpoints, and any context that helps an agent decide whether and how to use you./llms-full.txt — An extended version with richer context./.well-known/agent.json — Machine-readable identity: who you are, what you offer, what protocols you support./.well-known/agent-card.json — A2A (Agent-to-Agent) protocol discovery./.well-known/mcp.json — MCP (Model Context Protocol) discovery. Even a minimal stub that points to your real API is better than a 404./openapi.json — If you have an API, this is the most important file on the list. A well-structured OpenAPI 3.0 spec is what allows an agent to actually call your endpoints. Without it, everything else is just decoration./robots.txt — Standard, but worth auditing. Make sure you're not accidentally blocking AI user-agent strings.Schema.org JSON-LD in
<head> — Structured data markup that feeds AI search indexing. For a data-oriented site, Dataset type is appropriate.Total deployment time: about 30 minutes. Score impact: +35 points.
I'll be honest — I didn't expect static files to move the needle that much. It's a sign that most sites are starting from nearly zero on agent readiness. The baseline is low, which means the early movers have a genuine opportunity.
The Unexpected Problem: Your WAF
This is the part that surprised me most — and it's why I wanted to write this post.
After deploying the Tier 1 files, I re-ran the scan and checked the WAF logs. What I saw was this:
1
2
3
4
5
6
CHALLENGE /guides/llms.txt
CHALLENGE /api
CHALLENGE /docs
CHALLENGE /swagger.json
BLOCK /
CHALLENGE /mcpThe WAF was challenging or blocking ora's crawler on many of the paths I'd just created. The files existed. They were correct. But the agents scanning for them were being stopped before they could read anything.
This makes sense from a WAF perspective: Bot Control is designed to identify and challenge automated traffic. An AI agent probing dozens of well-known paths in rapid succession looks, behaviorally, very similar to a scraper. Bot Control is doing exactly what it's supposed to do.
The instinct here — and I've seen this pattern often — is to create an Allow rule for the agent discovery paths. Something like: "if the URI starts with
/.well-known/ or equals /llms.txt, allow it." It seems tidy. It's also a security mistake.An explicit Allow rule at the top of your rule priority list bypasses your entire rule stack for matching requests — including IP reputation lists, anonymous IP checks, and rate limiting. A sophisticated attacker who identifies your allowlist can craft requests to those paths as a bypass vector. The paths you meant to exempt become a gap in your defenses.
The better pattern uses two rules working together:
Rule 1: Count and label, don't allow
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
{
"Name": "label-agent-paths",
"Priority": 1,
"Statement": {
"OrStatement": {
"Statements": [
{ "ByteMatchStatement": { "SearchString": "/llms.txt", "PositionalConstraint": "EXACTLY" }},
{ "ByteMatchStatement": { "SearchString": "/openapi.json", "PositionalConstraint": "EXACTLY" }},
{ "ByteMatchStatement": { "SearchString": "/.well-known/", "PositionalConstraint": "STARTS_WITH" }},
{ "ByteMatchStatement": { "SearchString": "/api", "PositionalConstraint": "STARTS_WITH" }},
{ "ByteMatchStatement": { "SearchString": "/docs", "PositionalConstraint": "STARTS_WITH" }}
]
}
},
"Action": { "Count": {} },
"RuleLabels": [{ "Name": "agent:discovery:path" }]
}This rule fires on agent discovery paths, increments a counter (useful for metrics), and attaches a label to the request. Critically,
Count is a non-terminating action — the request continues through the rest of your rule stack. IP reputation rules still apply. Rate limiting still applies. Only Bot Control needs to be told to step aside.Rule 2: Scope-down Bot Control
Add a scope-down statement to the Bot Control managed rule group configuration:
1
2
3
4
5
6
7
8
9
10
11
12
{
"ScopeDownStatement": {
"NotStatement": {
"Statement": {
"LabelMatchStatement": {
"Scope": "LABEL",
"Key": "agent:discovery:path"
}
}
}
}
}Bot Control now skips any request already carrying the
agent:discovery:path label. Your discovery paths are accessible. Your application remains fully protected everywhere else.That said, exempting paths from Bot Control doesn't mean leaving them unprotected. You should still have downstream controls in place — rate-based rules to prevent individual IPs from hammering your discovery endpoints, and the AWS WAF Anti-DDoS managed rule group to catch volumetric abuse patterns. The scope-down removes the bot challenge, not the safety net.
⚠️ Important: This pattern is appropriate for truly static discovery paths — files that exist purely to be read, with no backend logic behind them. UseEXACTLYmatching wherever possible. Be careful withSTARTS_WITHon functional paths like/api— if real endpoints live under that prefix, you are exempting them from Bot Control scrutiny entirely. In that case, either tighten the scope toEXACTLYmatches for each individual discovery file, or ensure those endpoints are protected by authentication so that Bot Control is not your last line of defense.
What Moved, What Didn't
After both changes — static files and WAF scope-down — the score settled at 49.
The biggest contributors:
/openapi.json— Agent Integration score improved immediately/.well-known/agent.json+/.well-known/agent-card.json— lifted Identity- WAF scope-down — removed the CHALLENGE noise that was making valid paths appear unreachable
/llms.txt— baseline Discovery signal (full indexing takes longer)
What didn't move:
Discovery (2/20) is the stubborn category. Scoring well here requires your site to appear in AI search results when agents look for services in your domain. Static files help signal that you're agent-ready, but actually getting indexed takes crawl cycles and time. There's no shortcut.
User Experience (3/10) reflects that ora's crawler was blocked on the homepage itself — Bot Control flagging the behavioral signature of an automated scan. I left this intentionally: for a security demo, homepage protection is the point. In a production application you'd likely want a more nuanced approach.
Takeaways
A few things I'd carry forward from this:
Agent readiness is the new SEO. In the early 2000s, sites that invested in search engine optimization before it was mainstream captured significant advantages. The same dynamic is playing out now with AI agents. The bar is currently very low — most sites score in the 10–20 range.
The Tier 1 files are nearly free. Static content served from a CDN costs almost nothing. The return per hour of effort here is unusually high right now, precisely because adoption is still early.
Audit ythe WAF logs before assuming you're accessible. If you run Bot Control, check what ora (or any agent scanner) actually sees when it visits your site. The experience may be very different from what you'd expect browsing it yourself.
Prefer Count + Label + scope-down over Allow rules. Allow rules in WAF have a way of expanding over time and becoming unintentional security gaps. The label pattern achieves the same result without punching a hole in your rule stack.
An OpenAPI spec is non-negotiable. Agents can't use an API they can't read. A well-maintained OpenAPI 3.0 spec is the most impactful single artifact you can produce for agent integration.
What's Next
Getting from 49 to 80+ will require more than static files:
- A real
/askendpoint for natural-language queries - MCP server integration for richer agent interactions
- API key auth for programmatic agent access
- Time, for AI search indexing to catch up
None of that is trivial. But it builds on a foundation that now exists — which is where I'd suggest anyone start.
Tzoori Tamam is a Principal Solutions Architect Specialist for Edge Services at AWS, based in Israel. He focuses on CloudFront, AWS WAF, and AWS Shield. The views expressed here are his own.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article