AWS Builder Center

Builder Center

When AI Agents Knock on the Web: Why Websites Are Blocking Them

When AI Agents Knock on the Web: Why Websites Are Blocking Them

AI agents are evolving from chatbots into autonomous software that can browse, shop, book and transact on behalf of users. But the web was built for humans, creating a new challenge around agent identity, authorization, security and trust.

Rishabh Kankariya

Cloud Computing Student | AWS Enthusiast | Exploring Cloud, DevOps & AI

When AI Agents Knock on the Web: Why Websites Are Blocking Them

Introduction

AI agents are moving beyond answering questions.
Instead of simply telling us which flight is cheapest, which laptop to buy, or which restaurant has availability, the new generation of AI agents is expected to perform the task:
1
2
3
4
5
6
7
8
9
Find a flight
↓
Compare prices
↓
Choose the best option
↓
Fill passenger details
↓
Book the flight
The problem is that the web was not designed for this.
Websites were primarily built around a simple assumption:
A human is sitting behind the browser.
Now there is a second type of user: software acting on behalf of a human.
And many websites are not ready for it.
Recent reporting shows personal AI agents being blocked or disrupted by websites including Amazon, airlines, Yelp, eBay, Zillow and other services. Some blocks are deliberate, while others are the result of traditional anti-bot systems that were never designed to distinguish a legitimate personal AI agent from a scraper or malicious bot.
This is creating a new infrastructure problem:
How can a website know that an AI agent is legitimate, who it represents, and what it is authorized to do?
That question could become one of the foundations of the emerging Agentic Internet.

1. The Web Was Built for Humans

Consider a traditional web interaction:
1
2
3
4
5
6
7
8
9
Human
↓
Browser
↓
Website
↓
Button
↓
Action
A human opens a website, logs in, searches for something and clicks a button.
Websites have developed decades of security infrastructure around this model:
  • Cookies
  • Sessions
  • CAPTCHAs
  • Rate limits
  • IP reputation
  • Browser fingerprints
  • User-Agent strings
  • Login authentication
  • Fraud detection
These systems generally attempt to answer:
"Is this request coming from a legitimate user or an unwanted automated process?"
AI agents complicate this question.
A personal AI agent might genuinely represent a real customer.
For example:
1
2
3
4
5
6
7
8
9
10
11
Rishabh
↓
Personal AI Agent
↓
Amazon
↓
Find laptop
↓
Compare products
↓
Add to cart
From the website's perspective, however, the request is still automated.
That creates the fundamental conflict.

2. Not Every Bot Is the Same

The term "AI bot" is becoming too broad.
There are several fundamentally different types of automated traffic.

Training crawlers

These collect information that may later be used for model training.
1
2
3
4
5
6
7
Website
↓
Crawler
↓
Collect content
↓
Training dataset

Search crawlers

These collect information so an AI search engine can retrieve it later.
1
2
3
4
5
6
7
8
9
Website
↓
Crawler
↓
Index
↓
AI search
↓
User

Personal AI agents

These act on behalf of a specific user in real time.
1
2
3
4
5
6
7
Human
↓
Personal Agent
↓
Website
↓
Action
The third category is fundamentally different.
A personal agent isn't necessarily trying to copy an entire website or train a model.
It may simply be trying to complete a task for a paying customer.
Cloudflare now explicitly distinguishes Search, Agent and Training traffic in its AI bot controls, reflecting this growing difference.

3. Why Websites Are Blocking AI Agents

There isn't a single reason.
There are several.

Security

An autonomous agent can potentially do much more than a crawler.
For example:
1
2
3
4
5
6
7
8
9
10
11
Login
↓
Search
↓
Fill form
↓
Change settings
↓
Purchase
↓
Send message
If the agent is compromised or manipulated, the consequences can be much greater than a simple page request.

Fraud

A website may not know whether an agent is:
  • A legitimate customer
  • A scraper
  • A fraud operation
  • A credential-stuffing system
  • A purchasing bot
  • A malicious automated workflow

Resource consumption

Agents can generate enormous numbers of requests.
A poorly designed agent may repeatedly retrieve the same page, reload content, or navigate inefficiently.

Business models

This is perhaps the most complicated issue.
A website may earn money when a human:
1
2
3
4
5
6
7
Visits
↓
Views advertisement
↓
Clicks
↓
Purchases
An AI agent could instead:
1
2
3
4
5
6
7
Visit
↓
Read
↓
Summarize
↓
Return answer
The user may never visit the original website.
The website provided the information but potentially lost the advertising or engagement opportunity.
Cloudflare describes this as part of a broader economic problem: being discoverable by AI does not automatically mean that publishers are compensated for agent activity.

4. CAPTCHA Was Designed for a Different Internet

CAPTCHA is one of the clearest examples of this problem.
The traditional model is:
1
2
3
4
5
Website
↓
"Prove you are human"
↓
CAPTCHA
But an AI agent is explicitly not human.
So the question becomes:
Why should a legitimate AI agent have to pretend to be human?
This is where the industry is beginning to move toward a different model.
Instead of:
1
2
Human = allowed
Bot = blocked
the future could look like:
1
2
3
4
5
6
7
8
9
Unknown Agent
↓
Identity verification
↓
Authorization check
↓
Policy check
↓
Allowed / Denied
AWS has already implemented support for Web Bot Auth in Amazon Bedrock AgentCore Browser to help agents establish verifiable cryptographic identities when accessing websites. AWS describes the protocol as a way for websites to verify signed agent requests rather than relying entirely on traditional bot-detection mechanisms.

5. The Missing Layer: Agent Identity

This may be the most important concept in the entire discussion.
Traditional web security largely asks:
Who is the user?
The agentic web needs to ask:
Who is the agent, who does it represent, and what is it allowed to do?
Consider this example:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Human:
Rishabh

Agent:
ShoppingAgent-01

Purpose:
Shopping

Permissions:
READ_PRODUCTS
COMPARE_PRODUCTS
ADD_TO_CART

Restricted:
PURCHASE
CHANGE_ACCOUNT
ACCESS_PRIVATE_DATA
Now the website has something much more useful than an IP address.
It has an identity and a policy.
The architecture becomes:
1
2
3
4
5
6
7
8
9
10
11
Human
↓
Personal AI
↓
Agent Identity
↓
Authentication
↓
Authorization
↓
Website
This begins to resemble cloud IAM.
And that is where the problem becomes particularly interesting for cloud engineers.

6. Web Bot Auth: Giving Agents a Cryptographic Identity

One emerging approach is Web Bot Auth.
The basic idea is straightforward:
1
2
3
4
5
6
7
8
9
10
11
AI Agent
↓
HTTP Request
↓
Cryptographic Signature
↓
Website
↓
Verify Signature
↓
Identify Agent
Instead of trusting:
1
2
3
IP address
User-Agent
Cookies
the website can verify a cryptographic signature associated with the agent.
AWS's implementation of Web Bot Auth in AgentCore Browser uses signed requests, allowing participating security infrastructure to verify the identity of the automated client.
OpenAI's Cloud Browser also documents Web Bot Auth support. Its requests use HTTP Message Signatures, including Signature, Signature-Input and Signature-Agent headers, allowing website operators to verify that traffic actually originates from ChatGPT.
This is an important architectural change.
Instead of hiding the fact that an agent is a bot, the system can say:
Yes, I am an agent — and here is cryptographic proof of who I am.

7. Authentication Is Not Enough

Knowing who the agent is doesn't solve everything.
The website also needs to know:
What is this agent allowed to do?
This is authorization.
For example:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Agent: ShoppingAgent-01

Allowed:
✓ Search products
✓ Read product information
✓ Compare prices
✓ Add products to cart

Requires confirmation:
⚠ Purchase above ₹50,000

Not allowed:
✗ Change password
✗ Access email
✗ Transfer money
This looks very similar to concepts already familiar in cloud security:
  • IAM
  • OAuth
  • RBAC
  • Scopes
  • Policies
  • Tokens
  • Audit logs
The difference is that the subject receiving the permission is no longer necessarily a human.
It is an autonomous software agent.

8. The Agentic Internet Needs More Than Browsing

Simply allowing agents to browse websites isn't enough.
The Internet needs to become easier for agents to interact with.
Cloudflare describes its vision for an Agentic Internet using four properties:
Readable
Agents can efficiently consume information.
Discoverable
Agents can find resources and services.
Callable
Agents can invoke actions through machine-friendly interfaces.
Payable
Agents can pay for resources or services.
This can be represented as:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
AGENTIC INTERNET

┌───────┐
│ Human │
└───┬───┘
│
▼
┌───────────┐
│ AI Agent │
└─────┬─────┘
│
┌───────────┼───────────┐
▼ ▼ ▼
Discover Read Call
│ │ │
└───────────┼───────────┘
│
▼
Pay
│
▼
Web Service
This is much more than making browsers smarter.
It is about changing how software interacts with the web.

9. From Clicking Buttons to Calling Services

Today's agent often has to operate a website like a human:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Open browser
↓
Find button
↓
Click
↓
Wait
↓
Read page
↓
Find form
↓
Type
↓
Submit
This is fragile.
A website changes its HTML or UI and the agent can break.
A more agent-native architecture could look like:
1
2
3
4
5
6
7
Agent
↓
Structured API / Protocol
↓
Business Service
↓
Structured Response
For example:
1
2
3
4
5
6
7
{
"action": "search_flights",
"from": "Pune",
"to": "Bangalore",
"date": "2026-11-12",
"passengers": 1
}
The service could return:
1
2
3
4
5
6
7
8
9
{
"flights": [
{
"flight": "XX123",
"departure": "09:20",
"price": 4820
}
]
}
No visual browser is required.
No DOM scraping is required.
No CAPTCHA is required.
The interaction becomes a structured software-to-software transaction.

10. But Browsers Will Not Disappear

This does not mean traditional websites will suddenly disappear.
There will probably be two interfaces.

Human interface

1
2
3
4
5
6
7
HTML
CSS
JavaScript
Images
Buttons
Forms
Navigation

Agent interface

1
2
3
4
5
6
7
APIs
Structured data
Agent protocols
Authentication
Authorization
Policies
Payment interfaces
Conceptually:
1
2
3
4
5
6
7
8
9
10
WEBSITE
│
┌─────────┴─────────┐
│ │
▼ ▼
Human Interface Agent Interface
│ │
Browser API
│ │
Human AI Agent
The interesting question is whether companies will expose both.

11. The Economics of Agent Traffic

Security is only half of the problem.
The other half is economics.
Imagine a news website.
A human reads:
1
2
3
4
5
Article
↓
Advertisement
↓
Subscription opportunity
An agent might instead:
1
2
3
4
5
6
7
Article
↓
Extract information
↓
Summarize
↓
Return to user
The agent may consume the site's information without producing the same economic value as a human visitor.
That creates a difficult question:
If an AI agent consumes a website's information on behalf of a customer, who should pay?
Cloudflare is exploring infrastructure around this problem, including mechanisms for making resources such as web pages, datasets, APIs and MCP tools monetizable.
This suggests that agent payments could become a major part of the future web.

12. The Web Could Eventually Have Agent-Specific Pricing

Consider an API.
Today:
1
GET /products
might be free.
But an agent could make thousands of requests.
The website could eventually say:
1
2
3
4
5
6
7
8
9
10
11
Human browsing:
Free

Verified Agent:
100 requests/day

High-volume Agent:
₹X / 1,000 requests

Premium Data:
₹X / API call
This would transform the economics of web data.
Instead of:
1
Scrape → Free
the model could become:
1
2
3
4
5
6
7
8
9
Discover
↓
Authenticate
↓
Authorize
↓
Use
↓
Pay
This is one reason agent identity and agent payments are closely connected.

13. Cloud Infrastructure Becomes the Foundation

For cloud engineers, this entire architecture starts looking familiar.
An Agentic Internet could require:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
AI AGENT
│
▼
Agent Identity
│
▼
Authentication
│
▼
Authorization
│
▼
Agent Gateway
│
┌────────────┼────────────┐
▼ ▼ ▼
Rate Limit WAF Policy
│ │ │
└────────────┼────────────┘
▼
API / Service
│
Kubernetes
│
Cloud
Then add:
  • Observability
  • Distributed tracing
  • Audit logs
  • Secrets management
  • Zero Trust
  • Fraud detection
  • Cost controls
  • Agent evaluation
  • Runtime security
This is why the Agentic Internet is not just an AI research problem.
It is also a distributed-systems, security and cloud-infrastructure problem.

14. The New Security Model

Traditional web security:
1
Can this user access the resource?
Agentic security:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Who owns this agent?

What is this agent?

What task is it performing?

What permissions does it have?

What resource is it accessing?

What action is it attempting?

Does the action require human approval?

How much can it spend?

Can the action be audited?
A production-grade agent could therefore have a policy such as:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
agent:
name: shopping-agent
owner: user-123

permissions:
- read_products
- compare_products
- add_to_cart

limits:
purchase_amount: 50000
requests_per_minute: 60

approval:
purchases_above: 10000
This is essentially IAM for autonomous software.

15. The Biggest Shift: Bots Are Becoming a New Type of Internet Actor

Cloudflare makes an important distinction in its Agentic Internet work: the old idea that "bots are bad and humans are good" is becoming inadequate.
Some automated activity is undesirable:
1
2
3
4
Scraping millions of pages
Ignoring robots.txt
Circumventing access controls
Abusing resources
But other automated activity may be desirable:
1
2
3
4
5
Verified customer agent
Reading permitted content
Calling approved APIs
Making authorized purchases
Paying for resources
The question is therefore no longer simply:
Bot or human?
It becomes:
What kind of agent is this, who authorized it, and what is it trying to do?
That is a much more sophisticated security model.

16. Where This Is Going

The likely progression looks something like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
Traditional Web
│
▼
AI Search
│
▼
AI Copilots
│
▼
Browser Agents
│
▼
Authenticated Agents
│
▼
Authorized Agents
│
▼
Agent APIs / Protocols
│
▼
Agent-to-Agent Communication
│
▼
Agentic Commerce
│
▼
Agentic Internet
The important transition is between browser agents and authenticated/authorized agents.
Today's agents often have to behave like humans because websites were designed for humans.
The future could allow agents to identify themselves openly and interact through interfaces designed specifically for machine-to-machine communication.

17. What Happens to CAPTCHA?

CAPTCHA may not completely disappear.
Instead, its role could change.
Today:
1
2
3
Are you human?
↓
CAPTCHA
Future:
1
2
3
4
5
6
7
8
9
10
11
Who are you?
↓
Cryptographic identity

What can you do?
↓
Authorization

Are you behaving correctly?
↓
Policy / security controls
That is a much more useful model for autonomous software.
AWS's AgentCore Browser Web Bot Auth integration is an early example of this transition: rather than making an agent pretend to be human, it allows participating infrastructure to recognize a verified agent.

18. Why This Matters for Developers

This emerging problem creates several new engineering areas.

Agent Identity

How does an agent prove who it is?

Agent Authorization

What is it allowed to do?

Agent Security

How do we stop malicious agents or compromised agents?

Agent Observability

How do we know what an autonomous agent did?

Agent Gateways

How do websites manage thousands or millions of agent requests?

Agent Payments

How does an agent pay for resources?

Agent Protocols

How do agents communicate with services without relying on fragile browser automation?

Agent Governance

How do organizations enforce policies around autonomous actions?
These are not merely prompt-engineering problems.
They are infrastructure problems.

19. The Opportunity for Cloud and DevOps Engineers

This is where the trend becomes especially interesting.
A developer who only knows:
1
2
3
4
5
Python
+
LLM API
+
Prompt
can build an AI application.
But an engineer who understands:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
AI Agents
+
Linux
+
Networking
+
IAM
+
OAuth
+
HTTP
+
Docker
+
Kubernetes
+
Cloud
+
Security
+
Observability
can potentially build the infrastructure that allows thousands of agents to operate safely.
The future stack could look like:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
AI APPLICATION
│
AGENTS
│
┌─────────┴─────────┐
│ │
Agent Identity Agent Memory
│ │
Authorization Context
│ │
└─────────┬─────────┘
│
Agent Gateway
│
┌───────────┼───────────┐
│ │ │
WAF IAM Observability
│ │ │
└───────────┼───────────┘
│
APIs
│
Kubernetes
│
Cloud
That intersection between AI and cloud infrastructure could become one of the most interesting engineering domains of the agent era.

Conclusion

The story is not simply that:
"Websites don't like AI agents."
The deeper story is that the Internet is entering a transition.
For decades, the primary actor on the web was the human.
Now software agents are becoming another class of user.
The Internet therefore needs new mechanisms for:
  • Identity
  • Authentication
  • Authorization
  • Discovery
  • Communication
  • Security
  • Observability
  • Payments
The future may not be a web where AI agents disguise themselves as humans.
It may be a web where agents openly identify themselves, receive scoped permissions, interact through machine-readable interfaces, and pay for the resources they consume.
In other words:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
OLD WEB

Human
↓
Browser
↓
Website

AGENTIC WEB

Human
↓
AI Agent
↓
Identity
↓
Authorization
↓
Protocol
↓
Service
↓
Transaction
The challenge is no longer simply building smarter agents.
It is building an Internet where those agents can be trusted.
And that may become one of the defining infrastructure problems of the next generation of AI.

Further Reading

# agentic-ai# ai-security# cybersecurity# agents
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article