AWS Builder Center

Builder Center

Is Your AWS Setup Blocking AI Search Crawlers?

Is Your AWS Setup Blocking AI Search Crawlers?

A practical checklist for sites on Amazon S3, CloudFront and AWS WAF: confirm robots.txt is really served, make sure Bot Control isn't blocking verified AI crawlers, and check that your pages render without JavaScript.

Los Angeles SEO Inc

SEO, Technical SEO & AI Search Optimization
More people now find websites through AI assistants such as ChatGPT search and Perplexity, not only through classic search results. Those assistants can only cite pages their crawlers can fetch and read. If you host on AWS, three layers decide whether that happens: your robots.txt, your AWS WAF rules, and how your pages render.
This checklist walks through each layer for a typical setup of Amazon S3 or an application origin behind Amazon CloudFront, protected by AWS WAF.

1. Know which crawlers you're dealing with

AI companies separate crawlers for search from crawlers for model training, so you can allow one and block the other:
User agent / token
Purpose
 OAI-SearchBot 
Indexes pages for ChatGPT search results
 GPTBot 
Collects content that may be used to train OpenAI models
 ChatGPT-User 
Fetches a page when a ChatGPT user asks about it
 PerplexityBot 
Indexes pages for Perplexity answers
 Google-Extended 
robots.txt token that controls use of your content for Gemini training; it doesn't change Google Search
 Applebot-Extended 
robots.txt token that controls use of your content for Apple's AI training
Names change over time, so check each provider's own documentation:  OpenAI crawlers ,  PerplexityBot ,  Google's common crawlers  and  About Applebot .

2. Make sure robots.txt is really served

A robots.txt file only helps if crawlers receive it as plain text. Start with a quick check:
curl -sI https://www.example.com/robots.txt
You want  HTTP/2 200  and  content-type: text/plain . Two problems are common on S3 and CloudFront:
  • Single-page app error rules. Many SPA distributions use a  CloudFront custom error response  that turns every 403 or 404 into  /index.html  with a 200 status. If robots.txt is missing, crawlers get your HTML app shell instead, with a 200 status. Make sure the file exists in the bucket.
  • Wrong content type. If you upload without a content type, S3 may store a generic one. Set it explicitly.
Here's an example that allows AI search crawlers, opts out of training crawlers and keeps normal search open:
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml
Adjust the allow and disallow lines to your own policy. Then upload the file and clear the cached copy:
aws s3 cp robots.txt s3://YOUR-BUCKET/robots.txt --content-type text/plain
aws cloudfront create-invalidation --distribution-id YOUR-DISTRIBUTION-ID --paths "/robots.txt"
Keep in mind that robots.txt is a request, not an enforcement tool. Some bots ignore it, which is why AWS WAF matters too.

3. Check what AWS WAF Bot Control is doing with AI bots

The Bot Control managed rule group labels each request with the bot's name, category, organization and verification status. A bot counts as verified only when AWS can confirm it, for example through a valid Web Bot Auth signature or a known user agent that also matches the operator's IP ranges or reverse DNS. The user agent alone never verifies a bot, because clients can set it to anything.
The important detail for AI search is the rule group version:
  • Before version 4.0.0, Bot Control blocked the  CategoryAI  rule's bots by default.
  • From version 4.0.0, Bot Control Common no longer blocks verified bots.
If your web ACL is pinned to an older static version, verified AI search crawlers may be getting blocked without anyone noticing. To check:
  1. Open your web ACL in the AWS WAF console and look at the version of  AWSManagedRulesBotControlRuleSet .
  1. Review sampled requests or your WAF logs for requests where  CategoryAI  was the terminating rule.
  1. Use the  AI activity dashboard in AWS WAF , if it's available in your Region, to see which AI bots and agents reach your site.
To fix it, move to a current version of the rule group, or override the  CategoryAI  rule action to Count. If other rules in the web ACL still block verified bots, add a custom rule after the Bot Control rule group that allows requests carrying the verified label:
{
  "Name": "allow-verified-bots",
  "Statement": {
    "LabelMatchStatement": {
      "Scope": "LABEL",
      "Key": "awswaf:managed:aws:bot-control:bot:verified"
    }
  },
  "Action": { "Allow": {} },
  "VisibilityConfig": {
    "SampledRequestsEnabled": true,
    "CloudWatchMetricsEnabled": true,
    "MetricName": "allow-verified-bots"
  }
}
The rule needs a higher priority number than the Bot Control rule group, so it's evaluated later and the labels exist when it runs. Avoid allowlisting AI crawlers by user-agent string alone; anyone can send  User-agent: GPTBot . Rely on the verified label instead.
Also review the rules that sit around Bot Control. Geo-match blocks, rate-based rules and older custom user-agent blocks can stop crawlers just as effectively. As AWS recommends, test changes in Count mode first and switch to Block only after the logs look right.

4. Make sure your content exists without JavaScript

Many crawlers read the raw HTML a server returns and may not run JavaScript the way a browser does. If your site is a client-rendered single-page app, a crawler may see only an empty  <div id="root"></div> .
Compare what a crawler gets with what you see in the browser:
curl -s https://www.example.com/services/ | grep -i "<h1"
If your main heading and body text don't appear in the raw HTML, consider one of these options:
  • Pre-render (static generation) your pages at build time and deploy the HTML to S3 behind CloudFront.
  • Server-side rendering with a framework such as Next.js on AWS Amplify Hosting, which  supports SSR apps .
Structured data (JSON-LD) should be in the initial HTML too, not injected later by a script. If you'd rather not use the command line, a free page checker such as this  one-page SEO audit tool  reports the title, headings, noindex rules and other on-page signals for a URL.

Quick checklist

  •  robots.txt  returns 200 with  text/plain , not your SPA's  index.html 
  • Your robots.txt policy matches your intent for search crawlers versus training crawlers
  • The Bot Control rule group version is 4.0.0 or later, or  CategoryAI  is overridden to Count
  • Logs show no verified AI search crawlers blocked by  CategoryAI , geo, rate or user-agent rules
  • Main content and JSON-LD appear in the raw HTML
  • Your sitemap is listed in robots.txt and reachable
These are configuration issues, so they're usually quick to fix. Once crawlers can reach and read your pages, what gets cited comes down to clear, accurate content.

Sasan Raeisi is the founder of Los Angeles SEO Inc, a digital marketing agency in Los Angeles that works on technical SEO and AI search visibility.

References

# amazon-cloudfront# aws-waf# amazon-s3# aws-amplify# generative-ai
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article