AWS Builder Center

Builder Center

Application Security - Bot management

Application Security - Bot management

Bot management refers to solutions that handle the portion of web traffic coming from undesirable automated bots, operated by actors with financial incentives, causing a negative impact ranging from unnecessarily increasing infrastructure cost, to hurting business.

Achraf Souk

Last updated on 5 Jan 2026

Overview

A large portion of traffic hitting web applications is coming from undesirable automated bots, operated by actors with financial incentives, such as scraping valuable content, committing fraud, or improving their their position in certain markets. The negative impact of such automated bot traffic can range from unnecessarily increasing your infrastructure cost, to actually hurting your business. Bot management designates the techniques used to identify traffic coming from automated bots, and then appropriately filtering traffic from undesired bots, while allowing traffic from useful bots. The sophistication of such controls depends on how much bot operators are incentivized to target your web application. The higher is the financial gain potential, the more they invest in detection evasion techniques, requiring more advanced mitigation capabilities.
The core capability from AWS WAF that allows you to manage automated bot traffic is AWS WAF Bot Control managed rule group . Usually, it's first used in Count mode, i.e. without filtering traffic, to audit and gain visibility into the extent of non-human traffic and its categories. Then you can enable and tune layered protections to filter bot traffic according to your business needs.

Allowing traffic coming from useful bots

First, you want to allow traffic coming from useful bots for your business, such as search engines crawlers, social media preview bots, certain AI agents, your own health checks, etc.. The goal is to bypass protections against undesired bots. AWS WAF allows you to do it using the AWS WAF Bot Control managed rule group  configured with common inspection level to verify if requests are coming from useful bots.
Verification of self-identifying useful bots is mainly carried by checking the source IP from which the request is originated, and comparing it to the expected IP range of the bot identity claimed in the user-agent header. When the useful bot identity is verified, AWS WAF enriches the request with awswaf:managed:aws:bot-control:bot:verifiedlabel, which is metadata added to a request by a rule when the rule matches the request. Once added, a label is available for subsequent rules, which can use it to enrich their evaluation logic. If identity verification is not natively available in Bot Control for a specific useful bot your business require, you can create custom rules that combines source IP and user-agent header conditions to verify its identity.
In the case of agentic AI bots, Bot Control uses Web Bot Auth (WBA) , starting version 4.0, to authenticate legitimate bots in a stronger and more standardized way. It adds the awswaf:managed:aws:bot-control:bot:web_bot_auth:verifiedlabel to matching requests.

Filtering traffic coming from undesired generic bots

The internet is home for all sorts of low sophistication generic bots, that are scanning and crawling the web, such as for monitoring website uptime, scanning for vulnerabilities, AI training, research, etc.. Some of these bots are well behaved and respect robot.txt, but not all of them. In general, such bots can be identified and managed on the server side by analyzing the signature of requests based on HTTP attributes such as IP, User-Agent header or TLS fingerprint. AWS WAF allows you to filter such undesired bots using the following signature-based rules:
  • AWSManagedReconnaissanceList rule of the Amazon IP reputation list rule group , which detects requests coming IP addresses that are performing reconnaissance against AWS resources.
  • AnonymousIPList rule of the Anonymous IP list managed rule group , which detects requests coming from IP addresses of sources known to anonymize client information, like TOR nodes, temporary proxies, and other masking services.
  • HostingProviderIPList rule of the Anonymous IP list managed rule group , which detects requests coming from IP addresses from web hosting and cloud providers, which are less likely to source end-user traffic.
  • AWS WAF Bot Control managed rule group  configured with common inspection level to block self identifying bots, or bots with high confidence signatures. You can granularly configure  this rule group, for example to block certain bot categories, and rate limit others on certain paths of your application.
  • Honeypots  that can automatically trigger AWS WAF to block offending IPs.
  • Custom rules, such as based on rate limiting, ASNs, and JA3/JA4 fingerprints. Note that rules based on ASNs or JA3/JA4 fingerprints must be carefully crafted to avoid false positives.
  • JavaScript Challenge actions can be used to detect scripts that are not running on clients with browser capabilities.
  • Managed rules provided by security vendors in the AWS Marketplace, such as Bot Protection Rules by F5 , Bot Protection by Cloudbric  and Active Malicious Bots by ThreatSTOP .

Managing traffic coming from opportunistic bad bots

Some threat actors abuse the available resources of your web application opportunistically to commit fraudulent activities, that do not specifically target your own business, such as SMS pumping/OTP abuse, credit-card testing, contact form spam, etc.. They have higher incentives to invest in evasion techniques such as hiding behind distributed IPs from residential networks, advanced browser automation frameworks and automated CAPTCHA solving APIs.
To properly handle such bad bots, you need to invest more sophisticated and expensive protection techniques, ranging from static challenges to more advanced behavioral detections:
  • JavaScript / CAPTCHA Challenges. Configured rules in AWS WAF can have JavaScript and CAPTCHA challenge  actions, in addition to block, count or rate limit. When a rule is configured with the actions, clients are required to solve a problem to acquire a valid token, such as running a JavaScript script, or solving a CAPTCHA puzzle. When a client successfully acquires a token, they present it in subsequent requests, preventing WAF from requiring another challenge within a configurable immunity time. Using client side AWS WAF application integration SDKs , you can provide a better user experience, for example by making the JavaScript challenge invisible, and by integrating the CAPTCHA puzzle within your application UI. Read the following blog posts to understand in depth your options when using integrating JavaScript and CAPTCHA challenges in your web application:
  • AWS WAF Bot Control managed rule group  configured with targeted inspection level, allowing you to block or present the previously explained challenges dynamically when bots are detected. Detection is based on: Client-side fingerprinting which detects signals from automated browsers; behavioral analysis such as volumetric sessions compared to your normal traffic baseline, or sessions reusing tokens across IPs, geographies; ML based classification of surprising clusters of coordinated activities. Bot Control signals are presented independently and with different levels of confidence when applicable, to allow granular control.
Such protections make it more expensive for threat actors to achieve their goals, which opportunistically move their operations to exploit the resources of the next easiest web application.

Managing traffic from high-incentive targeted bad bots

Some threat actors invest in advanced bot infrastructure to target your own business, when financial incentives are high. Examples of such bots include: scraping content, such as jobs or real-estate listings, to monetize it elsewhere, credential stuffing to abuse loyalty programs, gathering competitive pricing information, or exploit cart APIs to by limited products faster than users and sell them in the black market for higher prices.
For managing traffic from such bad bots, you can employ the following protections, on top of AWS Bot Control:
  • Fraud control managed rules in AWS WAF, which are more specialized in detecting specific types of fraud the login and registration workflows.. Control Account Takeover Prevention  rule group is specialized in protecting against credential stuffing fraud, by monitoring the rate of login attempts, including failed attempts, and checking username and password combinations against a regularly-updated stolen credential databases. Account Creation Fraud Prevention  rule group is specialized in protecting against mass creation of fake accounts to conduct fraudulent activities such as abusing promotional sign up bonuses, posting fake comments, or credit card testing.
  • Application level detections, based on signals to identify abnormal behavior, based on what is expected by your application. For example, OLX  automatically query their website traffic logs gathered on the client side using New Relic, to detect suspicious behaviors by analyzing aspects like user navigation patterns. Using such signals, you can automate your response using AWS WAF, for example by blocking or challenging using CAPTCHA, requests coming from IPs with suspicious application level behavior. To get started with the concept of WAF automation based on application signals consider the examples in this AWS Solution .
  • Application level response, where you adapt the flow of your application based on threat information received from AWS WAF. AWS WAF can be configured to send such information upstream to your application using headers. For example, you can force users to authenticate when some level of bot activity is detected by AWS WAF. Read more about such application level adaptive responses in the following blog post:
  • Specialized bot management vendors from the AWS Marketplace, such as DataDome , Distill Networks , Human , and Kasada . Note that each security vendor has different strengths in terms of industry specific protections, capabilities and cost. In general, your application using CloudFront can integrate vendor solutions in one of two ways: SaaS reverse proxy-based solution, sitting between CloudFront and your origin or globally replicated Bot Mitigation API that can be called by Lambda@Edge for every incoming request (i.e. configured on viewer request event) to decide how to manage the request.

Resources

# security# aws-edge-services# waf
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article