When the WAF Is the Outage: Diagnosing and Tuning AWS Managed Rules Without Giving Up Protection
A production false positive from AWS WAF's Core rule set, how to find the exact rule in minutes with Logs Insights, and a Count → Label → Custom Rule pattern (CDK code included) that fixes it without weakening protection.
TL;DR
Here is a failure mode that catches teams on AWS WAF: real users start getting
403 Forbidden from an endpoint, the containers are healthy, and the application logs are empty, because WAF is rejecting the requests before they ever reach the service. The cause is often a single rule insideAWSManagedRulesCommonRuleSet, such as SizeRestrictions_BODY blocking legitimate large request bodies. Removing the managed group ends the outage and quietly deletes protection everywhere else. The better fix is to override only that one rule to Count, match on the label it leaves behind, and write a narrow custom rule that blocks oversized bodies everywhere except the one endpoint that legitimately needs them. This post walks through the diagnosis, the exact code, the security gap the fix opens (and how to close it), and the tests and alarms that keep it from silently happening again.1. The scenario: a 403 nobody can explain
Picture a typical setup: a FastAPI service on ECS Fargate behind an Application Load Balancer, with a web ACL attached and the AWS-managed Core rule set enabled in Block mode, because that's what most reference architectures show. It runs happily for weeks. Then a feature that sends larger payloads (a document upload, a bulk import) starts failing for real users.
Here is the shape of the debugging session, because it looks the same every time. The reports say "uploads fail." You check the ECS service: tasks running, no restarts, CPU and memory flat. You check the target group: all targets healthy. You check the application logs for the failing endpoint and find nothing at all, not an error, not a warning, not even an access-log line for the failing requests. Time passes before someone stops looking at the application and looks at what sits in front of it.
That is the first lesson, and it is the reason this post exists. A WAF block does not look like an application failure. It looks like silence. Every instinct you have for debugging a production service (check the deploy, check the tasks, check the logs) points at components that are working correctly.
2. Why a WAF block is invisible to your application
AWS WAF is attached to a resource, for example, an Application Load Balancer, and inspects each request before it is forwarded to a target. If a rule blocks the request, the load balancer answers the client itself with a 403 (by default), and the request never reaches your target group.
Figure 1. The web ACL evaluates every request before the ALB forwards it. A block is answered at the edge of the load balancer, so the Fargate tasks, the database, and the application logs never see the request. The only place the decision is recorded is WAF's own logging and metrics.
Three practical consequences follow from this picture:
- Your application logs cannot tell you about WAF blocks. The only record is WAF logging (and WAF's CloudWatch metrics). If you haven't turned logging on, the evidence does not exist.
- Health checks keep passing. Load balancer health checks go to your targets directly; they don't depend on the client-facing request path your users take, so a WAF false positive doesn't flip your targets unhealthy. Your dashboards stay green while users are failing.
- The failure is selective. A rule that triggers on payload size, content patterns or a header only breaks the requests that match. That's why it can look like a "flaky" bug that only some users hit.
3. How AWS WAF actually evaluates a request
To tune WAF safely, you need a precise mental model of how rules are evaluated. These are the parts that matter for this incident.
Rules run in priority order. A web ACL's rules are evaluated from the lowest priority number to the highest. Each rule either matches or doesn't.
Actions are terminating or non-terminating.
Allow and Block (and the CAPTCHA/Challenge actions) are terminating: evaluation stops, and the action is applied. Count is non-terminating: WAF records that the rule matched, then keeps evaluating the remaining rules. If no rule terminates the evaluation, the web ACL's default action applies.Managed rule groups are bundles of rules.
AWSManagedRulesCommonRuleSet (the "Core rule set") is a group of individual rules, each with its own name, such as SizeRestrictions_BODY or CrossSiteScripting_BODY. It is built as a broad baseline for web applications. It is not tuned to your application.There are two different override mechanisms, and confusing them is dangerous.
| Setting | Scope | What it does |
|---|---|---|
| Group-level OverrideAction (None or Count) | The whole rule group | None means the rules in the group use their own actions. Count means every rule in the group is treated as Count, so nothing in the group blocks. |
| RuleActionOverrides | Individual rules inside the group | Changes the action of just the named rules (for example, one rule to Count) and leaves the rest of the group untouched. |
Setting the whole group to Count to "quickly fix" an outage switches off every protection in that group. Overriding a single rule is the surgical option, and it is the one you want. (Older templates use
ExcludedRules, which is the deprecated predecessor of RuleActionOverrides.)Labels connect rules. When a rule matches, even in Count mode, it can add a label to the request. Rules evaluated later in the same web ACL can match on that label with a
LabelMatchStatement. Managed rule groups add labels in the format awswaf:managed:aws:<group>:<rule>. Labels exist only for the duration of that request's evaluation, and they also show up in the WAF logs, which is how you can read the exact label string for your own account.This label mechanism is what turns a blunt on/off switch into something you can program: "let the managed rule detect, and let me decide what to do with the detection."
Core rule set rules that commonly produce false positives
Every application is different, so treat this as a map of where to look first, not a prediction.
| Rule | What it looks at | Legitimate traffic that can trip it |
|---|---|---|
| SizeRestrictions_BODY | Request body size | File uploads, bulk JSON imports, base64 payloads |
| CrossSiteScripting_BODY | Script-like content in the body | Rich-text or HTML stored in JSON fields |
| NoUserAgent_HEADER | Requests with no User-Agent | Internal scripts, some webhook senders |
4. Root cause: SizeRestrictions_BODY and the 8 KB reality
SizeRestrictions_BODY blocks requests whose body is larger than 8 KB (8,192 bytes). Suppose an endpoint such as a document upload or bulk import legitimately sends payloads from tens of kilobytes up to several megabytes. Nothing is wrong with that traffic. The rule was written for the average web request, and this endpoint is not average.There's a deeper detail worth understanding, because it determines what you can and can't do about it: WAF can only inspect a limited prefix of the request body, and that limit depends on the resource type it's attached to. From the AWS documentation:
- For an Application Load Balancer (and AWS AppSync), the body inspection limit is fixed at 8 KB.
- For CloudFront, API Gateway, Amazon Cognito, App Runner, and Verified Access, the default is 16 KB, and you can raise it in the web ACL's association configuration in 16 KB steps up to 64 KB, for additional processing fees.
When a body is larger than the limit, the host service forwards only the part within the limit to WAF. What WAF does about the rest is controlled by oversize handling, which has three options:
CONTINUE (inspect what's available normally), MATCH (treat the oversize component as matching the rule) and NO_MATCH. For rule groups you don't own, like the managed Core rule set, AWS has already chosen the oversize handling inside each rule, which is why you can't tune it from outside and have to work with the rule's output instead.Two things follow, and both matter for the fix:
- Raising the limit isn't an option on an ALB. It is fixed at 8 KB, so "just let WAF inspect bigger bodies" isn't available. (If your stack fronts the ALB with CloudFront, the limit is a CloudFront-side setting and the calculus is different.)
- WAF cannot enforce an arbitrary body-size limit above its inspection limit. It can tell that a body is "over the limit", but not whether it's 10 KB or 50 MB. If you need a hard cap on upload size, the application (or a reverse proxy in front of it) has to enforce it. I'll come back to this in section 9, because relaxing the size rule on one path means that path's bodies are only inspected up to the first 8 KB.
5. Step one: logging you can actually query
You can't debug what you didn't record. Here's a WAF logging setup worth putting in front of every web ACL, written in AWS CDK (Python).
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
from aws_cdk import (
ArnFormat,
RemovalPolicy,
Stack,
aws_logs as logs,
aws_wafv2 as wafv2,
)
def add_waf_logging(stack: Stack, web_acl: wafv2.CfnWebACL) -> logs.LogGroup:
# The log group name MUST start with "aws-waf-logs-" or WAF will
# refuse to use it as a logging destination.
log_group = logs.LogGroup(
stack,
"WafLogGroup",
log_group_name="aws-waf-logs-app-web-acl",
retention=logs.RetentionDays.ONE_MONTH,
removal_policy=RemovalPolicy.DESTROY,
)
# Gotcha: CDK's log_group.log_group_arn ends with ":*", and the WAF
# logging configuration rejects ARNs with that suffix. Build the ARN
# without it.
log_group_arn = stack.format_arn(
service="logs",
resource="log-group",
resource_name=log_group.log_group_name,
arn_format=ArnFormat.COLON_RESOURCE_NAME,
)
wafv2.CfnLoggingConfiguration(
stack,
"WafLoggingConfig",
resource_arn=web_acl.attr_arn,
log_destination_configs=[log_group_arn],
# Never write credentials into a log group.
redacted_fields=[
wafv2.CfnLoggingConfiguration.FieldToMatchProperty(
single_header={"name": "authorization"}
),
wafv2.CfnLoggingConfiguration.FieldToMatchProperty(
single_header={"name": "cookie"}
),
],
)
return log_group
A few notes on this setup:
- Redact sensitive headers. WAF logs include request headers. Without
redacted_fields, yourAuthorizationandCookieheaders land in CloudWatch Logs in clear text. - Mind the permissions. The identity that deploys the stack needs the permissions WAF requires to set up log delivery and update the log group's resource policy. If the first deploy fails on logging, check the "logging permissions" section of the WAF developer guide before anything else.
- Mind the cost. WAF logs every request by default, including the healthy ones. During tuning, that's what you want. In steady state, consider a shorter retention period or a logging filter, but be careful: a filter that drops what you later need defeats the purpose.
- Logs go to CloudWatch Logs, S3, or Kinesis Data Firehose. CloudWatch Logs is the simplest choice because Logs Insights makes the incident-time queries below possible without any extra setup. For long-term analysis, S3 plus Athena is cheaper.
The console's "sampled requests" view is useful as a first look, but it only holds a limited window of recent requests per rule. It's not a substitute for logs.
6. Step two: finding the rule in minutes, not hours
Each WAF log record is a JSON document. These are the fields that matter for tuning:
| Field | What it tells you |
|---|---|
| action | The final action applied: ALLOW, BLOCK, ... |
| terminatingRuleId | The rule (or rule group) that made the final decision |
| ruleGroupList | For managed groups: the individual rule inside the group, in terminatingRule, and any Count matches in nonTerminatingMatchingRules |
| labels | Labels attached to the request during evaluation |
| httpRequest.uri, httpMethod, headers | What the request was |
For managed rule groups,
terminatingRuleId is the group. The specific rule inside the group is in ruleGroupList. That's the detail that tells you which individual rule to tune.
Figure 2. A runbook for this class of incident. Steps 2 and 3 (logging and querying) are what turn a multi-hour guessing session into a few minutes.
Query 1: Which rules are blocking, and on which URIs?
Run this in CloudWatch Logs Insights against the WAF log group:
1
2
3
4
5
6
fields httpRequest.uri as uri
| filter action = "BLOCK"
| stats count(*) as blocks by terminatingRuleId, ruleGroupList.0.terminatingRule.ruleId, uri
| sort blocks desc
| limit 25
If the group is the only one in the web ACL with a terminating match,
ruleGroupList.0 is the right index. If your ACL has several groups, the terminating rule may sit at a different index, so check one raw record first.In a scenario like this one, the top row shows the upload URI blocked many times by the Core rule set's size rule, all from legitimate clients. That one row is the whole diagnosis.
Query 2: When did it start?
1
2
filter action = "BLOCK"
| stats count(*) as blocks by bin(5m)
Line this up against your deploy history. Spikes usually coincide with the day the managed group was attached or the day a client release started sending larger payloads.
Query 3: everything a given rule has matched, including Count matches
1
2
3
filter @message like /SizeRestrictions_/
| stats count(*) as hits by httpRequest.uri, httpRequest.httpMethod, action
| sort hits desc
This works for blocked requests and for requests where the rule is in Count mode, because the rule name appears in the record either way. After the fix in section 8, this is the query that shows you what the rule is still detecting.
Measure before you decide
"Allow bigger bodies on the upload endpoint" is a guess until you know how big the bodies actually are. Export the matching records from Logs Insights (or pull them with
boto3's start_query and get_query_results) and compute the p50, p95, and maximum of the Content-Length header per URI. Two caveats: Logs Insights returns at most 10,000 rows per query, so narrow the window on busy systems, and Content-Length is absent on chunked requests. The numbers answer the question that decides the fix: is it one endpoint with legitimately large bodies, or is the size rule catching something broader? Only then can you choose how wide the exception should be.7. Two tempting fixes to avoid
At 2 a.m., there are two very tempting ways out. Both end the outage. Both are wrong.
Tempting fix #1: remove the rule group (or set the whole group to Count). This ends the incident immediately and silently deletes protection on every other endpoint: the XSS checks, the file-inclusion checks, the SSRF-to-metadata checks, all of it. The blast radius of the fix is far larger than the blast radius of the bug.
Tempting fix #2: scope the whole managed group away from the affected path. A
ScopeDownStatement on a managed rule group tells WAF to evaluate the group only for requests that match the statement. If you write "everything except the upload path", then the upload path is no longer inspected by any rule in the group. You wanted to relax one size check and instead exempted the endpoint from XSS, injection, and inclusion checks too. That's a worse trade than it looks, because an upload endpoint that accepts rich input is exactly where you want those checks.What you want instead is to keep the managed rule's detection, change what you do about it, and make the exception as narrow as you can justify: one rule, one path.
8. The fix: Count, then Label, then a narrower custom rule
The pattern has three moving parts, shown in Figure 3.
Figure 3. Priority 10 detects and labels but doesn't block; priority 11 decides. Request A (large body, upload path) is allowed. Request B (large body, any other path) is blocked. Request C (small body) never gets labelled at all. The other Core rules keep enforcing on every path.
Step 1: override only the offending rule to Count. The group keeps its normal
OverrideAction: None, so every other rule still blocks. Only SizeRestrictions_BODY is changed, via RuleActionOverrides. In Count mode, the rule still matches and still adds its label; it just doesn't terminate evaluation.Step 2: add a custom rule at a higher priority number (so it's evaluated after the group) that matches on the label and on the URI path not being the upload path, with action Block.
Step 3: Prove it in Count before you enforce it. The custom rule's action is a parameter. Deploy it as Count first, let real traffic flow, review the logs, and only then flip it to Block.
Here is the complete web ACL, in CDK (Python). I've included a rate-based rule for the upload path (explained in section 9) and the ALB association.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
from aws_cdk import (
ArnFormat,
RemovalPolicy,
Stack,
aws_elasticloadbalancingv2 as elbv2,
aws_logs as logs,
aws_wafv2 as wafv2,
)
from constructs import Construct
# Example values: adjust these to your application
UPLOAD_PATH = "/api/v1/documents/upload"
WEB_ACL_NAME = "app-web-acl"
# Copy the exact label string from the "labels" field of a real log record
# in YOUR account. Format: awswaf:managed:aws:<group>:<rule>
OVERSIZE_LABEL = "awswaf:managed:aws:core-rule-set:SizeRestrictions_Body"
def vis(metric_name: str) -> wafv2.CfnWebACL.VisibilityConfigProperty:
return wafv2.CfnWebACL.VisibilityConfigProperty(
cloud_watch_metrics_enabled=True,
metric_name=metric_name,
sampled_requests_enabled=True,
)
def upload_path_match() -> wafv2.CfnWebACL.StatementProperty:
"""True when the request URI path starts with the upload path."""
return wafv2.CfnWebACL.StatementProperty(
byte_match_statement=wafv2.CfnWebACL.ByteMatchStatementProperty(
search_string=UPLOAD_PATH,
field_to_match=wafv2.CfnWebACL.FieldToMatchProperty(uri_path={}),
positional_constraint="STARTS_WITH",
text_transformations=[
wafv2.CfnWebACL.TextTransformationProperty(priority=0, type="NONE")
],
)
)
class WafStack(Stack):
def __init__(
self,
scope: Construct,
construct_id: str,
*,
alb: elbv2.IApplicationLoadBalancer,
enforce_oversize_block: bool = False, # start in Count, flip after review
**kwargs,
) -> None:
super().__init__(scope, construct_id, **kwargs)
# --- Priority 5: rate limit the upload path (see section 9) ---------
upload_rate_rule = wafv2.CfnWebACL.RuleProperty(
name="RateLimitUploads",
priority=5,
action=wafv2.CfnWebACL.RuleActionProperty(block={}),
statement=wafv2.CfnWebACL.StatementProperty(
rate_based_statement=wafv2.CfnWebACL.RateBasedStatementProperty(
limit=100, # tune from your real traffic
aggregate_key_type="IP",
evaluation_window_sec=300,
scope_down_statement=upload_path_match(),
)
),
visibility_config=vis("RateLimitUploads"),
)
# --- Priority 10: Core rule set, ONE rule overridden to Count --------
core_rule_set = wafv2.CfnWebACL.RuleProperty(
name="AWSManagedRulesCommonRuleSet",
priority=10,
# Group-level override stays None: every other rule still enforces.
override_action=wafv2.CfnWebACL.OverrideActionProperty(none={}),
statement=wafv2.CfnWebACL.StatementProperty(
managed_rule_group_statement=wafv2.CfnWebACL.ManagedRuleGroupStatementProperty(
vendor_name="AWS",
name="AWSManagedRulesCommonRuleSet",
rule_action_overrides=[
wafv2.CfnWebACL.RuleActionOverrideProperty(
name="SizeRestrictions_BODY",
action_to_use=wafv2.CfnWebACL.RuleActionProperty(count={}),
)
],
)
),
visibility_config=vis("CoreRuleSet"),
)
# --- Priority 11: block oversized bodies everywhere EXCEPT uploads ---
oversize_action = (
wafv2.CfnWebACL.RuleActionProperty(block={})
if enforce_oversize_block
else wafv2.CfnWebACL.RuleActionProperty(count={})
)
oversize_rule = wafv2.CfnWebACL.RuleProperty(
name="BlockOversizedBodyExceptUploads",
priority=11,
action=oversize_action,
statement=wafv2.CfnWebACL.StatementProperty(
and_statement=wafv2.CfnWebACL.AndStatementProperty(
statements=[
wafv2.CfnWebACL.StatementProperty(
label_match_statement=wafv2.CfnWebACL.LabelMatchStatementProperty(
scope="LABEL",
key=OVERSIZE_LABEL,
)
),
wafv2.CfnWebACL.StatementProperty(
not_statement=wafv2.CfnWebACL.NotStatementProperty(
statement=upload_path_match()
)
),
]
)
),
visibility_config=vis("OversizedBodyOutsideUploads"),
)
web_acl = wafv2.CfnWebACL(
self,
"AppWebAcl",
name=WEB_ACL_NAME,
scope="REGIONAL",
default_action=wafv2.CfnWebACL.DefaultActionProperty(allow={}),
visibility_config=vis(WEB_ACL_NAME),
rules=[upload_rate_rule, core_rule_set, oversize_rule],
)
wafv2.CfnWebACLAssociation(
self,
"AlbAssociation",
resource_arn=alb.load_balancer_arn,
web_acl_arn=web_acl.attr_arn,
)
# --- Logging (section 5) --------------------------------------------
log_group = logs.LogGroup(
self,
"WafLogGroup",
log_group_name=f"aws-waf-logs-{WEB_ACL_NAME}",
retention=logs.RetentionDays.ONE_MONTH,
removal_policy=RemovalPolicy.DESTROY,
)
log_group_arn = self.format_arn(
service="logs",
resource="log-group",
resource_name=log_group.log_group_name,
arn_format=ArnFormat.COLON_RESOURCE_NAME,
)
logging_config = wafv2.CfnLoggingConfiguration(
self,
"WafLoggingConfig",
resource_arn=web_acl.attr_arn,
log_destination_configs=[log_group_arn],
redacted_fields=[
wafv2.CfnLoggingConfiguration.FieldToMatchProperty(
single_header={"name": "authorization"}
),
wafv2.CfnLoggingConfiguration.FieldToMatchProperty(
single_header={"name": "cookie"}
),
],
)
logging_config.node.add_dependency(log_group)
Why each design choice matters
The group-level
OverrideAction stays None. This one line decides whether you made a surgical fix or switched off a security control. RuleActionOverrides changes one rule; OverrideAction: Count changes all of them.Priority numbers do real work. The managed group (10) must run before the custom rule (11), because the label doesn't exist until the group has evaluated the request. Swap them and the custom rule silently does nothing.
Match the path carefully.
STARTS_WITH On the URI path is cheap and easy to reason about, but trailing-slash and case variations can make an exemption miss the intended path or match more than intended. Check how your router normalizes paths and add a LOWERCASE text transformation if it's case-insensitive.Start in Count.
enforce_oversize_block=False means the first deploy only observes. Run Query 3 from section 6 against the new rule, confirm it would only have blocked oversized requests to other paths, then redeploy with enforce_oversize_block=True.The result: large uploads on the upload path work again, and oversized bodies on every other endpoint are still blocked, which is what you wanted from the Core rule set in the first place.
9. The gap this fix leaves open, and how to close it
I want to be honest about what this fix costs, because it's the part most tutorials skip.
Section 4 established that on an ALB, WAF inspects only the first 8 KB of a body. By exempting the upload path from the size rule, you've told WAF "bodies on this path may be large." The body-content rules (XSS, injection patterns, file inclusion) still run on that path, but they only ever see the first 8 KB. Anything beyond the first 8 KB on that path is not inspected by WAF at all. An attacker who knows this can pad a payload. And because WAF can't enforce a hard size cap above its inspection limit, "how large is too large?" is now a question the WAF can no longer answer for that path.
That's an acceptable trade only if you put compensating controls behind it. These are the ones I'd put in place, in order of value.
9.1 Enforce an explicit size cap in the application
The application (or a reverse proxy in front of it) has to own the real limit. Here's a pure-ASGI middleware for FastAPI that rejects oversized requests twice: early, from the declared
Content-Length, and again while streaming the body, which catches requests that lie about their length or use chunked encoding.1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
from fastapi import FastAPI, HTTPException
from fastapi.responses import JSONResponse
from starlette.types import ASGIApp, Message, Receive, Scope, Send
# Example limits: use your own
PATH_LIMITS = {
"/api/v1/documents/upload": 25 * 1024 * 1024, # 25 MB
}
DEFAULT_LIMIT = 1 * 1024 * 1024 # 1 MB everywhere else
class BodySizeLimitMiddleware:
def __init__(self, app: ASGIApp, path_limits: dict[str, int], default_limit: int):
self.app = app
self.path_limits = path_limits
self.default_limit = default_limit
def _limit_for(self, path: str) -> int:
for prefix, limit in self.path_limits.items():
if path.startswith(prefix):
return limit
return self.default_limit
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
if scope["type"] != "http":
await self.app(scope, receive, send)
return
limit = self._limit_for(scope["path"])
# 1) Early reject from the declared Content-Length.
headers = dict(scope["headers"])
declared = headers.get(b"content-length")
if declared is not None and declared.isdigit() and int(declared) > limit:
response = JSONResponse({"detail": "Payload too large"}, status_code=413)
await response(scope, receive, send)
return
# 2) Enforce while streaming (covers chunked bodies and false headers).
received = 0
async def limited_receive() -> Message:
nonlocal received
message = await receive()
if message["type"] == "http.request":
received += len(message.get("body", b""))
if received > limit:
# Raise HTTPException (not a custom exception): FastAPI's
# body parsing turns other exception types into a generic 400.
raise HTTPException(status_code=413, detail="Payload too large")
return message
await self.app(scope, limited_receive, send)
app = FastAPI()
app.add_middleware(
BodySizeLimitMiddleware,
path_limits=PATH_LIMITS,
default_limit=DEFAULT_LIMIT,
)
Defense in depth works best when the two layers disagree on purpose: WAF is strict everywhere by default, and the application is the one place that knows the real per-endpoint limit.
9.2 Rate-limit the exempted path
An endpoint that accepts large bodies is also an expensive endpoint to hit repeatedly. The rate-based rule in the web ACL above (priority 5) caps requests per source IP on the upload path. Two caveats are worth knowing: rate-based rules aggregate by IP by default, which misbehaves when many legitimate users share one address (corporate NAT) or when a proxy or CDN sits in front, so use the forwarded-IP aggregation option or custom keys where that applies; and there's a minimum limit value, so very tight limits aren't expressible.
9.3 Require authentication before the expensive work
WAF can't validate a JWT, but your application can. Make sure the upload endpoint authenticates before it reads and processes the body, so unauthenticated bulk data is rejected as cheaply as possible.
9.4 Consider taking large bodies off the WAF-protected path entirely
If the "large bodies" are file uploads, the cleanest long-term fix is often not to send them through the load balancer at all. Have the API issue a short-lived pre-signed S3 upload, and let the client send the file straight to S3. A pre-signed POST policy can enforce a
content-length-range, so the size cap is enforced by S3 itself rather than by your containers, and the bytes never touch the WAF inspection limit. The WAF then only ever sees small JSON requests, and the size rule can stay on everywhere. If your uploads can move to that model, it removes the exception entirely.10. Prove it works: tests that run on every deploy
A WAF rule is production code and deserves a regression test. These tests send real HTTP requests to the staging environment and assert on what the WAF does with them.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
# tests/waf/test_oversize_rule.py
import json
import os
import pytest
import requests
BASE_URL = os.environ["SMOKE_BASE_URL"] # e.g. https://staging.example.com
UPLOAD_PATH = "/api/v1/documents/upload" # example path: use your own
OTHER_PATH = "/api/v1/auth/login" # example path: use your own
HEADERS = {"Content-Type": "application/json"}
def body_of_kb(kb: int) -> bytes:
return json.dumps({"blob": "a" * (kb * 1024)}).encode()
def post(path: str, kb: int) -> requests.Response:
return requests.post(
BASE_URL + path, data=body_of_kb(kb), headers=HEADERS, timeout=15
)
def test_large_body_is_not_blocked_on_upload_path():
# Assumption: unauthenticated requests to this path get a 401/422 from the
# app. A 403 would therefore come from the WAF, not the application.
response = post(UPLOAD_PATH, kb=12)
assert response.status_code != 403
def test_large_body_is_blocked_on_other_paths():
response = post(OTHER_PATH, kb=12)
assert response.status_code == 403
def test_small_body_is_never_blocked_by_size_rule(path):
response = post(path, kb=2)
assert response.status_code != 403
And the CI job that runs them after the staging deploy (GitHub Actions):
1
2
3
4
5
6
7
8
9
10
11
12
13
waf-smoke-tests:
needs: deploy-staging
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: pip install pytest requests
- name: WAF regression tests
run: pytest tests/waf -q
env:
SMOKE_BASE_URL: ${{ vars.STAGING_URL }}
Notice what this buys: the next person who edits the web ACL, or the next AWS managed rule group update, trips a red build instead of a customer complaint. Make sure the 403s you assert on are WAF-generated by choosing test endpoints where your application never returns 403 itself.
11. Monitoring and operating it long term
Alarm on blocks. Create two CloudWatch alarms on the
BlockedRequests metric in the AWS/WAFV2 namespace: one on the whole web ACL (Rule = ALL), which catches any new false positive or attack, and one on the custom rule's own metric, which catches the specific regression in this post (for example, a client release that starts sending larger payloads). Set thresholds from a normal week of traffic and treat missing data as not breaching. Confirm the exact metric dimensions for your web ACL in the CloudWatch console before relying on the alarms. Keep Query 1 and Query 3 from section 6 as saved Logs Insights queries, so nobody has to remember the syntax at 2 a.m.Operate managed groups carefully.
- Roll out new groups in Count mode first, and run them against a full traffic cycle (weekday peaks, batch jobs, release days) before enforcing rule by rule.
- Pin and review versions. Managed rule groups are versioned and updated by AWS. Pin a version and review updates in a lower environment, so "AWS changed a rule" becomes a code review instead of a surprise.
- Keep WAF configuration in code, reviewed and tested like the application.
- Document every exception with its owner and the compensating controls behind it, and delete it when the reason goes away (for example, if uploads move to pre-signed S3).
12. The checklist
If you could go back to the day the managed group was attached, change exactly one thing: enable logging first, and run every managed rule group in Count mode before turning on blocking.
- Enable WAF logging to a log group named
aws-waf-logs-*before attaching rules, withAuthorizationandCookieredacted. - Run new managed groups in Count and review a full traffic cycle in Logs Insights.
- Measure real request sizes per endpoint before deciding what to relax.
- Enforce rule by rule, never by disabling a whole group.
- Make every exception narrow (one rule, one path), documented, and backed by an application-level control.
- Add a post-deploy WAF regression test to CI, and alarm on spikes in blocked requests.
A managed rule group is a well-tested baseline, not a guarantee of safety for your particular application, and in Block mode a false positive is a silent outage. Log first, measure real traffic, relax exactly one rule on exactly one path, and put your own limits behind it. That is how you keep both the protection and the uptime.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article