OpenSearch for AWS WAF Anomaly Detection - Using ML Capabilities to Detect and Visualize Anomalies in WAF Logs

This article is a step-by-step technical guide for building an ML-powered WAF anomaly detection system using Amazon OpenSearch Service. It shows how to stream AWS WAF logs through Amazon Data Firehose into OpenSearch, where the built-in Random Cut Forest (RCF) algorithm learns normal traffic patterns and automatically flags deviations — like slow-and-low attacks or unusual geographic spikes, without requiring any ML expertise.

Anindita Paul

Technical Account Manager

Published Jun 16, 2026

Last modified Jun 22, 2026

Introduction

Web applications face an ever-evolving threat landscape: sophisticated DDoS attacks, SQL injection attempts, credential stuffing, and bot-driven abuse. While AWS WAF provides powerful rule-based protection, traditional threshold-based monitoring often misses subtle, emerging attack patterns that don't trigger predefined rules.

Amazon OpenSearch Service solves this challenge with built-in machine learning capabilities that detect anomalies in real-time streaming data — without requiring any ML expertise. By combining AWS WAF's comprehensive logging with OpenSearch's Random Cut Forest (RCF) algorithm, you can build an intelligent security monitoring system that autonomously identifies unusual patterns, surfaces emerging threats, and alerts your team before damage occurs.

This post walks you through building an end-to-end WAF anomaly detection pipeline — from log ingestion to ML-powered detection to actionable visualization.

Why ML-Based Anomaly Detection for WAF Logs?

Traditional WAF monitoring relies on static thresholds: alert when blocked requests exceed X per minute, or when a single IP generates more than Y requests. These approaches have critical limitations:

1. They miss slow-and-low attacks

Adversaries who stay just below thresholds are invisible to threshold-based systems.

Analogy: A shoplifter who steals one item per visit, every day, for months. No single visit triggers the theft alarm because they stay under the threshold.

In web terms: An attacker sends 99 requests/minute when your alert threshold is 100. They're attacking you — just slowly enough to stay invisible.

2. They generate false positives during legitimate traffic spikes

Product launches and marketing campaigns cause false alarms that train teams to ignore alerts.

Analogy: A store fire alarm that goes off every time you have a big sale because too many people are in the building. Now your staff ignores the alarm entirely.

In web terms: Your marketing team launches a viral campaign. Traffic spikes 5x. Your static rules scream ATTACK! and your security team wastes hours investigating for no reason.

3. They cannot adapt to evolving baselines

Seasonal patterns and organic growth make static thresholds obsolete.

Analogy: Setting your thermostat to 72°F in January. By July, 72°F means the AC runs non-stop because "normal" changed with the seasons.

In web terms: Your app grows from 1,000 to 50,000 daily users over 6 months. Your old threshold of "alert at 500 requests/minute" now fires constantly because that's just… Tuesday.

4. They require constant manual tuning

As traffic patterns change, manually maintained rules become exhausting and error-prone.

Analogy: Think of a security guard who needs new instructions every time the neighborhood changes.

ML-based anomaly detection learns what "normal" looks like for your specific application and automatically flags deviations — an unusual geographic distribution of requests, an abnormal ratio of blocked-to-allowed traffic, or a sudden shift in targeted URI patterns.

Architecture Overview

The solution uses a fully serverless pipeline that streams WAF logs through transformation and into OpenSearch for real-time anomaly detection.

Component	Role	Key Configuration
AWS WAF	Generates web ACL logs (allowed/blocked requests, IP, URI, rules matched)	Enable logging to Firehose delivery stream
Amazon Data Firehose	Streams WAF logs in near real-time with buffering	60-second buffer interval, 5 MB buffer size
AWS Lambda	Transforms/enriches log data (geo-IP lookup, field normalization, timestamp formatting)	Python 3.12 runtime, 256 MB memory
Amazon OpenSearch Service	Stores, indexes, and runs anomaly detection via RCF algorithm	t3.small.search with 1 node (test-grade)
OpenSearch Dashboards	Visualizes anomalies, creates alerting rules	Custom dashboards with anomaly overlays
Amazon SNS	Delivers alerts to security teams	Email/SMS natively; Slack and PagerDuty via Lambda or AWS Chatbot webhooks

The ML Engine: Random Cut Forest (RCF)

OpenSearch's anomaly detection is powered by the Random Cut Forest algorithm — an unsupervised ML technique purpose-built for streaming data anomaly detection. Here's why it's ideal for WAF logs:

No labeled training data required. RCF learns from the data stream itself, building a model of "normal" behavior. The algorithm needs a warm-up period of continuous data (see Step 8) to establish its baseline, and can cold-start from historical data if the index already spans enough time.
Real-time capability. Processes data points as they arrive, not in batch.
Captures short-term temporal context. Uses shingling (grouping consecutive data points) to detect shape changes in recent traffic patterns.
Assigns anomaly grades. Returns a score (0 to 1) indicating how anomalous each observation is.
Self-adapting. Continuously updates its model as traffic patterns evolve.

How RCF Works (Simplified)

Imagine a veteran doctor who has seen thousands of patients over 20 years. When a new patient walks in, the doctor instantly knows if something "feels off" — not because they memorized a checklist, but because they have deep pattern recognition from years of experience. RCF works the same way:

It builds experience. The algorithm continuously observes your traffic data, building a rich understanding of what "normal" looks like across multiple dimensions (volume, geography, timing, behavior).
It spots what doesn't fit. When a new data point arrives, RCF asks: "If I tried to fit this into my existing understanding of normal, how much would I have to rearrange my mental model?" A routine Monday morning spike? No rearrangement needed. A sudden flood of requests from a country you've never seen? Major rearrangement → high anomaly score.
It forgets gracefully. Like a doctor who stays current with medical advances, RCF gradually phases out old patterns and adapts to your evolving traffic. What was anomalous six months ago might be normal today.
It scores confidence. Every observation gets a grade from 0 (completely normal) to 1 (highly anomalous), letting you set your own sensitivity threshold.

Implementation Walkthrough

This section walks you through deploying the complete WAF anomaly detection pipeline — from infrastructure provisioning to receiving your first anomaly alert email. All resources are deployed using AWS CloudFormation for repeatability.

Step 1: Deploy the Infrastructure (CloudFormation)

A single CloudFormation template provisions all pipeline components: the OpenSearch domain, Firehose delivery stream, Lambda transform function, WAF Web ACL, SNS alert topic, and supporting IAM roles.

Component	Purpose
OpenSearch Service domain	Stores WAF logs with fine-grained access control enabled
Firehose delivery stream	Buffers and delivers logs (named `aws-waf-logs-<env>`)
Lambda function	Transforms raw WAF logs into a clean, indexed format
WAF Web ACL	AWS Managed Rules: Common Rule Set + SQL Injection rules
SNS topic	Delivers anomaly alert notifications
S3 bucket	Firehose failed delivery backup

Note: The Lambda function (deployed inline via the template) derives three fields not native to WAF logs: request_size (from Content-Length header), is_blocked (1 if BLOCK, 0 otherwise), and rule_group_count (length of ruleGroupList). These power Detector 3 and the index mappings in Step 6.

Deploy Command:

1
2
3
4
5
6
7
8
9
aws cloudformation create-stack \
  --stack-name waf-anomaly-detection \
  --template-body file://waf-anomaly-detection-stack.yaml \
  --parameters \
    ParameterKey=EnvironmentName,ParameterValue=waf-anomaly \
    ParameterKey=OpenSearchMasterUserPassword,ParameterValue='<YourSecurePassword>' \
    ParameterKey=AlertEmail,ParameterValue=your-email@example.com \
  --capabilities CAPABILITY_NAMED_IAM \
  --region us-east-1

Verify Deployment:

1
2
3
4
aws cloudformation describe-stacks \
  --stack-name waf-anomaly-detection \
  --query 'Stacks[0].Outputs' \
  --output table

Note: The OpenSearch domain takes approximately 15–20 minutes to provision. The deploying IAM user needs permissions for: CloudFormation, OpenSearch, Firehose, Lambda, IAM, S3, SNS, WAFv2, and CloudWatch Logs.

Note: The template defaults to a test-sized configuration (t3.small.search, 1 node). For production workloads, update the OpenSearchInstanceType parameter to r6g.large.search and increase InstanceCount to 3 with zone awareness enabled.

Step 2: Subscribe to Alert Notifications

The stack creates an SNS topic, but you must confirm your email subscription before alerts can be delivered.

1
2
3
4
5
aws sns subscribe \
  --topic-arn arn:aws:sns:us-east-1:<account-id>:<env>-anomaly-alerts \
  --protocol email \
  --notification-endpoint your-email@example.com \
  --region us-east-1

📬 Check your inbox and click the Confirm subscription link in the AWS notification email before proceeding.

Step 3: Attach the Web ACL to Your Application

The WAF Web ACL must be associated with a resource (ALB, API Gateway, or CloudFront) to start generating logs.

1
2
3
4
aws wafv2 associate-web-acl \
  --web-acl-arn <WebACLArn-from-stack-outputs> \
  --resource-arn arn:aws:elasticloadbalancing:us-east-1:<account-id>:loadbalancer/app/<alb-name>/<id> \
  --region us-east-1

Tip: For testing without an existing ALB, deploy a test ALB stack that creates a VPC, two public subnets, and an internet-facing ALB with a fixed "200 OK" response. Associate the Web ACL during stack creation.

Step 4: Configure the OpenSearch Access Policy

OpenSearch's fine-grained access control requires an open resource-based policy so browser-based Dashboards access can reach the login page. Authentication is enforced by the internal user database.

Save the following as open-access-policy.json:

1
2
3
4
5
6
7
8
9
10
11
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "AWS": "*" },
      "Action": "es:ESHttp*",
      "Resource": "arn:aws:es:us-east-1:<account-id>:domain/<env-name>/*"
    }
  ]
}

Update Access Policy:

1
2
3
4
aws opensearch update-domain-config \
  --domain-name <env-name> \
  --access-policies file://open-access-policy.json \
  --region us-east-1

Note: This access policy update takes 5–10 minutes to apply.

⚠️ SECURITY NOTE: This open policy (Principal:* with Action:es:ESHttp*) is only safe because fine-grained access control (FGAC) and HTTPS are enforcing authentication. For production deployments, add an IP-based or VPC-based condition to the policy and scope the action.

Step 5: Map the Firehose Role in OpenSearch

Firehose needs permission to write data into OpenSearch indices. With fine-grained access control enabled, the Firehose IAM role must be mapped to a custom OpenSearch role scoped to write to waf-logs-* only. This follows the principle of least privilege.

Step 5a: Create a Custom Role Scoped to `waf-logs-*` (Dev Tools)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
PUT _plugins/_security/api/roles/firehose-waf-writer
{
  "cluster_permissions": [
    "cluster:monitor/main",
    "cluster:monitor/health"
  ],
  "index_permissions": [
    {
      "index_patterns": ["waf-logs-*"],
      "allowed_actions": [
        "indices:data/write/index",
        "indices:data/write/bulk",
        "indices:admin/mapping/put",
        "indices:admin/create"
      ]
    }
  ]
}

Step 5b: Map the Firehose IAM Role (Dev Tools)

1
2
3
4
PUT _plugins/_security/api/rolesmapping/firehose-waf-writer
{
  "backend_roles": ["arn:aws:iam::<account-id>:role/<env>-firehose-role"]
}

Step 6: Apply the Index Template

The index template ensures WAF log fields are mapped with correct types (keyword, ip, date) for efficient aggregation and anomaly detection.

⚠️ Important: This template MUST be applied before the first ingestion. Once a field's type is set by OpenSearch auto-mapping, it cannot be changed in place. If fields are already mapped incorrectly, the only fix is to create a new index with the correct mapping and reindex the data into it (with an alias swap to avoid downtime).

Dev Tools Query:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
PUT _index_template/waf-logs-template
{
  "index_patterns": ["waf-logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 0
    },
    "mappings": {
      "properties": {
        "@timestamp":       { "type": "date" },
        "action":           { "type": "keyword" },
        "client_ip":        { "type": "ip" },
        "country":          { "type": "keyword" },
        "uri":              { "type": "keyword" },
        "http_method":      { "type": "keyword" },
        "terminating_rule": { "type": "keyword" },
        "rule_group_count": { "type": "integer" },
        "is_blocked":       { "type": "integer" },
        "request_size":     { "type": "long" }
      }
    }
  }
}

Note: number_of_replicas is set to 0 for the test configuration since this walkthrough uses a single-node domain. For production multi-node deployments, set number_of_replicas to 1 or higher.

Note: The fields is_blocked and rule_group_count are derived fields that the Lambda transformation function (Step 1) populates from the raw WAF log. Ensure your Lambda function includes logic to set these values.

Step 7: Generate Traffic and Verify Data Flow

Send requests to your WAF-protected endpoint to produce both ALLOW and BLOCK log entries. Firehose buffers for 60 seconds before flushing to OpenSearch.

⚠️ Important: Anomaly detectors need continuous data across consecutive detection intervals for hours before they can build a reliable baseline and transition from INIT to RUNNING state. The short curl bursts below illustrate the request patterns, but you must run sustained traffic generation for 3+ hours.

Normal Traffic Pattern (Example)

1
2
3
4
# Generate normal traffic
for i in $(seq 1 100); do
  curl -s http://<your-alb-endpoint>/ > /dev/null
done

Attack Traffic Patterns (Examples)

1
2
3
4
5
# SQL injection patterns
curl -s "http://<your-alb-endpoint>/?id=1%27%20OR%20%271%27%3D%271"

# XSS patterns
curl -s "http://<your-alb-endpoint>/?q=%3Cscript%3Ealert(1)%3C/script%3E"

Sustained Traffic Generator (Run for 3+ Hours)

Run this script for 3+ hours to build the detector baseline. With shingle_size:8 and 5-min intervals, expect 2–3 hours to reach RUNNING state.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
#!/bin/bash
# Sustained traffic generator - run for 3+ hours
# Detectors require continuous data across consecutive intervals
# With shingle_size:8 and 5-min intervals, expect ~2-3 hours to reach RUNNING state

END_TIME=$((SECONDS + 10800))  # Run for 3 hours (10800 seconds)
INTERVAL=30                    # Send a batch every 30 seconds

while [ $SECONDS -lt $END_TIME ]; do
  # Normal traffic batch
  for i in $(seq 1 20); do
    curl -s http://<your-alb-endpoint>/ > /dev/null &
  done

  # Occasional "attack" traffic to create variety in the baseline
  if [ $((RANDOM % 5)) -eq 0 ]; then
    curl -s "http://<your-alb-endpoint>/?id=1%27%20OR%20%271%27%3D%271" > /dev/null
  fi

  wait
  echo "Batch sent at $(date). Elapsed: ${SECONDS}s / ${END_TIME}s"
  sleep $INTERVAL
done
echo "Traffic generation complete. Detectors should now be in RUNNING state."

Tip: Use a cron job (crontab -e) to run a curl batch every minute if you prefer a cron-based approach. The key requirement is that data arrives at consistent intervals for the full duration of the baseline-building period.

Verify Data in OpenSearch (Dev Tools)

1
2
GET waf-logs-*/_count
GET waf-logs-*/_search?size=5

Step 8: Create Anomaly Detection Jobs

With data flowing, create detectors that use the RCF algorithm to learn traffic baselines and flag deviations. Each detector targets a different attack vector.

Note on window_delay: This should be at least the end-to-end ingest latency (Firehose buffer + delivery + index refresh, typically ~2 minutes) so the detector only evaluates windows whose data is fully searchable.

Detector 1: Volumetric Attack Detection

Detects unusual spikes in blocked request counts or sudden increases in unique attacking IPs.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
POST _plugins/_anomaly_detection/detectors
{
  "name": "waf-volumetric-anomaly",
  "description": "Detects unusual spikes in blocked requests",
  "time_field": "@timestamp",
  "indices": ["waf-logs-*"],
  "detection_interval": { "period": { "interval": 5, "unit": "Minutes" } },
  "window_delay": { "period": { "interval": 2, "unit": "Minutes" } },
  "feature_attributes": [
    {
      "feature_name": "blocked_request_count",
      "feature_enabled": true,
      "aggregation_query": {
        "blocked_count": {
          "filter": { "term": { "action": "BLOCK" } },
          "aggs": { "count": { "value_count": { "field": "action" } } }
        }
      }
    },
    {
      "feature_name": "unique_blocked_ips",
      "feature_enabled": true,
      "aggregation_query": {
        "unique_ips": {
          "filter": { "term": { "action": "BLOCK" } },
          "aggs": { "distinct_ips": { "cardinality": { "field": "client_ip" } } }
        }
      }
    }
  ],
  "shingle_size": 8
}

Design note: category_field is intentionally omitted. Using category_field:["country"] combined with cardinality("country") would not work — partitioning by country means each entity bucket always has exactly one distinct country (cardinality = 1), so the detector would never fire.

Detector 2: Geographic Anomaly Detection

Detects when requests arrive from an unusual number of distinct countries. Modeled as a single-stream detector (no category_field) because the goal is to track the global count of distinct countries.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
POST _plugins/_anomaly_detection/detectors
{
  "name": "waf-geo-anomaly",
  "description": "Detects unusual geographic request patterns",
  "time_field": "@timestamp",
  "indices": ["waf-logs-*"],
  "detection_interval": { "period": { "interval": 10, "unit": "Minutes" } },
  "window_delay": { "period": { "interval": 2, "unit": "Minutes" } },
  "feature_attributes": [
    {
      "feature_name": "country_diversity",
      "feature_enabled": true,
      "aggregation_query": {
        "country_count": { "cardinality": { "field": "country" } }
      }
    }
  ],
  "shingle_size": 4
}

Detector 3: Application-Layer Attack Detection

Detects when attackers probe many different URI paths or send unusually large requests.

⚠️ Warning: The request_size, is_blocked, and rule_group_count fields are not native to AWS WAF logs — they are derived by the Lambda function in Step 1. If Lambda does not populate these fields, features using them will return null and won't contribute to detection.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
POST _plugins/_anomaly_detection/detectors
{
  "name": "waf-app-layer-anomaly",
  "description": "Detects unusual URI targeting patterns",
  "time_field": "@timestamp",
  "indices": ["waf-logs-*"],
  "detection_interval": { "period": { "interval": 5, "unit": "Minutes" } },
  "window_delay": { "period": { "interval": 2, "unit": "Minutes" } },
  "feature_attributes": [
    {
      "feature_name": "blocked_uri_diversity",
      "feature_enabled": true,
      "aggregation_query": {
        "uri_diversity": {
          "filter": { "term": { "action": "BLOCK" } },
          "aggs": { "unique_uris": { "cardinality": { "field": "uri" } } }
        }
      }
    },
    {
      "feature_name": "avg_request_size",
      "feature_enabled": true,
      "aggregation_query": {
        "avg_size": { "avg": { "field": "request_size" } }
      }
    }
  ],
  "shingle_size": 8
}

Start Each Detector

Use the _id returned in the creation response:

1
POST _plugins/_anomaly_detection/detectors/<detector-id>/_start

Step 9: Configure Alerting

⚠️ Important: Detectors require continuous data across multiple detection intervals to build a baseline. With shingle_size:8 and a 5-minute interval, expect approximately 2–3 hours before the detector transitions from INIT to RUNNING. This is why the sustained traffic generator in Step 7 is essential.

9a. Create a Notification Channel

This connects OpenSearch alerting to your SNS topic.

Create IAM Role (CLI):

1
2
3
4
5
6
7
8
9
10
11
12
13
# Create a role that OpenSearch's managed service can assume
aws iam create-role \
  --role-name <env>-sns-alert-role \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": { "Service": "es.amazonaws.com" },
        "Action": "sts:AssumeRole"
      }
    ]
  }'

Grant SNS Publish Permission:

Save as sns-publish-policy.json:

1
2
3
4
5
6
7
8
9
10
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sns:Publish",
      "Resource": "arn:aws:sns:us-east-1:<account-id>:<env>-anomaly-alerts"
    }
  ]
}

1
2
3
4
aws iam put-role-policy \
  --role-name <env>-sns-alert-role \
  --policy-name sns-publish \
  --policy-document file://sns-publish-policy.json

Note: The OpenSearch service principal (es.amazonaws.com) is sufficient for the managed service to assume this role and publish to SNS.

Create Notification Channel (Dev Tools):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
POST _plugins/_notifications/configs
{
  "config_id": "waf-anomaly-sns-channel",
  "config": {
    "name": "WAF Anomaly SNS",
    "description": "SNS channel for WAF anomaly alerts",
    "config_type": "sns",
    "is_enabled": true,
    "sns": {
      "topic_arn": "arn:aws:sns:us-east-1:<account-id>:<env>-anomaly-alerts",
      "role_arn": "arn:aws:iam::<account-id>:role/<env>-sns-alert-role"
    }
  }
}

9b. Create the Alerting Monitor (Dashboards UI)

The recommended approach is to use the OpenSearch Dashboards UI:

Note: OpenSearch stores anomaly results in protected system indices. Monitors created through the UI use the plugin's internal search, which has the necessary permissions.

Navigate to OpenSearch Plugins → Anomaly Detection
Click on your detector (e.g., waf-app-layer-anomaly)
Select the Set up alerts tab
Set Anomaly grade threshold to 0.7
Select notification channel: WAF Anomaly SNS
Save the monitor

Step 10: Trigger and Verify an Anomaly Alert

Once detectors reach RUNNING state, create a traffic pattern that deviates dramatically from the learned baseline.

Send Burst Attack Traffic

Send 500 requests to random URI paths with SQL injection payloads:

1
2
3
for i in $(seq 1 500); do
  curl -s "http://<your-alb-endpoint>/path-$RANDOM?x=$RANDOM%27%20DROP%20TABLE" > /dev/null
done

Verify Anomaly Was Detected (after 5–10 minutes)

1
2
3
4
5
6
7
8
9
10
11
12
13
POST _plugins/_anomaly_detection/detectors/results/_search
{
  "size": 5,
  "query": {
    "bool": {
      "must": [
        { "term": { "detector_id": "<your-detector-id>" } },
        { "range": { "anomaly_grade": { "gt": 0 } } }
      ]
    }
  },
  "sort": [{ "data_end_time": "desc" }]
}

A successful anomaly result includes:

anomaly_grade: 0 to 1 (severity; values above 0.7 are high severity)
confidence: 0 to 1 (model certainty; increases over time)
feature_data: The actual metric values during the anomalous interval
attribution: Which feature contributed most to the anomaly

Note: Confidence starts low (~0.2) for newly initialized detectors and increases over days of operation. For initial testing, set the anomaly grade threshold to 0.7 without a confidence requirement. In production (after 1–2 weeks), add a confidence threshold of 0.5+ to reduce false positives.

📧 Expected outcome: Check your email. You should receive an alert from the SNS topic with the anomaly details.

Troubleshooting Quick Reference

Symptom	Likely Cause	Fix
No data in `waf-logs-*`	Firehose role can't write to OpenSearch	Map the Firehose role in security role mappings (Step 5)
Detector stuck in `INIT`	Not enough continuous data	Maintain steady traffic for 2–3 hours using the sustained generator (Step 7)
"Cannot get training data" error	No data in the detection intervals	Ensure traffic is flowing and check timestamp alignment
Monitor shows 0 hits	System index permissions	Use the Dashboards UI to create monitors (Step 9b)
SNS "AccessDenied" error	OpenSearch can't assume the SNS role	Verify the trust policy uses only the `es.amazonaws.com` service principal (Step 9a)
Fields mapped as "text"	Index created before template was applied	Use `.keyword` sub-fields or reindex after applying the template (Step 6)

Visualization Best Practices

Build OpenSearch Dashboards that provide immediate situational awareness:

Anomaly Timeline — Time-series chart with anomaly confidence bands overlaid on request volume.
Geographic Heatmap — World map showing anomaly intensity by country.
Top Offenders Table — IPs/URIs with highest anomaly correlation during detection windows.
Rule Trigger Correlation — Side-by-side view of anomaly events vs. WAF rule activations.
Detector Health Panel — Status of all active detectors with last anomaly timestamps.

Conclusion

By combining AWS WAF's comprehensive logging with OpenSearch's built-in ML capabilities, you can build an intelligent security monitoring system that:

✅ Detects threats that rule-based systems miss
✅ Requires zero ML expertise to deploy and operate
✅ Adapts automatically to your application's evolving traffic patterns
✅ Integrates with your existing alerting and incident response workflows

The entire solution can be deployed using infrastructure-as-code (CloudFormation/CDK) and requires no ongoing ML model management. OpenSearch handles model training, updating, and inference automatically.

Published by the ES-TAM-Builder community.

Follow our ES-TAM-Builder Space for more tools, guides, and technical deep-dives from fellow TAMs

# amazon-opensearch # opensearch # generative-ai

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

OpenSearch for AWS WAF Anomaly Detection - Using ML Capabilities to Detect and Visualize Anomalies in WAF Logs

Anindita Paul

Introduction

Why ML-Based Anomaly Detection for WAF Logs?

1. They miss slow-and-low attacks

2. They generate false positives during legitimate traffic spikes

3. They cannot adapt to evolving baselines

4. They require constant manual tuning

Architecture Overview

The ML Engine: Random Cut Forest (RCF)

How RCF Works (Simplified)

Implementation Walkthrough

Step 1: Deploy the Infrastructure (CloudFormation)

Step 2: Subscribe to Alert Notifications

Step 3: Attach the Web ACL to Your Application

Step 4: Configure the OpenSearch Access Policy

Step 5: Map the Firehose Role in OpenSearch

Step 5a: Create a Custom Role Scoped to waf-logs-* (Dev Tools)

Step 5b: Map the Firehose IAM Role (Dev Tools)

Step 6: Apply the Index Template

Step 7: Generate Traffic and Verify Data Flow

Normal Traffic Pattern (Example)

Attack Traffic Patterns (Examples)

Sustained Traffic Generator (Run for 3+ Hours)

Verify Data in OpenSearch (Dev Tools)

Step 8: Create Anomaly Detection Jobs

Detector 1: Volumetric Attack Detection

Detector 2: Geographic Anomaly Detection

Detector 3: Application-Layer Attack Detection

Start Each Detector

Step 9: Configure Alerting

9a. Create a Notification Channel

9b. Create the Alerting Monitor (Dashboards UI)

Step 10: Trigger and Verify an Anomaly Alert

Send Burst Attack Traffic

Verify Anomaly Was Detected (after 5–10 minutes)

Troubleshooting Quick Reference

Visualization Best Practices

Conclusion

Step 5a: Create a Custom Role Scoped to `waf-logs-*` (Dev Tools)