AWS Builder Center

πŸš€ Mastering the Production Crunch: Deploying a Multi-Service AI WAFR Agent on AWS Fargate

This log details the tense, one-day production deployment of an AI-powered AWS WAFR platform. Orchestrated via 13 AWS CDK CloudFormation stacks, the multi-service architecture integrates a security scanner, GenAI, a graph database, PostgreSQL, Redis, and React. A routine launch became a 6-hour troubleshooting marathon, exposing 9 critical failures across container networking, configuration, Lambda packaging, and AI integration. It offers the engineering community an unfiltered, battle-tested post-mortem.

Cloud Architect | AWS Community Builder | AI & DevSecOps
🎯 The Business Case
Organizations running workloads on AWS need continuous security assessments aligned with the Well-Architected Framework. Traditional approaches involve expensive consulting engagements ($15K–$50K per review) or manual audits that take weeks to complete.
The WAFR Scan Portal solves this by combining:
  • Security Scanner for automated cloud infrastructure checking (400+ checks across AWS services)
  • Amazon Bedrock with an advanced LLM model for AI-powered recommendations
  • Graph Database for relationship mapping between security findings
  • A custom React SPA that guides users through the entire assessment workflow
The goal: deliver in minutes what traditionally takes weeks, at a fraction of the cost, with zero human intervention required after credential submission.
πŸ—οΈ The Production Architecture
High-Level System Design
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ WAFR Scan Portal β€” Production Architecture β”‚
β”‚ Region: [AWS_REGION] β”‚
β”‚ β”‚
β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚
β”‚ β”‚ β”‚ β”‚ CloudFront │────▢│S3 Bucket (React SPA) β”‚ β”‚
β”‚ β”‚ End Users │─────▢│ Distribution β”‚ β”‚β€’ index.html β”‚ β”‚
β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚β€’ static/js/*.js β”‚ β”‚
β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚β€’ static/css/*.css β”‚ β”‚
β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
β”‚ β”‚ β”‚
β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚
β”‚ β”‚ β”‚ β”‚
β”‚ β–Ό β–Ό β”‚
β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚
β”‚ β”‚ API Gateway β”‚ β”‚ Application Load Balancer β”‚ β”‚
β”‚ β”‚ (REST API) β”‚ β”‚ (Internal Services) β”‚ β”‚
β”‚ β”‚ + Cognito Auth β”‚ β”‚ β”‚ β”‚
β”‚ β”‚ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚
β”‚ β”‚ /scans β”‚ β”‚ β”‚ /api/* β†’ :8080 β”‚ β”‚ β”‚
β”‚ β”‚ /scans/{id}/creds β”‚ β”‚ β”‚ /* β†’ :3000 β”‚ β”‚ β”‚
β”‚ β”‚ /scans/{id}/prereqs β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚
β”‚ β”‚ /scans/{id}/assess β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
β”‚ β”‚ /scans/{id}/report β”‚ β”‚ β”‚
β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚
β”‚ β”‚ β”‚ β”‚
β”‚ β–Ό β–Ό β”‚
β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚
β”‚ β”‚ Lambda Functions (6) β”‚ β”‚ ECS Fargate Cluster β”‚ β”‚
β”‚ β”‚ β”‚ β”‚ β”‚ β”‚
β”‚ β”‚ β€’ wafr-form-handler β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚
β”‚ β”‚ β€’ wafr-credential-val β”‚ β”‚ β”‚ Frontend UIβ”‚β”‚ Core API β”‚ β”‚ β”‚
β”‚ β”‚ β€’ wafr-prerequisite β”‚ β”‚ β”‚ (Next.js) β”‚β”‚ (Django) β”‚ β”‚ β”‚
β”‚ β”‚ β€’ wafr-assessment β”‚ β”‚ β”‚ 1vCPU/2GB β”‚β”‚ 2vCPU/4GB β”‚ β”‚ β”‚
β”‚ β”‚ β€’ wafr-pdf-generator β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚
β”‚ β”‚ β€’ wafr-email-sender β”‚ β”‚ β”‚ β”‚
β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚
β”‚ β”‚ β”‚ β”‚ Scanner Worker β”‚ β”‚ β”‚
β”‚ β–Ό β”‚ β”‚ (FARGATE_SPOT) β”‚ β”‚ β”‚
β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ 4vCPU/8GB β”‚ β”‚ β”‚
β”‚ β”‚ Amazon Bedrock β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚
β”‚ β”‚ Agent: [AGENT_ID] β”‚ β”‚ β”‚ β”‚
β”‚ β”‚ Model: [LLM_MODEL] β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚
β”‚ β”‚ Knowledge Base: S3 β”‚ β”‚ β”‚ Graph Database β”‚ β”‚ β”‚
β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ 1vCPU/2GB β”‚ β”‚ β”‚
β”‚ β”‚ β”‚ graph.wafr.local β”‚ β”‚ β”‚
β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
β”‚ β”‚
β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚
β”‚ β”‚ Data Layer β”‚ β”‚
β”‚ β”‚ β”‚ β”‚
β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚
β”‚ β”‚ β”‚ RDS PostgreSQL β”‚ β”‚ ElastiCache β”‚ β”‚ DynamoDB β”‚ β”‚ β”‚
β”‚ β”‚ β”‚ (Scanner Data) β”‚ β”‚ Valkey/Redis β”‚ β”‚ (WAFR Scans) β”‚ β”‚ β”‚
β”‚ β”‚ β”‚ Port: 5432 β”‚ β”‚ Port: 6379 β”‚ β”‚ 3 Tables β”‚ β”‚ β”‚
β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚
β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
β”‚ β”‚
β”‚ Supporting Services: Cognito Β· SES Β· S3 (Reports) Β· CloudMap (DNS) β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
Network Architecture Detail
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ VPC (WafrVpcStack) β”‚
β”‚ β”‚
β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚
β”‚ β”‚ Public Subnets (2 AZs) β”‚ β”‚
β”‚ β”‚ β”‚ β”‚
β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚
β”‚ β”‚ β”‚ Application Load Balancer β”‚ β”‚ β”‚
β”‚ β”‚ β”‚ SecurityGroup: [ALB_SG_ID] β”‚ β”‚ β”‚
β”‚ β”‚ β”‚ Listeners: 80 β†’ 443 (HTTPS) β”‚ β”‚ β”‚
β”‚ β”‚ β”‚ Target Groups: β”‚ β”‚ β”‚
β”‚ β”‚ β”‚ β€’ /api/* β†’ core-api:8080 β”‚ β”‚ β”‚
β”‚ β”‚ β”‚ β€’ /* β†’ frontend-ui:3000 β”‚ β”‚ β”‚
β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚
β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
β”‚ β”‚
β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚
β”‚ β”‚ Private Subnets (2 AZs) β”‚ β”‚
β”‚ β”‚ β”‚ β”‚
β”‚ β”‚ ECS Tasks ─── SecurityGroup: [ECS_TASK_SG_ID] β”‚ β”‚
β”‚ β”‚ β”‚ β”‚ β”‚
β”‚ β”‚ β”œβ”€β”€β†’ RDS PostgreSQL ─── SecurityGroup: [RDS_SG_ID] β”‚ β”‚
β”‚ β”‚ β”œβ”€β”€β†’ ElastiCache Valkey ─── SecurityGroup: [REDIS_SG] β”‚ β”‚
β”‚ β”‚ └──→ Graph DB ─── SecurityGroup: [GRAPH_SG] β”‚ β”‚
β”‚ β”‚ β”‚ β”‚
β”‚ β”‚ Service Discovery (AWS Cloud Map): β”‚ β”‚
β”‚ β”‚ Namespace: wafr.local β”‚ β”‚
β”‚ β”‚ Service: graph β†’ graph.wafr.local β”‚ β”‚
β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
CloudFormation Stack Dependency Chain
CDKToolkit
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
CDKToolkit
β”‚
β–Ό
WafrVpcStack ──────────────────────────────────────────────────┐
β”‚ β”‚
β”œβ”€β”€β–Ά WafrAuthStack (Cognito) β”‚
β”‚ β”‚
β”œβ”€β”€β–Ά WafrDatabaseStack (DynamoDB) β”‚
β”‚ β”‚
β”œβ”€β”€β–Ά WafrSesStack (SES) β”‚
β”‚ β”‚
β”œβ”€β”€β–Ά WafrFrontendStack (S3 + CloudFront) β”‚
β”‚ β”‚
β”œβ”€β”€β–Ά WafrDataStack (RDS + ElastiCache) β”‚
β”‚ β”‚ β”‚
β”‚ β”œβ”€β”€β–Ά WafrGraphStack (Graph ECS + Cloud Map) β”‚
β”‚ β”‚ β”‚ β”‚
β”‚ β”‚ └──▢ WafrEcsStack (3 ECS Services) β”‚
β”‚ β”‚ β”‚ β”‚
β”‚ └──────────────────────── β”‚
β”‚ β–Ό β”‚
β”‚ WafrAlbStack (ALB) β”‚
β”‚ β”‚
β”œβ”€β”€β–Ά WafrLambdaStack (6 Lambda Functions) β”‚
β”‚ β”‚ β”‚
β”‚ └──▢ WafrApiStack (API Gateway + Routes) β”‚
β”‚ β”‚ β”‚
β”‚ └──▢ WafrBedrockAgentStack β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
Total Stacks: 13 | Deployment Order: Sequential with cross-stack references via Fn::ImportValue
🀿 The Production Log: 9 Problems, 6 Hours
Incident Overview Matrix
Target DepthDetected IssueImpact LayerMTTR
10mMissing Graph DB runtime dependencyGraph Infrastructure25 min
20mEncryption key environment variable mismatchApplication Config20 min
30mGunicorn loopback interface binding trapContainer Networking35 min
35mDjango security middleware host rejectionHTTP Gateway5 min
40mCloudFormation capacity provider deletion deadlockIaC Orchestration15 min
50mHoisted dependencies missing from Lambda bundleBuild & Assembly40 min
55mStricter Bedrock API validation logic schema errorAI Integration15 min
60mHardcoded Docker Compose service discovery inside UIContainer Forensics30 min
65mS3 data parsing validation anomaly (IAM CSV format)Business Logic20 min
🀿 Dive #1 β€” The Missing Graph Database (Depth: 10m)
Context: The underlying open-source application engine introduced a Graph Database as a required dependency for its relational-based security mapping model. This wasn't accounted for during the initial Fargate task definition provisioning.
Symptom:
1
2
3
4
Container: core-api
Exit Code: 1
Restart Count: Increasing
Health Checks: Never passing
CloudWatch Logs:
1
2
manage.py migrate: error: argument --database: invalid choice: 'admin'
django.core.exceptions.ImproperlyConfigured: Set the GRAPH_HOST environment variable
Analysis: The Django backend performs a strict check for database endpoints during module initialization. Lacking a running graph interface, the wrapper fails and crashes before spawning the WSGI server workers.
Resolution:We provisioned a dedicated isolated stack (WafrGraphStack) within the private network perimeter:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ WafrGraphStack β”‚
β”‚ β”‚
β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚
β”‚ β”‚ ECS Cluster: wafr-graph-cluster β”‚ β”‚
β”‚ β”‚ β”‚ β”‚
β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚
β”‚ β”‚ β”‚ Task Definition: wafr-graph:1 β”‚ β”‚ β”‚
β”‚ β”‚ β”‚ Image: [GRAPH_DATABASE_IMAGE]:latest β”‚ β”‚ β”‚
β”‚ β”‚ β”‚ Ports: 7474 (HTTP), 7687 (Bolt) β”‚ β”‚ β”‚
β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚
β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
β”‚ β”‚
β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚
β”‚ β”‚ AWS Cloud Map (Service Discovery) β”‚ β”‚
β”‚ β”‚ Namespace: wafr.local (Private DNS) β”‚ β”‚
β”‚ β”‚ Service: graph β†’ graph.wafr.local β”‚ β”‚
β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
The Core API resolves the location dynamic lookup at runtime via graph.wafr.local using AWS Cloud Map.
🀿 Dive #2 β€” The Encryption Key Cascade (Depth: 20m)Context: With the Graph DB accessible, the application wrapper progressed further in its execution loop but threw a validation error immediately after.
Symptom:
1
2
3
File "/home/appuser/backend/api/models.py", line 63
fernet = Fernet(settings.SECRETS_ENCRYPTION_KEY.encode())
ValueError: Fernet key must be 32 url-safe base64-encoded bytes.
Analysis: The framework relies on django-environ under the hood. To ensure configuration variables inject properly through the native engine wrapper, they must adhere strictly to the expected prefix conventions.
Resolution:
Renamed the target environment variables to append the explicit prefix requirement. The complete structural variable runtime footprint deployed onto ECS Fargate includes:
1
2
3
4
5
6
7
8
9
10
11
12
13
─────────────────────────────────────────────────────────────────────┐
β”‚ Core API β€” Required Environment Variables (ECS Fargate) β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ DJANGO_BIND_ADDRESS = 0.0.0.0 β”‚
β”‚ DJANGO_ALLOWED_HOSTS = * β”‚
β”‚ DJANGO_SECRETS_ENCRYPTION_KEY = [SECURE_32_BYTE_BASE64_KEY] β”‚
β”‚ DJANGO_SETTINGS_MODULE = config.django.production β”‚
β”‚ DJANGO_DEBUG = False β”‚
β”‚ GRAPH_HOST = graph.wafr.local β”‚
β”‚ GRAPH_PORT = 7687 β”‚
β”‚ POSTGRES_HOST = [RDS_PRIVATE_ENDPOINT] β”‚
β”‚ VALKEY_HOST = [ELASTICACHE_PRIVATE_ENDPOINT] β”‚
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
🀿 Dive #3 β€” The Loopback Trap (Depth: 30m)
Context: The container backend was now booting without immediate crashes. However, the external Application Load Balancer health checks consistently timed out.
Symptom: ECS cluster tasks stuck cycling endlessly. Stack update hanging in
CREATE_IN_PROGRESS.
CloudWatch Logs:
[INFO] Listening at: http://127.0.0.1:8080
[INFO] Using worker: gthread
Analysis: The WSGI server was explicitly binding to loopback (127.0.0.1). Because Fargate maps the task interface explicitly to its Elastic Network Interface (ENI), the load balancer routes incoming probes directly to the host private allocation IP, missing a loopback-locked server entirely.
Troubleshooting Matrix:
GUNICORN_BIND=0.0.0.0:8080 ──▢ Ignored (custom entrypoint configuration logic overrides standard parameters)
GUNICORN_CMD_ARGS=--bind=0.0.0.0:8080 ──▢ Ignored (intercepted by internal script execution maps)
DJANGO_BIND_ADDRESS=0.0.0.0 ──▢ Success
Resolution: The image logic requires explicitly setting DJANGO_BIND_ADDRESS inside the task definition to override the default local loopback constraint.
🀿 Dive #4 β€” Django's Gatekeeper (Depth: 35m)
Context: The server interface was now broad-binding to all adapters. The load balancer reached the open port, but queries returned HTTP 400 (Bad Request).
Symptom: Target group logs flagged state checks as failing consistently with code 400.
Analysis: Django's internal host header validation middleware drops any traffic matching non-whitelisted destinations. When the load balancer probes targets, it drops standard domain names, sending the bare destination private IP (e.g., 10.0.x.x) inside the host request parameters.
Resolution: Deployed DJANGO_ALLOWED_HOSTS=* to allow target group validation behind the corporate secure load balancer tier.
🀿 Dive #5 β€” The CloudFormation Deadlock (Depth: 40m)
Context: To guarantee clean state reconciliation across structural network modifications, we executed a stack tear-down request. The automation controller stalled.
Symptom:
DELETE_FAILED: ProwlerClusterXXXXXXXX
The specified capacity provider is in use and cannot be removed
(ClusterCapacityProviderAssociations)
Analysis: CloudFormation cannot delete an active ECS cluster resource while its automated deployment profiles remain structurally linked to target auto-scaling or Spot configurations, ignoring parent collection drop requests.
Resolution:
We executed cleanup routines directly from the CLI environment:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
# Step 1: Force drop cluster capacity allocations
aws ecs put-cluster-capacity-providers \
--cluster WafrEcsStack-Cluster-XXXXXXXX \
--capacity-providers [] \
--default-capacity-provider-strategy []

# Step 2: Deregister blocked task definitions
for rev in 8 9 10; do
aws ecs deregister-task-definition \
--task-definition WafrEcsStackCoreApiTaskDefXXXXXXXX:$rev
done

# Step 3: Re-trigger cleanup pipeline
aws cloudformation delete-stack --stack-name WafrEcsStack
🀿 Dive #6 β€” The Invisible node_modules (Depth: 50m)
Context: The infrastructure engine was handling API tasks smoothly. Attention shifted toward testing the companion orchestration handlers deployed onto the serverless compute layers.
Symptom:
1
2
Runtime.ImportModuleError: Error: Cannot find module 'uuid'
Require stack: /var/task/dist/handlers/form-handler.js
Analysis: The project organizes code bases utilizing an npm workspace monorepo layout. Dependencies are hoisted up to the root folder hierarchy. Because the asset deployment hook pointers were bound explicitly to sub-folder spaces, the generated build artifacts packaged clean code but missed the parent hoisted modules directory completely.
Resolution:
Executed a targeted build deployment pack injection routine:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# 1. Synthesize hoisted libraries down into the workspace sub-tier
cp -r node_modules backend/node_modules
rm -rf backend/node_modules/@wafr-portal

# 2. Link shared dependencies manually
cp -r shared/dist backend/node_modules/@wafr-portal/shared

# 3. Clean up non-production development frameworks to compress size footprint (88MB -> 33MB)
cd backend/node_modules
rm -rf @aws-cdk aws-cdk aws-cdk-lib constructs typescript eslint prettier jest ts-jest

# 4. Generate deployment bundle
cd backend && zip -qr /tmp/backend-lambda.zip dist/ node_modules/ package.json

# 5. Push payload to runtime engine targets
for fn in form-handler credential-validator prerequisite-checker assessment-orchestrator pdf-generator email-sender; do
aws lambda update-function-code --function-name "wafr-${fn}" --zip-file fileb:///tmp/backend-lambda.zip
done
🀿 Dive #7 β€” The Bedrock Agent Schema (Depth: 55m)
Context: The core processing pipelines rely on an automated Amazon Bedrock Agent leveraging an advanced LLM model to match system findings directly against the target security architecture criteria.
Symptom: Stack creation aborted abruptly with validation alerts pointing to the OpenAPI schema design configurations.
Analysis: Bedrock's endpoint execution integration engines enforce validation rules that are far more strict than standard API Gateway parsers. They strictly require explicit parameter schemas, complete application media declarations, and structured operationId definitions for all exposed entry points.
Resolution: Refactored the OpenAPI schemas to insert absolute compliance declarations across all endpoints.
🀿 Dive #8 β€” The Hardcoded Docker Hostname (Depth: 60m)
Context: The API layers were passing system states cleanly. Users authenticated successfully through Amazon Cognito and mapped core tasks. However, triggering user management or dashboard portal sections threw an error.
Symptom:
[auth][error] CallbackRouteError
[auth][cause] Error: Error in trying to get token
Analysis: Next.js bakes environment parameters directly into the optimized production bundles during compilation. The local development environment carried a hardcoded fallback mapping back to internal Docker Compose hostnames (http://core-api:8080), failing to resolve in an enterprise multi-service Fargate structure.
Resolution:
We executed an emergency stream edit patch adjustment across compiled asset pools inside a derived Dockerfile wrapper:
1
2
3
4
5
6
7
8
9
10
11
12
Dockerfile
FROM [ACCOUNT_ID].dkr.ecr.[AWS_REGION].amazonaws.com/wafr/ui:latest
RUN find /app/.next -name "*.js" -exec sed -i \
's|http://core-api:8080/api/v1|https://[ALB_PRODUCTION_DNS]/api/v1|g' \
{} +


Bash
docker build -t wafr-ui-patched .
docker tag wafr-ui-patched [ACCOUNT_ID].dkr.ecr.[AWS_REGION].amazonaws.com/wafr/ui:latest
docker push [ACCOUNT_ID].dkr.ecr.[AWS_REGION].amazonaws.com/wafr/ui:latest
aws ecs update-service --cluster wafr-cluster --service wafr-ui --force-new-deployment
🀿 Dive #9 β€” The CSV Format Mismatch (Depth: 65m)
Context: The application wrapper was operational. Users authenticated successfully and reached the validation ingestion stages. However, loading standard IAM credential files crashed the ingestion workflow.
Symptom: The user web interface flagged data streams as unreadable.
Analysis: The developer mock scripts expected a clean key-value serialization format. However, the live outputs exported directly from the AWS identity management console deliver an alternate layout featuring custom-wrapped comma-separated value column headers.
Resolution:
We refactored the extraction utility (credential-extractor.ts) to transparently normalize incoming structures on the fly:
1
2
3
4
5
6
TypeScript// Dual-mode structural normalization logic
if (row1.includes("Access key ID")) {
parseAsHeaderData(fileContent);
} else {
parseAsLabelValue(fileContent);
}
At the same time, we migrated the processor from legacy spreadsheet configurations over to raw CSV parsing pipelines, dropping heavy node dependencies while adding a collapsible policy assistance dashboard directly within the user UI layer.
🏁 Final State: Full Production Deployment
Following a systematic 6-hour remediation run, the complete multi-tier execution ecosystem reached absolute stable operational production mapping parameters:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ FINAL PRODUCTION STATE β”‚
β”‚ [DEPLOYMENT_TIMESTAMP] β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ β”‚
β”‚ INFRASTRUCTURE β”‚
β”‚ β”œβ”€β”€ 13 CloudFormation Stacks .............. ALL CREATE_COMPLETE βœ… β”‚
β”‚ β”œβ”€β”€ VPC + 4 Subnets (2 public, 2 private) Operational βœ… β”‚
β”‚ └── 5 Security Groups .................... Properly configured βœ… β”‚
β”‚ β”‚
β”‚ COMPUTE β”‚
β”‚ β”œβ”€β”€ ECS Fargate Services (4) ............. ALL Running 1/1 βœ… β”‚
β”‚ β”‚ β”œβ”€β”€ core-api (2vCPU/4GB) ............. Healthy, passing HC βœ… β”‚
β”‚ β”‚ β”œβ”€β”€ frontend-ui (1vCPU/2GB) .......... Healthy, login works βœ… β”‚
β”‚ β”‚ β”œβ”€β”€ scanner-worker (4vCPU/8GB) ....... Running scans βœ… β”‚
β”‚ β”‚ └── graph-db (1vCPU/2GB) ............. Healthy, bolt active βœ… β”‚
β”‚ β”œβ”€β”€ Lambda Functions (6) ................. ALL Operational βœ… β”‚
β”‚ └── Bedrock Agent (Nova Pro) ............. Active, responding βœ… β”‚
β”‚ β”‚
β”‚ DATA LAYER β”‚
β”‚ β”œβ”€β”€ RDS PostgreSQL ....................... Available βœ… β”‚
β”‚ β”œβ”€β”€ ElastiCache Valkey ................... Available βœ… β”‚
β”‚ β”œβ”€β”€ DynamoDB (3 tables) .................. Active βœ… β”‚
β”‚ └── Graph Database ....................... Connected βœ… β”‚
β”‚ β”‚
β”‚ QUALITY ASSURANCE BRAND RESILIENCE β”‚
β”‚ β”œβ”€β”€ Unit Tests Passed ..................... 196/196 Core Steps βœ… β”‚
β”‚ β”œβ”€β”€ QA Infrastructure Report ............. Green Status βœ… β”‚
β”‚ └── QA Application Execution .............. Validated βœ… β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
πŸ“‹ Lessons Learned
1. The Local-to-Cloud Architectural Paradigm Shift
Open-source assets are heavily optimized to support streamlined, single-host Docker Compose local execution workflows. Migrating these platforms into enterprise cloud environments requires rewriting assumption maps regarding interface attachment parameters, service availability visibility vectors, and network space isolation boundaries.
2. Treat Containers as Black Boxes: Inspect Configuration Layouts Early
Never assume standard environment variables will change runtime behaviour across customized tool builds. Gunicorn and Django wrapper modifications inside community images regularly divert from standard conventions. Inspecting custom configuration maps (like guniconf.py) must occur before designing cloud deployment steps.
3. Decouple Build Assets from Runtime Tasks within Next.js Implementations
Baking location endpoints directly into application chunks at build time remains the single most prevalent point of friction encountered across containerized web portal deployments. Target variable lookups must wrap tightly around clean public configuration parameters or feature structural injection hooks at startup to survive orchestrator transitions.
βš–οΈ Technical & Legal Safe Harbor Disclaimer
AUTHORSHIP AND INDEPENDENT CAPACITY: This publication is authored solely by me in my individual and private capacity. The views, methodologies, and technical workflows expressed herein are my own and do not necessarily reflect the official policy, position, or strategic direction of my current or former employers, clients, or any legal entity I am affiliated with.
INTELLECTUAL PROPERTY & CONFIDENTIALITY COMPLIANCE:
Zero Proprietary Disclosure: This content has been developed using publicly available information and personal research. No confidential information, trade secrets, internal proprietary source code, or proprietary infrastructure schemas belonging to my employer or any third party have been used, referenced, or disclosed.
Independent Development: The workflows described are based on general industry best practices and were not developed as a "work for hire" or as part of assigned organizational duties.
LIMITATION OF LIABILITY (NO WARRANTY): All code snippets, scripts, and architectural patterns are provided "AS IS" without warranty of any kind, express or implied.
COMPLIANCE: This contribution is made in good faith under the AWS Builder Terms and the MIT-0 License for any included source code.
What has been your biggest friction point with hybrid AWS deploymentsβ€”host quotas, networking latency, or operational tooling? I'm curious to hear what's breaking (or finally working) in the field. πŸ‘‡
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article