Slaying Demons & Deploying Pods: Running Real-Time DOOM-Triggered Kubernetes Grading on AWS Serverless

1. Introduction & Executive Summary

• In-Game Actions occur in milliseconds (e.g., picking up armor, defeating an enemy, or triggering a DOOM sector event).
• Infrastructure Operations (e.g., provisioning pods, testing network policies, verifying ingress controllers, running Pytest suites against Kubernetes API endpoints) frequently take between 30 seconds to several minutes.

The 29-Second API Gateway Hard Timeout Constraint

2. High-Level AWS Serverless Architecture

3. AWS Service Matrix

4. The Abstract Client Trigger Model

1. In-Game Action: When a player interacts with a target element (e.g., picking up an item or triggering a DOOM event), the game client constructs a simple JSON frame:
   {
   "action": "talk",
   "apiKey": "USER_ENCRYPTED_API_KEY",
   "game": "game01",
   "npc": "doom"
   }
2. Local Game Pause: The game client immediately pauses local interactions and displays a status overlay.
3. Asynchronous Waiting: The game client listens on the WebSocket channel for state updates pushed asynchronously by AWS Lambda via API Gateway.

5. Overcoming the 29-Second Timeout: Edge Real-Time Layer & Concurrency Locks

5.1 How Asynchronous WebSocket Dispatch Defeats the 29s Limit

1. Instant Edge Response (<50ms): When the player triggers a test in DOOM, game-ws-handler authenticates the connection, acquires the DynamoDB lock, and returns an immediate "QUEUED" acknowledgement over the WebSocket channel. This finishes the API Gateway synchronous route execution in under 50 milliseconds—well below the 29-second threshold.
2. Unconstrained Background Execution: game-ws-handler asynchronously invokes game-command-handler via boto3. Because the worker runs asynchronously outside the HTTP request-response loop, it has access to AWS Lambda's full 15-minute execution budget to perform intensive Pytest evaluations, retry transient cluster connectivity, and wait for Kubernetes pod readiness.
3. Out-of-Band Push Notifications: As grading progresses, the durable worker calls the API Gateway Management API (post_to_connection) to stream live progress updates (RUNNING, PASSED, FAILED, COMPLETED) directly back through API Gateway to the browser connection.

5.2 Connection Lifecycle & Registration

• $connect Route: API Gateway invokes the connection handler, decrypts the Fernet API key to extract the user's email, and saves the connection row into GameWsConnectionTable in Amazon DynamoDB:
  • connection_id (Partition Key)
  • email
  • game
  • connected_at
• subscribe Route: Returns an immediate snapshot of the user's current progress (TaskStateTable) so browser refreshes instantly recover the exact question, instructions, and score without losing context.

5.3 Concurrency Safety with DynamoDB Execution Guard Locks

1. Rate Limiting: A 2-second per-user cooldown check prevents accidental trigger spamming.
2. Execution Guard Lock: Before invoking the durable worker, the handler attempts to create an atomic conditional write in DynamoDB under the key scope:

6. Dual State Machine & Task Progression Engine

6.1 High-Level Task Lifecycle

• NOT_STARTED: Initial state before the player attempts the task. Task instructions render dynamic Jinja2 session variables (e.g., {{ namespace }}).
• IN_PROGRESS: The task is active. The user is actively modifying their Kubernetes cluster to satisfy task requirements.
• COMPLETED: All phases (setup, ready, challenge, check) passed successfully. Points are awarded.
• ABANDONED: (Exam Mode) Maximum allowed attempts exceeded. Task locks until an explicit reset occurs.

6.2 Granular Phase Lifecycle

• PENDING: Phase is queued for execution.
• RUNNING: AWS Lambda is actively executing the Pytest suite against the target cluster.
• PASSED: The Pytest suite returned exit code 0.
• FAILED: The Pytest suite failed assertions. Detailed report uploaded to S3.

7. DOOM Execution Engine (DOOM_TASK_SOURCE) & Phase Auto-Chaining

7.1 Multi-Phase Auto-Chaining

1. Bootstrap Trigger (setup -\> ready):
   • Automatically prepares student cluster resources (setup).
   • Immediately verifies cluster readiness (ready).
   • Advances state to challenge so instructions display what the player needs to configure next.
2. Gameplay Trigger (challenge -\> check):
   • Automatically runs cluster verification checks (check).
   • If check passes: Marks task COMPLETED and advances to the next task in the manifest.
   • If check fails: Rewinds status back to challenge, allowing the student to inspect error reports and try again.

7.2 Infinite Retries (DOOM_TASK_SOURCE Bypass Rules)

• Bypassing NPC Lockout: DOOM_TASK_SOURCE bypasses single-NPC assignment locks (NpcLockTable), allowing seamless task progression.
• Bypassing Attempt Lockout: Attempt counting (_counts_attempts) is disabled, permitting infinite retries so students can debug their Kubernetes YAML manifests and re-trigger checks as many times as needed.
• Dynamic Parameter Rendering: Instructions for NOT_STARTED tasks render Jinja2 parameters dynamically from session data (e.g., {{ namespace }}, {{ pod_name }}) before execution begins, ensuring students always see actual cluster target values.

8. Exam Mode vs. Game/DOOM Mode Execution Matrix

9. Executing Kubernetes Unit Tests inside AWS Lambda

1. Private Test Source Fetching:
   • Pytest test files are packaged and stored in an encrypted Amazon S3 Bucket (GameSourceTable).
   • Upon invocation, Lambda downloads the task .zip archive and extracts it into /tmp/\<game\>.
2. Dynamic Kubeconfig Assembly:
   • Lambda loads the student's K8s endpoint URL, client certificate, and client key from AccountTable.
   • It writes client cert/key files to /tmp and dynamically generates a temporary kubeconfig file.
3. Pytest Execution & Report Upload:
   • Lambda invokes Pytest via python -m pytest programmatically:

1. • If assertions fail, the generated HTML test report is uploaded to Amazon S3 (TestResultBucket), and a pre-signed S3 URL is returned in the WebSocket payload so the player can view exact Pytest tracebacks.

10. AWS Best Practices & Performance Optimizations

10.1 Lazy Initialization for Cold Start Reduction

10.2 Flat Durable Step Wrappers

10.3 Ephemeral /tmp Filesystem Hygiene across Warm Invocations

11. Infrastructure as Code (AWS SAM template.yaml)

12. Conclusion

• Strict Single Entry Point: The browser DOOM client connects exclusively to Amazon API Gateway WebSockets. Durable Lambda workers stream updates back via API Gateway's Management API.
• Overcoming the 29s Limit: Decoupling requests via API Gateway WebSockets allows instant acknowledgements (<50ms) while durable Lambda workers utilize up to 15 minutes of execution time for long-running Kubernetes cluster evaluations.
• Zero Idle Costs: The entire backend runs on pay-per-use AWS serverless primitives—no EC2 clusters, no persistent WebSocket servers.
• Concurrency Safety: DynamoDB-backed Execution Guard locks guarantee single-threaded cluster modifications without race conditions.
• Resilient State Management: Dual state machines ensure deterministic task progression, auto-chaining, and infinite retries for continuous hands-on learning.

Resources & References

• AWS Serverless Application Model (SAM) 
• Amazon API Gateway WebSocket APIs 
• AWS Lambda Durable Execution Patterns 
• Amazon DynamoDB On-Demand Capacity 
• doom.ts source