AWS Builder Center

Builder Center

CodeCompass Cloud Architecture: A Journey to Serverless AWS

CodeCompass Cloud Architecture: A Journey to Serverless AWS

Building a highly complex Tri-Modal RAG engine is difficult. Deploying it to the public internet while dealing with timeouts, serverless constraints, and asynchronous processing is an entirely different beast. This article details the step-by-step AWS Cloud architecture of CodeCompass, how data flows through the system, and a transparent look at the critical architectural mistakes I made and fixed along the way.


Note to Readers: This article dives deep into the backend AWS cloud infrastructure and deployment architecture of Code Compass. If you aren't familiar with the project yet, I highly recommend first reading my previous article on  what Code Compass actually does and how the AI Engine works  before diving into the serverless components below!


The AWS Serverless Flow

The entire backend is 100% serverless, eliminating the need to manage EC2 instances or long-running servers. Here is exactly how a request travels through the system from the moment you click "Index Repo" on the frontend.


1. The Entry Point: API Gateway & The RAG-Dispatcher Lambda

When a user submits a GitHub repository, the frontend sends a  POST  request to  /ingest . This request is intercepted by the AWS API Gateway and routed to the RAG-Dispatcher Lambda. The Dispatcher has one job: Act fast.
  1. It checks the DynamoDB cache. If the repo is already indexed, it skips the entire indexing process and returns repo is indexed to the frontend of the website.
  1. If the repo is not indexed, it writes an initial "processing" state to DynamoDB.
  1. It asynchronously invokes the Worker Lambda using  InvocationType='Event' .
  1. It instantly returns an HTTP 202 (Accepted) response to the frontend, closing the connection cleanly.

2. The Heavy Lifting: The RAG-Ingestion-Worker Lambda

Because the Dispatcher triggered it asynchronously, the RAG-Ingestion-Worker Lambda spins up in the background. AWS allows this Lambda to run for up to 15 minutes. It clones the repository, runs the Tree-sitter AST parser, generates vectors using the Gemini API, calculates PageRank, and compiles the BM25 dictionary. Once finished, it uploads the massive  faiss.bin ,  bm25.pkl , and  graph.db  databases directly into an Amazon S3 Bucket. It then updates DynamoDB to say "Completed".

3. The Polling Mechanism: The Status Lambda

While the Worker is churning in the background, the frontend UI is showing a loading spinner. The frontend continuously sends  GET  requests to the  /status  endpoint. These requests hit the RAG-Status Lambda, which simply peeks at the DynamoDB table and reports back to the frontend whether the worker is still "processing" or has "completed".

4. The Conversation: The Chat Lambda

Once ingestion is complete, the user asks a question. The request goes to  /chat , hitting the RAG-Chat Lambda. This Lambda retrieves the databases from S3, fires up the LangGraph Agentic state machine, executes the Tri-Modal search, and streams the reasoning and final answer (using Claude 3.5 Haiku) back through the API Gateway to the user's screen.
____________________________________________________________________________

Core AWS Services Used

Before tracking a request, here are the core AWS services that make up the architecture. Notably, the system is split across 4 distinct Lambda functions to safely handle asynchronous workloads and bypass strict API timeouts.
  • Amazon Route 53 & CloudFront: The DNS and CDN edge layer. Route 53 manages the custom domain ( code-compass.app ) and SSL certificates, while CloudFront securely hosts and globally distributes the static frontend (HTML/CSS/JS) to users at edge locations.
  • AWS API Gateway: The backend front door. Routes HTTP requests from the frontend to the correct Lambda function.
  • Amazon DynamoDB: A lightning-fast NoSQL database used purely as a cache and state-tracker (e.g., storing whether a repo is  processing  or  completed ).
  • Amazon S3 (Simple Storage Service): The massive storage bucket where the generated  faiss.bin ,  bm25.pkl , and  graph.db  databases are saved for long-term retrieval.
  • AWS Lambda (The 4 Compute Engines):
  1. RAG-Dispatcher Lambda: A lightning-fast orchestrator. It instantly checks the DynamoDB cache, triggers the RAG-Ingestion-Worker Lambda, and returns a 202 Accepted response to the user in under 1 second.
  1. RAG-Ingestion-Worker Lambda: The heavy lifter. It runs completely in the background (allowed up to 15 minutes). It downloads the codebase, runs the Tree-sitter AST, calls the Gemini embedding API, calculates PageRank, compiles BM25, and uploads the final databases to S3.
  1. RAG-Status Lambda: A lightweight polling endpoint. The frontend continuously calls this Lambda, which reads the DynamoDB state and reports whether the background worker is finished.
  1. RAG-Chat Lambda: The conversational engine. It pulls the databases from S3, executes the Tri-Modal RAG search, runs the LangGraph state machine with Claude 3.5 Haiku, and streams the answer back to the user.


Security & IAM Roles (The Principle of Least Privilege)

In a distributed serverless architecture, security is paramount. Code Compass strictly adheres to the AWS best practice of Least Privilege using Identity and Access Management (IAM) Roles.
Because we decoupled the architecture into 4 distinct Lambda functions, we were able to assign hyper-specific permissions to each one rather than granting a single, dangerous "God Mode" role to the entire backend:
  • RAG-Dispatcher Role: This Lambda only has permissions to  dynamodb:PutItem ,  dynamodb:GetItem , and  lambda:InvokeFunction  (to trigger the Worker). It cannot touch the S3 bucket.
  • RAG-Ingestion-Worker Role: This heavy-lifter is the only function granted  s3:PutObject . It is allowed to write the massive database files into the S3 bucket, but it does not have permissions to delete them.
  • RAG-Status Role: This is a read-only endpoint. It is strictly limited to  dynamodb:GetItem .
  • RAG-Chat Role: The conversational AI endpoint only has  s3:GetObject  permissions. This guarantees that a malicious actor attempting a prompt injection attack on the LLM cannot possibly overwrite or delete the core databases, because the Lambda running the LLM physically lacks the AWS permissions to do so.


Lessons Learned: Critical Mistakes and Architectural Blunders

No system is designed perfectly on the first try. Here are the major architectural mistakes made during development and how we solved them.

Mistake 1: The 29-Second API Gateway Guillotine

The Flaw: Initially, my plan was to use a single Lambda function for  /ingest . Knowing that the vector generation process was slow, I assumed I could just configure the API Gateway route as "async". My goal was to have this single Lambda "reply early" to the frontend (to beat the Gateway's hard 29-second timeout limit), while it quietly finished cloning the repo and parsing the code in the background.
The Reality: This assumption was completely wrong on two fundamental levels:
  1. API Gateway HTTP APIs cannot do this: The modern, cheaper "HTTP API" version of API Gateway literally does not have an asynchronous invocation setting. That capability only exists on the older, more expensive "REST API" type.
  1. Lambdas cannot "reply early": Even if the Gateway allowed it, a single Lambda function fundamentally cannot reply to a user and then keep working. A Lambda function only returns an HTTP response at the very end of its execution, right before it shuts down. It is physically impossible for a single Lambda to send an HTTP response and then continue processing code in the background.
The Fix (The Asynchronous Rewrite): I had to completely tear down the architecture and rebuild it using an Asynchronous Dispatcher/Worker pattern. I split the single slow operation into two distinct Lambda functions:
  1. The Dispatcher (Lightweight): When  /ingest  is called, the Dispatcher wakes up. Instead of processing the code, it uses the AWS SDK ( boto3 ) to trigger the Worker Lambda asynchronously with the crucial flag  InvocationType='Event' . This magical flag tells AWS to fire the Worker in the background and immediately release the Dispatcher.
  1. The Worker: The Worker spins up independently with a 15-minute timeout allowance, doing the heavy lifting of indexing the repo, without the API Gateway watching it.
  1. The Handshake: Because the Dispatcher is instantly released, it replies to the frontend in under 1 second with an HTTP 202 (Accepted). The frontend then starts a polling loop ( setInterval ), asking a new  /status  endpoint every 3 seconds if the background worker has finished updating the DynamoDB table.

Mistake 2: Synchronous Chat Timeouts

The Flaw: Similar to the ingestion issue, I initially used slower LLMs for the LangGraph Agent. Because the agent dynamically plans, loops, and self-corrects, it often took 40+ seconds to finalize an answer. This hit the same 29-second API Gateway timeout. The Fix: I aggressively optimized the reasoning engine by switching the Agent to Claude 3.5 Haiku. Haiku is so blazingly fast that the entire state machine completes its complex loops in under 12 seconds, safely dodging the AWS timeout.

Mistake 3: The DynamoDB Caching Trap

The Flaw: To save compute costs, I implemented a DynamoDB cache. If a  repo_id  was marked "completed", I skipped ingestion. However, when users pushed new commits to their GitHub repos, CodeCompass refused to re-index the fresh code because it assumed the old cache was still valid.
The Fix: To solve this without ruining the UX with a manual "Force Re-Index" button, I implemented an Autonomous Smart Caching system. The Dispatcher Lambda now instantly makes a call to the GitHub API ( /commits/HEAD ) to fetch the repository's live commit SHA. It compares this against the exact SHA stored in DynamoDB during the last run. If they mismatch, it autonomously invalidates the cache and triggers a background re-index of the fresh code entirely behind the scenes.

Mistake 4: Deploying Without Measuring Package Size

The Flaw: Assumed zip deployment would just work without first checking the actual size of the dependency stack. The real footprint came in at ~1.4GB — nearly 6x over Lambda's 250MB uncompressed zip limit. The single biggest contributor was torch (~500MB), pulled in because sentence-transformers runs its embedding model locally, in-process. On top of that, genuinely dead weight had crept in: gradio (leftover from local prototyping, never needed in a deployed backend) and kubernetes (an unused transitive dependency of another package). Keeping local embeddings meant the only way forward was container-image deployment (10GB limit) — real added complexity (Docker, ECR) that wasn't actually necessary once the real problem (local model execution) was identified.
The Fix: Ran actual pip install + size checks before committing to any packaging approach. That audit made it clear torch was the dominant cost, which led to switching embedding generation to Gemini's API — eliminating torch/transformers/scipy/sklearn from the deployment package entirely and bringing the live query-path package down to a fraction of its original size, making standard zip deployment viable again.

Mistake 5: Naive LLM JSON Parsing

The Flaw: I originally asked the Agent to return JSON and used Python's strict  json.loads()  on the output. However, LLMs (even Claude) occasionally wrap their JSON in markdown code blocks (e.g.,  ```json { ... } ``` ) or append conversational text like "Here is your JSON:". This caused the entire pipeline to crash violently. The Fix: I had to build a bulletproof parsing engine that uses regular expressions to hunt for the outermost  {  and  }  brackets, mathematically extracting and repairing malformed JSON from the LLM's raw markdown output before it crashes the app.

Mistake 6: The  /tmp  Storage Overflow

The Flaw: By default, AWS Lambda only gives you 512MB of temporary disk space ( /tmp ). When the  RAG-Ingestion-Worker  clones a massive GitHub repository and generates large FAISS and SQLite databases on the disk before uploading them to S3, it can easily run out of space and crash with a  No space left on device  error.
The Fix: AWS recently allowed developers to manually increase the  /tmp  storage up to 10GB in the Lambda configuration settings, which gives the Worker enough breathing room to process massive enterprise repositories.

Mistake 7: Silent Memory Exhaustion (OOM Crashes)

The Flaw: By default, Lambdas are allocated just 128MB of RAM. Loading a large AST (Abstract Syntax Tree) into memory, alongside heavy libraries like  networkx  and parsing the FAISS database, caused the Worker Lambda to silently crash with an "Out of Memory" (OOM) error without any useful traceback logs.
The Fix: Bumping the Lambda memory up to 2048MB (2GB). Because AWS allocates CPU power proportionally to RAM, this not only prevented the crashes but made the indexing process significantly faster!


Is This Expensive? (Cost Analysis Breakdown)

One of the biggest questions developers have about serverless architectures is: "Is this going to cost me a fortune?"
The beautiful thing about a 100% serverless, event-driven architecture is that you scale to exactly zero. If nobody uses the app, you pay $0.00. There are no EC2 instances idling or RDS databases burning cash in the background.
Furthermore, Code Compass comfortably sits almost entirely within the AWS Free Tier:
  • AWS Lambda: The first 1 million requests and 400,000 GB-seconds of compute time per month are completely free. Even with the heavy Ingestion Worker, it takes a massive amount of traffic to break this tier.
  • Amazon API Gateway: The first 1 million HTTP API calls per month are free.
  • Amazon DynamoDB: You get 25 GB of storage and 25 Write/Read Capacity Units for free indefinitely. Since we only store a tiny  <repo_id, state>  hash, this costs exactly $0.
  • Amazon S3 & CloudFront: S3 offers 5GB of standard storage and 20,000 Get Requests for free. CloudFront offers a massive 1TB of free data transfer out per month.
  • Custom Domain: Even the premium  code-compass.app  custom domain was entirely free, secured through Name.com using the GitHub Student Developer Pack!
The only real external cost comes from the Claude 3.5 Haiku API and Gemini Embedding API. But because Haiku is highly optimized (fractions of a penny per token), and I implemented rotation of 3 free-tier Gemini API keys to bypass embedding rate limits without any cost, the total cost to host and run this entire enterprise-grade application for a portfolio or small team is effectively zero dollars.


Future Scalability: What Happens When Code Compass Goes Viral?

Right now, the architecture handles individual requests beautifully. But what happens if 10,000 developers all submit large GitHub repositories at the exact same time?
Currently, the Dispatcher Lambda would asynchronously trigger 10,000 parallel instances of the Worker Lambda. While AWS Lambda is massively scalable, an immediate spike of 10,000 concurrent, heavy Python functions (each lasting up to 15 minutes and requiring intense memory) could easily slam into our AWS Account concurrency limits or get throttled by the GitHub API.
The Solution: Amazon SQS (Simple Queue Service)
To scale this to an enterprise level, the next architectural step is introducing Amazon SQS as a buffer:
  1. Instead of the Dispatcher Lambda directly triggering the Worker Lambda, the Dispatcher would simply drop the  repo_id  into an SQS Queue.
  1. The SQS Queue would hold the requests securely.
  1. The Worker Lambda would be configured to pull messages from the SQS Queue at a controlled, steady pace (e.g., maximum 50 concurrent Workers at a time).
This completely eliminates the risk of hitting concurrency limits, ensures no repository ingestion requests are ever dropped during a traffic spike, and creates a highly resilient, buffered architecture capable of massive scale.
Live Demo:  https://code-compass.app/  
# aws-lambda# amazon-bedrock# amazon-dynamodb# serverless# rag
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article