
Building AeroCortex: Experience-Aware UAV Recovery on AWS Serverless
How a five-agent LangGraph cognitive stack detects in-flight failures, retrieves past missions, plans with Bedrock/Groq, gates every action through deterministic safety, and learns asynchronously on DynamoDB, S3, and SQS — with a live Mission Control dashboard on CloudFront.
Building AeroCortex: Experience-Aware UAV Recovery on AWS Serverless
Subtitle: How a five-agent LangGraph cognitive stack detects in-flight failures, retrieves past missions, plans with Bedrock/Groq, gates every action through deterministic safety, and learns asynchronously on DynamoDB, S3, and SQS — with a live Mission Control dashboard on CloudFront.
Author: Ashiha M S
Stack: Python · LangGraph · FastAPI · AWS SAM · Lambda · API Gateway · DynamoDB · S3 · SQS · Bedrock · CloudFront · Pinecone · Groq
Live links
What | URL |
|---|---|
Mission Control dashboard | https://dks6e4j8skruf.cloudfront.net |
API root | https://zrxrrszc26.execute-api.ap-south-1.amazonaws.com/ |
Health ( GET /healthz ) | https://zrxrrszc26.execute-api.ap-south-1.amazonaws.com/healthz |
OpenAPI / Swagger docs | https://zrxrrszc26.execute-api.ap-south-1.amazonaws.com/docs |
GitHub repository | https://github.com/VantyxLabs/AeroCortex |
AWS deploy runbook | docs/AWS_DEPLOY.md |
AWS integration spec | docs/AWS_INTEGRATION_SPEC.md |
Project README | README.md |
Region:
ap-south-1 (Mumbai) · Stack name: aerocortex Related product / docs links
1. The problem: alarms are not recovery
Autonomous UAVs fail in messy, real-world ways: GPS jamming or urban-canyon multipath, batteries that sag under load, wind shear, lost command links, sensor spikes, or several of those at once.
A typical flight stack response is:
- Raise an alarm, or
- Trigger a fixed behaviour — most often return-to-home (RTH).
RTH sounds safe until you notice the contradiction: if GPS is the thing that broke, “go home using GPS” is the worst plan you can invent.
What operators and researchers actually need is something closer to a human pilot’s loop:
- Notice that something is wrong (without hallucinating a failure).
- Remember what worked last time in a similar situation.
- Propose a recovery.
- Refuse any proposal that violates hard physics and mission limits.
- Remember the outcome so the next flight is smarter.
That is AeroCortex: an experience-aware cognitive memory and recovery architecture for autonomous edge UAVs — now also deployable as a full AWS serverless stack with a public Mission Control UI.
2. What AeroCortex does
AeroCortex ingests a telemetry frame (from a simulator, a MATLAB/Simulink twin, or eventually a flight controller), runs a LangGraph multi-agent pipeline, and returns a safety-validated recovery action. On AWS, missions and working state live in DynamoDB, knowledge snapshots in S3, learning is asynchronous over SQS FIFO, planning prefers Amazon Bedrock with Groq fallback, vectors sit in Pinecone, and operators use a CloudFront dashboard that proxies
/api to API Gateway + Lambda.
3. Who it is for
- Teams building autonomous robots that need to make better decisions from past experience: drones, delivery and warehouse robots, agricultural machines, inspection robots, and other systems that fail in the field and shouldn't repeat the same mistake.
- AeroCortex is an API and SDK that gives them a memory layer for failures and recoveries, plus a deterministic safety gate that checks every plan before it runs.
- Developers integrate it over REST or the Python SDK and get recall of similar past incidents, a proposed recovery plan, and a safety-checked decision, without building the memory and planning stack themselves.
4. The cognitive loop: five agents
Everything on the dashboard is a window onto one LangGraph pipeline.
4.1 Situation Agent — detect without an LLM
Raw telemetry hits deterministic thresholds (GPS status/accuracy, battery voltage/SoC, wind, IMU spikes, comms). Detection is sub-millisecond and never delegated to a language model. A model should not be the thing that invents whether your aircraft has a problem.
Example envelope idea for GPS interference: status
degraded or accuracy (HDOP-style metres) beyond ~3.0 m — live demos often show 6.4 m under GPS_INTERFERENCE .4.2 Memory Agent — hybrid retrieval
When an anomaly fires, memory retrieves:
- Episodic memory — past missions as vectors (Pinecone on the AWS demo).
- Semantic memory — IF–THEN operational rules (on the order of ~9 seeded rules, e.g. GPS → VIO).
- Knowledge graph — failure types linked to conditions, actions, and outcomes (embedded graph engine on the AWS demo).
- Working memory — volatile buffer for the active mission (DynamoDB-backed on AWS).
Ranking blends ~60% vector similarity + ~40% graph relevance. That fused score is what you see in the Hybrid Memory table (top row often near
1.000 for a strong match).4.3 Planner Agent — propose, don’t fly
Ordered tiers on AWS (
PLANNER_TIERS ):- Amazon Bedrock (Converse), when
BEDROCK_MODEL_IDis set and accessible
- Groq
- Offline cognitive reasoner (deterministic synthesis from retrieved context)
Locally the order can be Groq → Ollama → offline. The planner emits structured JSON (action, reason, steps, confidence, risk). It proposes; it does not command the airframe.
4.4 Safety Agent — mandatory gate
Every plan passes hard constraints: battery floors, altitude/wind limits, GPS-related policy, geofence-style envelopes. Outcomes:
- Approved → execute the planned action (e.g.
SWITCH_TO_VIO_DEAD_RECKONING)
- Rejected → deterministic fallback (often
RETURN_TO_HOMEor other fail-safes)
This is the product’s ethical and engineering centre: LLMs are creative; envelopes are not optional.
4.5 Learning Agent — remember the flight
The episode (telemetry, situation, retrieval, plan, safety verdict, outcome) is stored and used to reinforce rules/graph/vectors.
On AWS this is async:
- API Lambda enqueues the episode on SQS FIFO.
- Response can return quickly with learning queued.
- Learning Lambda commits to DynamoDB / S3 / Pinecone.
- A DLQ + CloudWatch alarm catches poison messages.
That split matters for demo UX: operators should not wait on every embedding write before seeing a recovery action.
5. The eight failure scenarios
The built-in simulator injects one of eight modes into a physically plausible 6-DoF telemetry stream (nineteen fields per frame). The dashboard KPI strip shows six of them; charts and the API see the rest.
# | Scenario | What typically changes |
|---|---|---|
1 | GPS_INTERFERENCE | GPS degraded , accuracy ~6.4 m (the live demo default) |
2 | GPS_LOSS | GPS lost , accuracy pegged very high |
3 | BATTERY_DEGRADATION | Voltage collapses under load |
4 | LOW_BATTERY | SoC under the safety floor (~15%) |
5 | COMMUNICATION_LOSS | Comms status lost |
6 | STRONG_WIND | Wind ~16.8 m/s vs ~12 m/s tolerance |
7 | SENSOR_ANOMALY | IMU spikes |
8 | COMBINED_FAILURE | GPS interference + wind shear |
Same pipeline for all eight — only the injected telemetry changes.
6. Reading the Mission Control dashboard
6.1 Health strip
On the AWS profile you should see tiles such as:
- System — overall
ok/degraded/unavailable
- DynamoDB — mission store
- Bedrock — planner tier configured / last-call health
- S3 — snapshot store
- SQS — episode queue
- Pinecone / Groq — vector and LLM fallback health
ok means dependencies are healthy. Soft fallbacks elsewhere do not mean the cognitive loop is dead — Step OK on the banner is the proof.6.2 Controls
- Failure scenario — which fault to inject
- API base URL — fixed
/api(CloudFront forwards/api/*to API Gateway)
- API key — sent as
X-API-Keyon protected routes
- Step Mission — one full cognitive cycle
- Reset — clear simulator + working memory (long-term stores remain)
6.3 KPI strip — what the numbers mean
KPI | Typical GPS_INTERFERENCE | Meaning |
|---|---|---|
Mission | SIM_MISSION_001 | Simulated mission id |
State | CRUISE | Mission phase in the packet |
Altitude | ~120 m AGL | Height above ground; nav fault ≠ automatic emergency dump |
Velocity | ~10 m/s | Forward speed; healthy airframe |
Battery | ~94–95% · ~16.6 V | Power is fine — fault is navigation |
Wind | ~6 m/s | Moderate; under the hard wind limit |
Story in one sentence: healthy cruise with a navigation problem.
6.4 Decision panels (pipeline order)
- Section 02 — Situation — e.g.
GPS_INTERFERENCE · HIGH, accuracy 6.4 m, confidence ~0.91 (rule engine).
- Section 03 — Hybrid Memory — ranked past episodes; score = hybrid retrieval.
- Section 04 — Knowledge Graph — relational path failure → condition → action → outcome.
- Section 05 — Planner — action
SWITCH_TO_VIO_DEAD_RECKONING, confidence ~0.95, riskLOW,sourcebedrockorgroq.
- Section 06 — Safety —
Approved, fallback named, executed action, constraints list.
6.5 Footer, charts, logs
- Episodes / Rules / KG nodes / edges — long-term memory footprint (episodes rise as learning commits).
- Telemetry Stream — battery; altitude & velocity; wind & GPS HDOP (one sample per step).
- Orchestration logs — Situation → Memory → Planner → Safety → Execution → Learning.
Golden rule for demos and blogs with screenshots: read the glass; never invent a number.
7. The REST API
Method | Path | Auth | Role |
|---|---|---|---|
GET | / | none | Service identity + mission_store / offline_mode |
GET | /healthz | none | Dependency health |
POST | /telemetry | X-API-Key | Production entry: live or twin telemetry → full pipeline |
POST | /simulate | X-API-Key | What Step Mission calls |
GET | /memory | X-API-Key | Rules, graph summary, last retrieval |
GET | /missions | X-API-Key | Persisted episodes (DynamoDB on AWS) |
GET | /status | X-API-Key | Working memory snapshot |
POST | /reset | X-API-Key | Clear sim + working memory |
The dashboard is not a second brain. It is a REST client. Anything you click, you can call from
curl , Python, or MATLAB.There is also a small Python SDK in the repo (
sdk/ ) wrapping the same routes.Health example (shape evolves with deploy; check live JSON):
GET https://zrxrrszc26.execute-api.ap-south-1.amazonaws.com/healthzExpect fields like
status , dependencies.dynamodb , dependencies.s3 , dependencies.sqs , dependencies.bedrock , dependencies.groq , dependencies.pinecone , mission_store: "dynamodb" .
8. How AWS is used
8.1 Build it (tooling / open stack)
- AWS SAM + SAM CLI — infrastructure as code and container packaging (
infra/template.yaml)
- AWS CLI — deploy, SSM, S3 sync, CloudFront invalidation
- Docker — Lambda container images (
Dockerfile.lambda)
- Boto3 — DynamoDB, S3, SQS, SSM, Bedrock Runtime
8.2 Ship it (services)
Service | Role in AeroCortex |
|---|---|
Amazon API Gateway (HTTP API) | Public REST surface |
AWS Lambda (API + Learning) | FastAPI via Mangum; async learner |
Amazon DynamoDB | Missions/episodes + working/sim state |
Amazon S3 | KG/rules snapshots + dashboard static origin |
Amazon SQS (FIFO + DLQ) | Async learning queue |
Amazon Bedrock | Primary planner tier |
Amazon CloudFront | Dashboard CDN + /api/* origin to API Gateway |
AWS Systems Manager Parameter Store | SecureString secrets ( API_KEY , Groq, Pinecone) |
AWS X-Ray / Amazon CloudWatch | Tracing, logs, metrics, DLQ/error alarms, ops dashboard |
CloudFront strips the
/api prefix with a CloudFront Function before forwarding to API Gateway, so the browser stays same-origin.Opt-in design: unset AWS env vars and the repo still runs the local/dev path. Production profile sets things like
MISSION_STORE=dynamodb , KNOWLEDGE_SNAPSHOT_BACKEND=s3 , LEARNING_MODE=async , PLANNER_TIERS=bedrock,groq,offline .
9. A concrete story: GPS interference
This is the narrative used in the live demo video.
- Operator selects GPS_INTERFERENCE and clicks Step Mission.
- Telemetry shows cruise ~120 m AGL, ~10 m/s, healthy battery, wind ~6 m/s, GPS accuracy 6.4 m / degraded.
- Situation classifies GPS_INTERFERENCE · HIGH (~0.91 confidence) — deterministic.
- Memory surfaces prior VIO successes (hybrid scores).
- Planner proposes SWITCH_TO_VIO_DEAD_RECKONING (~0.95, risk LOW, source Bedrock or Groq).
- Safety Approves; executed action matches the plan.
- Learning queues/commits an episode so the second Step can reuse experience; charts gain a second sample; episode counters move.
VIO = visual-inertial odometry: navigate from camera + IMU when GPS cannot be trusted. That is a coherent answer to interference; blind RTH is not.
10. What went well on AWS
- DynamoDB + SAM policies — pay-per-request mission store without babysitting a database in the Lambda image.
- SQS FIFO + DLQ — clean separation between “return the recovery now” and “commit the episode soon,” with an ops alarm when learning fails.
- S3 + CloudFront — fast path to a public Mission Control UI without CORS theatre.
- SSM SecureString — secrets stay out of git and out of the SAM template body.
- Bedrock Converse — same planner interface as Groq when model access is healthy; graceful fallthrough when it is not.
- CloudWatch + X-Ray — enough signal for a hackathon ops story without a second observability product.
11. What was painful
- Bedrock account/model access — onboarding and regional enablement can leave the primary tier dark while Groq still saves the demo.
- Lambda container cold starts — LangGraph + scientific Python are heavy; warm up before recording.
- SAM + browser login sessions — occasionally need explicit credential export before
sam deploy.
- Stale CloudFront content — UI fixes do nothing until
s3 sync+ invalidation.
- SSM SecureStrings — cannot be created by CloudFormation; one-time CLI bootstrap is easy to forget.
Specific friction is useful: it shows you actually shipped on these services, not just listed logos.
12. Solo build note
AeroCortex on this track is a solo project: product architecture, agents, memory, safety, simulation, FastAPI, dashboard, AWS SAM stack, deploy tooling, and demo narrative — end to end under VantyxLabs.
13. Try it yourself (5-minute path)
- Open the dashboard and hard-refresh.
- Confirm health tiles (DynamoDB / S3 / SQS / Pinecone / Groq; Bedrock if available).
- Scenario
GPS_INTERFERENCE→ Step Mission → read Situation, Planner, Safety, Memory.
- Step again — watch memory/episodes/charts compound.
- Clone the repo and read AWS_DEPLOY.md if you want to stand up your own stack in
ap-south-1.
14. Closing
AeroCortex is not “an LLM that flies drones.” It is a safety-gated, memory-backed recovery loop that happens to use LLMs for planning — and that ships on a concrete AWS serverless spine you can poke in the browser.
- Dashboard: https://dks6e4j8skruf.cloudfront.net
If you build on top of it — twin integration, tighter Bedrock routing, or provisioned concurrency for
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article