AWS Builder Center
Building AeroCortex: Experience-Aware UAV Recovery on AWS Serverless

Building AeroCortex: Experience-Aware UAV Recovery on AWS Serverless

How a five-agent LangGraph cognitive stack detects in-flight failures, retrieves past missions, plans with Bedrock/Groq, gates every action through deterministic safety, and learns asynchronously on DynamoDB, S3, and SQS — with a live Mission Control dashboard on CloudFront.

Building AeroCortex: Experience-Aware UAV Recovery on AWS Serverless

Subtitle: How a five-agent LangGraph cognitive stack detects in-flight failures, retrieves past missions, plans with Bedrock/Groq, gates every action through deterministic safety, and learns asynchronously on DynamoDB, S3, and SQS — with a live Mission Control dashboard on CloudFront.
Author: Ashiha M S Stack: Python · LangGraph · FastAPI · AWS SAM · Lambda · API Gateway · DynamoDB · S3 · SQS · Bedrock · CloudFront · Pinecone · Groq



Live links

What
URL
Mission Control dashboard
 https://dks6e4j8skruf.cloudfront.net 
API root
 https://zrxrrszc26.execute-api.ap-south-1.amazonaws.com/ 
Health ( GET /healthz )
 https://zrxrrszc26.execute-api.ap-south-1.amazonaws.com/healthz 
OpenAPI / Swagger docs
 https://zrxrrszc26.execute-api.ap-south-1.amazonaws.com/docs 
GitHub repository
 https://github.com/VantyxLabs/AeroCortex 
AWS deploy runbook
 docs/AWS_DEPLOY.md 
AWS integration spec
 docs/AWS_INTEGRATION_SPEC.md 
Project README
 README.md 
Region:  ap-south-1  (Mumbai) · Stack name:  aerocortex 

Related product / docs links




1. The problem: alarms are not recovery

Autonomous UAVs fail in messy, real-world ways: GPS jamming or urban-canyon multipath, batteries that sag under load, wind shear, lost command links, sensor spikes, or several of those at once.
A typical flight stack response is:
  1. Raise an alarm, or
  1. Trigger a fixed behaviour — most often return-to-home (RTH).
RTH sounds safe until you notice the contradiction: if GPS is the thing that broke, “go home using GPS” is the worst plan you can invent.
What operators and researchers actually need is something closer to a human pilot’s loop:
  1. Notice that something is wrong (without hallucinating a failure).
  1. Remember what worked last time in a similar situation.
  1. Propose a recovery.
  1. Refuse any proposal that violates hard physics and mission limits.
  1. Remember the outcome so the next flight is smarter.
That is AeroCortex: an experience-aware cognitive memory and recovery architecture for autonomous edge UAVs — now also deployable as a full AWS serverless stack with a public Mission Control UI.



2. What AeroCortex does

AeroCortex ingests a telemetry frame (from a simulator, a MATLAB/Simulink twin, or eventually a flight controller), runs a LangGraph multi-agent pipeline, and returns a safety-validated recovery action. On AWS, missions and working state live in DynamoDB, knowledge snapshots in S3, learning is asynchronous over SQS FIFO, planning prefers Amazon Bedrock with Groq fallback, vectors sit in Pinecone, and operators use a CloudFront dashboard that proxies  /api  to API Gateway + Lambda.
Try it live:  Mission Control  → choose  GPS_INTERFERENCE  → Step Mission.



3. Who it is for

  • Teams building autonomous robots that need to make better decisions from past experience: drones, delivery and warehouse robots, agricultural machines, inspection robots, and other systems that fail in the field and shouldn't repeat the same mistake.
  • AeroCortex is an API and SDK that gives them a memory layer for failures and recoveries, plus a deterministic safety gate that checks every plan before it runs.
  • Developers integrate it over REST or the Python SDK and get recall of similar past incidents, a proposed recovery plan, and a safety-checked decision, without building the memory and planning stack themselves.



4. The cognitive loop: five agents

Everything on the dashboard is a window onto one LangGraph pipeline.

4.1 Situation Agent — detect without an LLM

Raw telemetry hits deterministic thresholds (GPS status/accuracy, battery voltage/SoC, wind, IMU spikes, comms). Detection is sub-millisecond and never delegated to a language model. A model should not be the thing that invents whether your aircraft has a problem.
Example envelope idea for GPS interference: status  degraded  or accuracy (HDOP-style metres) beyond ~3.0 m — live demos often show 6.4 m under  GPS_INTERFERENCE .

4.2 Memory Agent — hybrid retrieval

When an anomaly fires, memory retrieves:
  • Episodic memory — past missions as vectors (Pinecone on the AWS demo).
  • Semantic memory — IF–THEN operational rules (on the order of ~9 seeded rules, e.g. GPS → VIO).
  • Knowledge graph — failure types linked to conditions, actions, and outcomes (embedded graph engine on the AWS demo).
  • Working memory — volatile buffer for the active mission (DynamoDB-backed on AWS).
Ranking blends ~60% vector similarity + ~40% graph relevance. That fused score is what you see in the Hybrid Memory table (top row often near  1.000  for a strong match).

4.3 Planner Agent — propose, don’t fly

Ordered tiers on AWS ( PLANNER_TIERS ):
  1. Amazon Bedrock (Converse), when  BEDROCK_MODEL_ID  is set and accessible
  1. Groq
  1. Offline cognitive reasoner (deterministic synthesis from retrieved context)
Locally the order can be Groq → Ollama → offline. The planner emits structured JSON (action, reason, steps, confidence, risk). It proposes; it does not command the airframe.

4.4 Safety Agent — mandatory gate

Every plan passes hard constraints: battery floors, altitude/wind limits, GPS-related policy, geofence-style envelopes. Outcomes:
  • Approved → execute the planned action (e.g.  SWITCH_TO_VIO_DEAD_RECKONING )
  • Rejected → deterministic fallback (often  RETURN_TO_HOME  or other fail-safes)
This is the product’s ethical and engineering centre: LLMs are creative; envelopes are not optional.

4.5 Learning Agent — remember the flight

The episode (telemetry, situation, retrieval, plan, safety verdict, outcome) is stored and used to reinforce rules/graph/vectors.
On AWS this is async:
  1. API Lambda enqueues the episode on SQS FIFO.
  1. Response can return quickly with learning queued.
  1. Learning Lambda commits to DynamoDB / S3 / Pinecone.
  1. A DLQ + CloudWatch alarm catches poison messages.
That split matters for demo UX: operators should not wait on every embedding write before seeing a recovery action.



5. The eight failure scenarios

The built-in simulator injects one of eight modes into a physically plausible 6-DoF telemetry stream (nineteen fields per frame). The dashboard KPI strip shows six of them; charts and the API see the rest.
#
Scenario
What typically changes
1
 GPS_INTERFERENCE 
GPS  degraded , accuracy ~6.4 m (the live demo default)
2
 GPS_LOSS 
GPS  lost , accuracy pegged very high
3
 BATTERY_DEGRADATION 
Voltage collapses under load
4
 LOW_BATTERY 
SoC under the safety floor (~15%)
5
 COMMUNICATION_LOSS 
Comms status lost
6
 STRONG_WIND 
Wind ~16.8 m/s vs ~12 m/s tolerance
7
 SENSOR_ANOMALY 
IMU spikes
8
 COMBINED_FAILURE 
GPS interference + wind shear
Same pipeline for all eight — only the injected telemetry changes.



6. Reading the Mission Control dashboard

6.1 Health strip

Polled from   /healthz   through same-origin  /api .
On the AWS profile you should see tiles such as:
  • System — overall  ok  /  degraded  /  unavailable 
  • DynamoDB — mission store
  • Bedrock — planner tier configured / last-call health
  • S3 — snapshot store
  • SQS — episode queue
  • Pinecone / Groq — vector and LLM fallback health
 ok  means dependencies are healthy. Soft fallbacks elsewhere do not mean the cognitive loop is dead — Step OK on the banner is the proof.

6.2 Controls

  • Failure scenario — which fault to inject
  • API base URL — fixed  /api  (CloudFront forwards  /api/*  to API Gateway)
  • API key — sent as  X-API-Key  on protected routes
  • Step Mission — one full cognitive cycle
  • Reset — clear simulator + working memory (long-term stores remain)

6.3 KPI strip — what the numbers mean

KPI
Typical  GPS_INTERFERENCE 
Meaning
Mission
 SIM_MISSION_001 
Simulated mission id
State
 CRUISE 
Mission phase in the packet
Altitude
~120 m AGL
Height above ground; nav fault ≠ automatic emergency dump
Velocity
~10 m/s
Forward speed; healthy airframe
Battery
~94–95% · ~16.6 V
Power is fine — fault is navigation
Wind
~6 m/s
Moderate; under the hard wind limit
Story in one sentence: healthy cruise with a navigation problem.

6.4 Decision panels (pipeline order)

  1. Section 02 — Situation — e.g.  GPS_INTERFERENCE · HIGH , accuracy 6.4 m, confidence ~0.91 (rule engine).
  1. Section 03 — Hybrid Memory — ranked past episodes; score = hybrid retrieval.
  1. Section 04 — Knowledge Graph — relational path failure → condition → action → outcome.
  1. Section 05 — Planner — action  SWITCH_TO_VIO_DEAD_RECKONING , confidence ~0.95, risk  LOW ,  source   bedrock  or  groq .
  1. Section 06 — Safety —  Approved , fallback named, executed action, constraints list.

6.5 Footer, charts, logs

  • Episodes / Rules / KG nodes / edges — long-term memory footprint (episodes rise as learning commits).
  • Telemetry Stream — battery; altitude & velocity; wind & GPS HDOP (one sample per step).
  • Orchestration logs — Situation → Memory → Planner → Safety → Execution → Learning.
Golden rule for demos and blogs with screenshots: read the glass; never invent a number.



7. The REST API

Method
Path
Auth
Role
 GET 
 / 
none
Service identity +  mission_store  /  offline_mode 
 GET 
 /healthz 
none
Dependency health
 POST 
 /telemetry 
 X-API-Key 
Production entry: live or twin telemetry → full pipeline
 POST 
 /simulate 
 X-API-Key 
What Step Mission calls
 GET 
 /memory 
 X-API-Key 
Rules, graph summary, last retrieval
 GET 
 /missions 
 X-API-Key 
Persisted episodes (DynamoDB on AWS)
 GET 
 /status 
 X-API-Key 
Working memory snapshot
 POST 
 /reset 
 X-API-Key 
Clear sim + working memory
The dashboard is not a second brain. It is a REST client. Anything you click, you can call from  curl , Python, or MATLAB.
There is also a small Python SDK in the repo ( sdk/ ) wrapping the same routes.
Health example (shape evolves with deploy; check live JSON):
GET https://zrxrrszc26.execute-api.ap-south-1.amazonaws.com/healthz
Expect fields like  status ,  dependencies.dynamodb ,  dependencies.s3 ,  dependencies.sqs ,  dependencies.bedrock ,  dependencies.groq ,  dependencies.pinecone ,  mission_store: "dynamodb" .



8. How AWS is used

8.1 Build it (tooling / open stack)

  • AWS SAM + SAM CLI — infrastructure as code and container packaging ( infra/template.yaml )
  • AWS CLI — deploy, SSM, S3 sync, CloudFront invalidation
  • Docker — Lambda container images ( Dockerfile.lambda )
  • Boto3 — DynamoDB, S3, SQS, SSM, Bedrock Runtime

8.2 Ship it (services)

Service
Role in AeroCortex
Amazon API Gateway (HTTP API)
Public REST surface
AWS Lambda (API + Learning)
FastAPI via Mangum; async learner
Amazon DynamoDB
Missions/episodes + working/sim state
Amazon S3
KG/rules snapshots + dashboard static origin
Amazon SQS (FIFO + DLQ)
Async learning queue
Amazon Bedrock
Primary planner tier
Amazon CloudFront
Dashboard CDN +  /api/*  origin to API Gateway
AWS Systems Manager Parameter Store
SecureString secrets ( API_KEY , Groq, Pinecone)
AWS X-Ray / Amazon CloudWatch
Tracing, logs, metrics, DLQ/error alarms, ops dashboard
CloudFront strips the  /api  prefix with a CloudFront Function before forwarding to API Gateway, so the browser stays same-origin.
Opt-in design: unset AWS env vars and the repo still runs the local/dev path. Production profile sets things like  MISSION_STORE=dynamodb ,  KNOWLEDGE_SNAPSHOT_BACKEND=s3 ,  LEARNING_MODE=async ,  PLANNER_TIERS=bedrock,groq,offline .
Deploy guide:  AWS_DEPLOY.md .



9. A concrete story: GPS interference

This is the narrative used in the live demo video.
  1. Operator selects GPS_INTERFERENCE and clicks Step Mission.
  1. Telemetry shows cruise ~120 m AGL, ~10 m/s, healthy battery, wind ~6 m/s, GPS accuracy 6.4 m / degraded.
  1. Situation classifies GPS_INTERFERENCE · HIGH (~0.91 confidence) — deterministic.
  1. Memory surfaces prior VIO successes (hybrid scores).
  1. Planner proposes SWITCH_TO_VIO_DEAD_RECKONING (~0.95, risk LOW, source Bedrock or Groq).
  1. Safety Approves; executed action matches the plan.
  1. Learning queues/commits an episode so the second Step can reuse experience; charts gain a second sample; episode counters move.
VIO = visual-inertial odometry: navigate from camera + IMU when GPS cannot be trusted. That is a coherent answer to interference; blind RTH is not.



10. What went well on AWS

  • DynamoDB + SAM policies — pay-per-request mission store without babysitting a database in the Lambda image.
  • SQS FIFO + DLQ — clean separation between “return the recovery now” and “commit the episode soon,” with an ops alarm when learning fails.
  • S3 + CloudFront — fast path to a public Mission Control UI without CORS theatre.
  • SSM SecureString — secrets stay out of git and out of the SAM template body.
  • Bedrock Converse — same planner interface as Groq when model access is healthy; graceful fallthrough when it is not.
  • CloudWatch + X-Ray — enough signal for a hackathon ops story without a second observability product.



11. What was painful

  • Bedrock account/model access — onboarding and regional enablement can leave the primary tier dark while Groq still saves the demo.
  • Lambda container cold starts — LangGraph + scientific Python are heavy; warm up before recording.
  • SAM + browser login sessions — occasionally need explicit credential export before  sam deploy .
  • Stale CloudFront content — UI fixes do nothing until  s3 sync  + invalidation.
  • SSM SecureStrings — cannot be created by CloudFormation; one-time CLI bootstrap is easy to forget.
Specific friction is useful: it shows you actually shipped on these services, not just listed logos.



12. Solo build note

AeroCortex on this track is a solo project: product architecture, agents, memory, safety, simulation, FastAPI, dashboard, AWS SAM stack, deploy tooling, and demo narrative — end to end under VantyxLabs.



13. Try it yourself (5-minute path)

  1. Open the  dashboard  and hard-refresh.
  1. Confirm health tiles (DynamoDB / S3 / SQS / Pinecone / Groq; Bedrock if available).
  1. Scenario  GPS_INTERFERENCE  → Step Mission → read Situation, Planner, Safety, Memory.
  1. Step again — watch memory/episodes/charts compound.
  1. Skim  Swagger  and raw   /healthz  .
  1. Clone the repo and read  AWS_DEPLOY.md  if you want to stand up your own stack in  ap-south-1 .



14. Closing

AeroCortex is not “an LLM that flies drones.” It is a safety-gated, memory-backed recovery loop that happens to use LLMs for planning — and that ships on a concrete AWS serverless spine you can poke in the browser.
If you build on top of it — twin integration, tighter Bedrock routing, or provisioned concurrency for
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article