From Blackbox to Insight: A Serverless Flight-Log Analytics Architecture on AWS
Drone blackbox logs record everything about a flight, yet most sit unread on SD cards. This architecture deep-dive shows how to turn INAV and Betaflight logs into a serverless flight data lake on AWS: S3 uploads trigger a Step Functions workflow that decodes, converts to Parquet and computes per-flight metrics; Athena and dashboards query every flight; and Amazon Bedrock writes plain-language debriefs from precomputed numbers. Covers storage layout, security, cost and design trade-offs.
Why flight logs deserve a pipeline
Every flight a drone makes is recorded, and almost none of it gets analysed. Flight controllers running INAV or Betaflight write high-rate blackbox logs: gyro, PID terms, motor outputs, RC input, battery voltage and GPS, often hundreds of samples a second. Those logs sit on an SD card or onboard flash until someone opens one by hand in a desktop viewer after a crash.
That workflow breaks down quickly. You can inspect one flight, but you can't ask questions across fifty: Is motor 3 working harder than the others? Did the last PID change actually reduce oscillation? Which batteries sag hardest under the same throttle?
This article walks through a serverless architecture on AWS that turns raw blackbox files into a queryable flight data lake, with dashboards and plain-language AI debriefs per flight. It's designed for a solo builder or a small team, so it costs almost nothing when idle and needs no servers to babysit.
Architecture at a glance
The whole system is event-driven: uploading one log to S3 is the only manual step, and every later stage runs on its own.
[ARCHITECTURE DIAGRAM HERE]
Read it top to bottom. The top row gets logs into S3, the middle band is the processing workflow, and the bottom row is where you query, chart and read about your flights. Each layer is covered in detail below.
Ingestion: one bucket, one upload, one convention
Ingestion is deliberately boring: a raw log lands in an S3 bucket, and everything else is triggered by that event. After a flying session, logs come off the flight controller over USB, and a small upload script (or the ground-control app) pushes them to S3.
Two things make the rest of the pipeline easy:
- A predictable key layout. Raw files go to
raw/aircraft=<id>/date=<YYYY-MM-DD>/<log-file>. The aircraft ID and date are known before decoding, so they become partitions for free. - A sidecar metadata file. Next to each log, a small JSON file records what the log can't: battery ID, propeller type, payload, firmware version, PID profile name and pilot notes. This context is what makes cross-flight comparisons meaningful.
For uploads from a phone or the field, the app requests a short-lived presigned URL from an API Gateway endpoint backed by Lambda, with users authenticated through Amazon Cognito. The device never holds long-term AWS credentials.
The raw bucket has versioning on and a lifecycle rule that moves logs older than a few months to a colder storage class. Raw logs are the source of truth: if the decoder improves, you can replay everything.
Decode and transform: an event-driven workflow
A new object in
raw/ starts an AWS Step Functions workflow that decodes, enriches and stores the flight, with retries and error handling at every step. The bucket sends events to Amazon EventBridge, and a rule filtered on the raw/ prefix and the log file suffix starts the workflow. Sidecar JSON uploads don't match the rule, so they never start a run on their own.The workflow has four stages:
- Validate. A small Lambda checks the file header, loads the sidecar metadata and rejects anything that isn't a blackbox log.
- Decode. A Lambda packaged as a container image bundles the open-source blackbox decoder (INAV maintains its own fork of blackbox-tools). One file can hold several flight segments, so the decoder emits one CSV per segment.
- Transform. A Python Lambda using pyarrow converts each segment to Parquet, normalises units, and computes a per-flight summary: duration, min voltage under load, peak current, average and peak motor output per motor, and gyro noise per axis.
- Publish. Curated Parquet goes to the
curated/prefix, the summary row goes tosummaries/, and the debrief step (covered below) is invoked.
Three design choices matter here. The workflow uses the S3 object version ID as its idempotency key, so a duplicate event can't double-count a flight. Each stage writes its output to S3 before the next starts, so a failed run resumes from the last good step instead of re-decoding. And failed runs land in an SQS dead-letter queue with an SNS notification, so a corrupt log announces itself rather than silently disappearing.
Lambda's 15-minute timeout and 10 GB memory ceiling comfortably cover typical hobby-scale logs. If you log at very high rates for long endurance flights, Step Functions can hand the decode step to an AWS Fargate task instead, without changing anything else in the workflow.
Storage and catalog: a small, well-partitioned data lake
The data lake has three zones in one bucket, each with a clear owner and lifetime:
| Prefix | Format | Partitioned by | Holds |
|---|---|---|---|
raw/ | Original blackbox file + sidecar JSON | aircraft, date | The untouched source of truth, replayable at any time |
curated/ | Parquet, one file per flight segment | aircraft, date | Full-rate time series: gyro, PID terms, motors, RC, battery, GPS |
summaries/ | Parquet, one row per flight | aircraft, month | Per-flight metrics plus the sidecar context, for fast cross-flight queries |
Parquet is the right default here. Blackbox data is wide (dozens of columns) but most questions touch only a few, and a columnar format means a query about battery voltage never reads gyro data.
The AWS Glue Data Catalog holds the table definitions, and Athena partition projection computes partitions from the key layout. That removes the need for a Glue crawler or a
MSCK REPAIR step after every upload: a new flight is queryable the moment its Parquet file lands.Keeping
summaries/ separate is the most important decision in this layout. Most questions are about flights, not samples, and scanning one small row per flight is far cheaper and faster than aggregating millions of samples every time.Query and visualise: SQL over every flight you've flown
Amazon Athena turns the data lake into something you can interrogate with plain SQL, paying only for the data each query scans. The questions from the introduction become short queries against the summaries table. For example, to spot a motor that's working harder than the rest:
1
2
3
4
5
6
7
SELECT flight_date,
flight_id,
battery_id,
avg_motor_1, avg_motor_2, avg_motor_3, avg_motor_4,
greatest(avg_motor_1, avg_motor_2, avg_motor_3, avg_motor_4)
- least(avg_motor_1, avg_motor_2, avg_motor_3, avg_motor_4) AS motor_spread
FROMEnjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article