AIdeas: Beyond The Box
Two ordinary smartphones become a non-invasive material detector. See through sealed boxes using BLE, magnetometers, and machine learning.
AIdeas: Beyond The Box
Seeing Through Cardboard with Two Phones and Machine Learning
App Category: Commercial Solutions.
Tags: #aideas-2025 #commercial-solutions #daily-life-enhancement #logistics #EMEA
My Vision
Before the phone. Before the scanner. Before the app there was a sealed box.
And inside that box, anything. Everything. A lie. A danger. A counterfeit. A dream. Something precious. Something lethal. Something that was not what it claimed to be.
And you had no way to know.
You trusted the label. You trusted the sender. You trusted that the world was mostly honest and that what things claimed to be, they were. Sometimes you were right. Sometimes you weren't.
This is the oldest problem in human commerce.
Not fraud. Not logistics. Not supply chain inefficiency.
Blindness.
We have always been blind to the inside of things. To what hides behind surfaces. To what a container conceals. The merchant in the bazaar tapping the melon. The doctor pressing the abdomen. The customs officer shaking the box. All of them doing the same thing humans have done for ten thousand years.
Guessing.
With their hands. With their ears. With their intuition trained by experience.
We built civilisations on guessing.
Then came the instruments.
The microscope. The telescope. The X-ray.
Each one peeled back a layer of blindness. Each one said, here is a world you could not see before. Look.
But every one of them belonged to institutions. To the powerful. To those who could afford the machine, maintain the machine, interpret the machine.
The rest of us kept guessing.
Until now.
You already have it.
In your pocket. Warm from your body. Scratched from ordinary life.
A smartphone. Unremarkable. Ubiquitous. Billions of them carried by billions of people who have no idea what they are holding.
What they are holding is a magnetometer. An accelerometer. A Bluetooth radio. A processor more powerful than the computers that sent men to the moon.
What they are holding is every instrument ever built.
Miniaturised. Democratised. Already paid for.
Beyond The Box is not an app.
Do not make that mistake.
Beyond The Box is the moment we stopped being blind.
Point two smartphones at a sealed box. One speaks, broadcasting a Bluetooth signal through the material. One listens, measuring how that signal returns changed, bent, absorbed, disturbed by whatever is inside. Metal speaks differently than liquid. Liquid differently than air. Counterfeit differently than real.
The phones hear the difference.
And with every scan, from every user in every city, the system grows more fluent in electromagnetic signatures of materials. A hundred scans. A thousand. A million. A billion. Each one a word in a vocabulary that has never existed before.
Beyond The Box is not an app. It is a sixth sense.
This aims to become the largest signal database ever assembled. A new sense for human beings. A new way to interact with the physical world. Beyond The Box, as it currently is, works. And it is just the beginning. The next step, a million scans, a foundation model for materials, and a world where no package stays secret.
This is what I built.
In three weeks, with a free AWS tier, no special hardware, and the stubborn refusal to accept that hidden must stay hidden, I built a complete end-to-end material detection system that transforms ordinary smartphones into devices that can see through closed containers.
No X-rays. No opening. No specialised hardware.
Just two phones, a box, and a new way of listening to the world.
Why This Matters
Every day, millions of packages change hands in e-commerce, logistics, customs, and personal deliveries. Yet we have no way to verify what is inside without opening them. This leads to fraud and counterfeits, luxury goods and pharmaceuticals sold as genuine when they are not. Security risks, prohibited items passing through checkpoints unchallenged. Waste, packages opened, returned, destroyed simply to verify contents. Inefficiency, manual checks, expensive X-ray machines, or trusting the label.
Think about who needs this.
The delivery driver holding a package that doesn't feel right.
The customs officer with a thousand boxes and the instinct that one of them is wrong.
The woman in the market who has been sold fake gold before and will not be sold it again.
The pharmacist in a country where counterfeit medicine kills.
The parent who wants to know what their child ordered.
The archaeologist who cannot afford to damage what might be priceless.
The customs officer with a thousand boxes and the instinct that one of them is wrong.
The woman in the market who has been sold fake gold before and will not be sold it again.
The pharmacist in a country where counterfeit medicine kills.
The parent who wants to know what their child ordered.
The archaeologist who cannot afford to damage what might be priceless.
None of them have X-ray machines.
All of them have phones.
All of them have phones.
This is what democratisation actually means.
Not making something cheaper. Making something possible that was previously impossible and putting it in the hands of everyone. Democratising signal knowledge with everyday, inexpensive smartphones. New eyes for common humanity.
The X-ray belongs to the hospital. This sixth sense belongs to everyone.
But the implications go far beyond security and logistics.
Imagine holding your phone to a piece of jewellery and knowing instantly whether it is real gold or steel dressed in a lie.
Imagine scanning a shipment of medicine and catching counterfeit pills before they reach patients.
Imagine airport security lines where passengers simply walk past two poles with phones, and bags are screened without unpacking.
Imagine recycling facilities where workers point their phones at mixed waste and hear what each item is made of.
Imagine a future where your phone can tell you if the fruit is ripe, if the wall has a water leak, if the soil contains metal ore.
Imagine scanning a shipment of medicine and catching counterfeit pills before they reach patients.
Imagine airport security lines where passengers simply walk past two poles with phones, and bags are screened without unpacking.
Imagine recycling facilities where workers point their phones at mixed waste and hear what each item is made of.
Imagine a future where your phone can tell you if the fruit is ripe, if the wall has a water leak, if the soil contains metal ore.
The future applications grow with every scan. Fake jewellery detected by magnetic signature. Counterfeit pharmaceuticals flagged by signal attenuation. Weapons screened at events without X-ray infrastructure. Food contamination caught before it ships. Structural integrity checked without tearing open walls. The ceiling does not exist because matter is everywhere and signals pass through all of it.
This is not science fiction. This is physics. This is machine learning. This is what happens when you combine the sensors already in our pockets with the intelligence to interpret them.
This could not have existed before.
Not ten years ago. Not five.
The phones weren't powerful enough. The machine learning wasn't mature enough. The cloud wasn't cheap enough. The sensors weren't sensitive enough.
Every gear had to turn. Every piece had to fall into place.
The AI age. The smartphone age. The cloud age. The democratisation of compute.
All of it converging, now, to make possible what was impossible for the entirety of human history before this moment.
The window is open.
It has never been open before.
It has never been open before.
Google did not invent information. They made it findable.
The camera did not create moments. It made them keepable.
Beyond The Box did not create the physical world.
It made it readable.
The camera did not create moments. It made them keepable.
Beyond The Box did not create the physical world.
It made it readable.
How I Built This
Figure 1: The complete Serverless data pipeline. Raw magnetic and BLE physics are filtered locally on the edge device, enriched with NLP, and processed via an auto-scaling Amazon SageMaker XGBoost endpoint.
The Physics: Why It Works
Different materials interact with electromagnetic fields in unique, measurable ways.
Ferrous metals like iron and steel distort Earth's magnetic field significantly. The magnetometer feels them.
Non-ferrous metals like aluminium and copper barely disturb the magnetic field but attenuate Bluetooth signals. The RSSI drops.
Liquids, especially water, absorb radio frequencies strongly. They cast a radio shadow.
Air and empty space let signals pass almost unchanged.
Bluetooth Low Energy is the star signal. It passes through most packaging materials, cardboard, plastic, wood, but is altered by what lies inside. The magnetometer adds a second, independent dimension. The accelerometer validates that the scan was performed correctly. Together, they create a fingerprint that machine learning can recognise.
But physics still has the majority vote. The sensors speak first. Language helps, but the material reveals itself through its electromagnetic behaviour. We are not guessing. We are listening.
Real-world physics is the final boss. Every material is a puzzle of electromagnetic interactions. Our job is not to fight physics, but to listen to what it already says.
The System: Two Phones, One Intelligence
The architecture is simple in concept, elegant in execution.
Phone A, the Emitter, broadcasts a continuous BLE signal from behind the target container.
Phone B, the Receiver, scans for that signal while simultaneously recording magnetometer and accelerometer data.
The cloud receives the sensor stream, extracts meaningful features, and returns a prediction. What is inside the box.
But simplicity ends where reality begins.
Built in Partnership with Kiro
This project was built in partnership with Kiro, an AI coding assistant, over three intense weeks from concept to deployed system. Every architectural decision was made in service of one goal: clean data at scale, a model that grows smarter with every scan. Kiro helped build the Flutter application, design the AWS architecture, engineer the NLP pipeline, debug BLE conflicts, confidence overflows, sensor alignment issues, and UI scaling problems across device sizes. The system would not exist at this scale or quality in three weeks without it. This project is itself evidence of what human-AI collaboration produces when both sides are working at full capacity.
Mobile App: The Architecture of a Single Scan
The Home Screen. A clean UI minimizes cognitive load for security operators.
The app is not just an interface. It is a precision instrument designed to force clean data out of a messy human world. Every screen, every vibration, every colour carries intent.
Phase 1: Setup and Hardware Orchestration
1. Dual-Role Node Toggling
The feature is a seamless UI toggle that switches the app between acting as the BLE signal source, Emitter, or the sensor array, Receiver.
The why is that it eliminates custom hardware costs. It transforms two standard smartphones into a controlled IoT environment where one establishes a baseline and the other hunts for anomalies. No specialised equipment. Just two phones and a toggle.
2. Contextual How to Use Screen
The feature is a highly visual instruction module detailing the physical mechanics of a scan.
The why is that magnetic and BLE anomaly scanning is a novel interaction. No user comes to this app with muscle memory. This screen bridges the gap between complex physics and human operation, ensuring the operator does not capture garbage data through simple ignorance of how to hold the phones.
GUIDELINES FOR HOW TO USE THE APP EFFECTIVELY
3. Ergonomic Camera Viewfinder
The feature is a camera UI designed to force the user to hold the phone in a specific orientation.
The why is engineering. The human body absorbs BLE signals. By forcing a specific physical grip via the UI layout, the operator's hand is physically prevented from blocking the sensor path. The signal remains pristine. The data remains clean.
Phase 2: Pre-Scan and Smart Inputs
4. The Stealth NLP Box
The feature is a 150-character text input for messy human notes. It handles typos, ignores dashes, and runs n-gram and fuzzy matching against an extensive internal dictionary. It also detects contradictions, flagging when the user's description conflicts with itself or with what the sensors are likely to find.
The why is that language becomes data. The user knows something about what they are scanning, a label, a suspicion, a description. This small box captures that knowledge and translates human shorthand into strict binary encoding at the edge. It hands the SageMaker model a pre-solved logic puzzle, dropping cloud compute time and boosting accuracy without needing the cloud to parse text. The dictionary covers hundreds of terms across multiple categories, ferrous metals, non-ferrous metals, liquids, containers, empty indicators. It understands "soda can" differently than "metal scrap." It knows the difference. And when the user writes "empty water bottle," it understands both the container and the contradiction.
5. Auto-Normalisation Engine
The feature is that the UI accepts human-friendly measurements like centimetres and millilitres, but secretly converts them into strict base units, metres and litres, in the background.
The why is that machine learning models break when units change. A model trained on metres cannot understand centimetres. A model trained on litres cannot understand centilitres. This provides a forgiving UX for the human while maintaining a mathematically rigid dataset for the XGBoost model. The user never knows the conversion happened. They only know the app understood them.
6. Colour-Coded Scanning Modes and Smart Recommendations
The feature is distinct UI colour themes for different scan types, Liquid mode in blue, Swipe mode in green, Stationary mode in amber, and dynamic physical technique recommendations based on target size and distance.
The why is that it prevents catastrophic operator error. If the user is scanning for liquids but the screen is in the Cardboard Swipe colour theme, they instantly know they are in the wrong mode. The colour screams the mismatch. It ensures the physical sweep matches the physics expected by the backend. A liquid scan requires vertical motion. A stationary scan requires stillness. The colours encode the correct behaviour.
Phase 3: The Active Scan
7. The Half-Second Calibration Pause
The feature is a mandatory half-second initialisation sequence before the scan actively records.
The why is physics. Magnetometers and accelerometers need a moment to stabilise. The physical act of pressing "Start" creates a spike, a jolt, an artifact. This pause eliminates that artifact, establishing a perfect ambient baseline before data collection begins. The user feels the haptic pulse that says "ready," and only then does the scan truly start.
8. Real-Time Telemetry and On-Screen Timer
The feature is live BLE RSSI, magnetometer X, Y, Z readings, and a visual countdown displayed dynamically during the scan.
The why is trust and pacing. Operators need to know the hardware is actually reading data. The numbers moving on screen eliminate the black box effect. The timer paces the user, ensuring the scan duration variable fed to the math engine stays consistent across thousands of scans.
9. Tactile Haptic Feedback
The feature is precise physical vibrations triggered at key events, scan start, scan stop, warnings, success.
The why is ergonomics. The operator is looking at the physical box they are scanning, not staring at their screen. Haptics provide non-visual cues so they can keep their eyes on the target. A buzz means start moving. A double buzz means stop. A long buzz means warning. The phone speaks to the hand.
10. Kinematic Velocity Smoothening and Strict Thresholds
The feature is a live algorithmic filter running on the accelerometer, paired with real-time speed monitoring that aborts the scan if the user moves too fast or too slow.
The why is that human hands shake. They create false magnetic peaks, false accelerometer readings. Smoothening acts as a software shock-absorber. And by ruthlessly aborting bad kinematic scans at the edge, zero garbage data ever reaches the cloud architecture. The scan either meets the strict velocity thresholds, or it does not exist.
11. Colour-Coded Warnings and Transient Snackbars
The feature is visual alerts, yellow for "Slow Down," red for "Aborted," and non-intrusive bottom-screen snackbars for background states like "BLE Connected" or "Uploading."
The why is that colour communicates severity instantly. Yellow means correct. Red means stop. Snackbars keep the user informed of background cloud processes without interrupting their workflow with annoying pop-ups. The information is there when needed, invisible when not.
Phase 4: Post-Scan and Human-in-the-Loop
12. Raw CSV Scan Inspector Screen
The feature is a developer and power-user screen displaying the fully assembled feature matrix before transmission.
The why is ultimate transparency. It proves the physics engine is actually doing the heavy statistical math locally on the edge device before the cloud takes over. Anyone can verify that the data leaving the phone is the data that should leave the phone.
Ultimate transparency. The Raw CSV Inspector proves the edge device is actively processing the 35-feature statistical math locally before the cloud ever gets involved.
13. Saved Scans and The Quarantine Hub
The feature is a management hub where operators can view, rename, delete, and rate the accuracy of past scans based on physical verification, opening the box and seeing what was actually inside.
The why is continuous learning. This is the human-in-the-loop architecture. If the AI is wrong, the user updates the rating. The app routes this payload via MQTT directly into an AWS S3 quarantine folder, actively capturing edge cases to retrain and evolve the SageMaker model. Every wrong prediction becomes fuel for a right one later.
NLP: When Language Meets Physics
Material identification cannot rely on sensors alone. The user often knows something about what they are scanning, a label, a suspicion, a description. That knowledge is gold.
The NLP engine processes the user's free-text notes, extracting semantic hints about the contents. It uses an extensive material dictionary covering hundreds of terms across multiple languages and colloquialisms. Priority-based matching ensures specific phrases like "soda can" override general ones like "metal" to prevent conflicts. N-gram generation breaks text into meaningful chunks to catch multi-word descriptions. Fuzzy matching tolerates typos and misspellings, because humans are human. Contradiction detection flags when the text describes something impossible, like "empty full bottle," preventing the model from learning from nonsense.
The output is a set of flags, is_ferrous, is_liquid, is_air, that become additional features for the machine learning model. Language becomes data. But physics still has the majority vote. The sensors speak first. Language simply helps interpret what they say.
Feature Engineering and Sensor Fusion
Every scan is transformed into a rich feature matrix locally on the device. Magnetometer statistics across all three axes, mean, standard deviation, range, energy, disturbance ratios. RSSI signal behaviour, averages, gradients, minimums, maximums. Accelerometer motion data, velocity, smoothness, validity flags. Scan duration, baseline deltas, sample rates.
Three ordinary sensors fused into one picture reveal something none of them could see alone. The magnetometer alone detects ferrous metals. Add BLE RSSI and liquids become visible, water absorbs radio waves in ways metal does not. Add the accelerometer and scan validity becomes measurable. Combination is intelligence. This is the core insight the entire system is built on.
Cloud Architecture: Dual Channels, One Purpose
The backend is designed for scale from day one. It separates two critical functions.
Channel 1: Real-time Inference (HTTP)
Flutter App sends sensor stream and metadata to AWS Lambda. Lambda extracts physics and NLP features, invokes a SageMaker endpoint, and returns a prediction with confidence. The user sees the result instantly, typically in under 200 milliseconds.
Channel 2: Ground Truth Collection (MQTT over TLS)
After the user verifies the actual contents, the app publishes the verified data via MQTT to AWS IoT Core. An IoT rule routes it to an S3 bucket, organised by material class. This quarantined data awaits human review. Once verified, it joins the training set for the next model iteration.
Why two channels? Inference must be fast, HTTP works. Ground truth must be reliable even in poor networks, MQTT with persistent sessions ensures delivery. Separation also means inference scales independently of data collection. The system can handle millions of inference requests while steadily building a world-class training dataset.
AWS Services Used:
- Lambda for zero-dependency inference code, feature extraction, and SageMaker invocation
- SageMaker Serverless for auto-scaling XGBoost endpoint, zero idle costs
- IoT Core for MQTT ingestion with X.509 authentication
- S3 for scalable, durable storage with quarantine, training, and rejected folders
- IoT Rules for automated routing to S3
Machine Learning: Teaching Phones to Listen
The model learns to map sensor patterns to material classes. With verified scans from real users, it achieves high accuracy across four primary classes: Air, Liquid, Ferrous Metal, Non-Ferrous Metal.
Features include physics features like magnetic field statistics, mean, standard deviation, range, energy, axes ratios, RSSI statistics, mean, standard deviation, minimum, maximum, gradient, velocity, scan duration, sample rate, and NLP features, boolean flags derived from user text.
Early testing across hundreds of scans suggests strong accuracy on ferrous metals and liquids. These numbers improve as the dataset grows, the system learns from every verified scan.
The training pipeline extracts verified scans from S3 where ground truth rating is high. Features are engineered identically to the Lambda function. An XGBoost model is trained with cross-validation. The model is deployed to a SageMaker serverless endpoint. Lambda is updated to use the new endpoint.
The loop is continuous. More scans lead to better model, which leads to more accurate predictions, which leads to more trusted scans.
Development Milestones: Three Weeks
Week One: The Foundation
Flutter app built end-to-end. Dual-role BLE toggling. Three scan modes. Sensor fusion. CSV export. The first time two phones talked through a box, we felt the shiver of possibility. The app could see, barely, but it could see. The core loop worked. Data flowed. Something was there.
Week Two: The Interface and Cloud
Defensive UX, calibration pause, velocity thresholds, haptic feedback, colour-coded warnings. The app became usable by someone other than its creator. AWS pipeline went live, Lambda inference engine, IoT Core MQTT, S3 quarantine architecture. NLP engine version one with material dictionary. The app stopped being code and started being an instrument.
Week Three: The Intelligence
NLP hardened with expanded dictionary, fuzzy matching, n-gram analysis, priority phrase system, contradiction detection. ML model trained and deployed, feature engineering pipeline, XGBoost training, SageMaker serverless endpoint. End-to-end testing. The moment of truth arrived. It worked. Consistently. Reliably. The system started guessing and started being right. Documentation, polish, branding. The project became a story.
Three weeks. From concept to working prototype. From blindness to sight.
Demo: Making the Box Speak
For thousands of years, a sealed box meant uncertainty. We built a system to detect hidden materials (like metal, liquids, or electronics) using nothing but the Bluetooth and magnetic sensors already in our pockets.
(Watch the live demonstration)
What I Learned
1. Real-world physics is the final boss.
Everything looks clean in code. Timestamps align perfectly. Sensors behave. The model trains beautifully. Then you hold two actual phones against an actual cardboard box and reality arrives. Human hands shake. Magnetometers drift. Bluetooth reflects off metal furniture across the room. The physical world does not care about your assumptions. Building defensive UX, calibration phases, velocity thresholds, haptic pacing, abort logic, was not polish. It was the foundation. Without it, the data is garbage and the model learns nothing.
2. Data quality starts with the user experience.
You cannot fix noisy data later. Forcing a calibration pause, aborting scans above a velocity threshold, asking users to rate their own confidence, these are not UX decisions. They are ML decisions wearing UX clothing. Every interaction is a data collection moment. Design it that way from the first screen.
3. Sensor fusion is the intelligence.
The magnetometer alone detects ferrous metals. Add BLE RSSI and liquids become visible, water absorbs radio waves in ways metal does not. Add the accelerometer and scan validity becomes measurable. Three ordinary sensors fused into one picture reveal something none of them could see alone. Combination is intelligence. This is the core insight the entire system is built on.
4. Scale changes the nature of the thing.
At a few hundred scans, Beyond The Box is a demo. At thousands, it is a dataset. At millions, it is a foundation model for the physical world. At billions, the largest signal database ever assembled, it becomes something that has never existed: a machine that genuinely learns the electromagnetic fingerprint of materials. I realised early that what I was building was not an app but an infrastructure. The value is not in the scan. The value is in the accumulation.
5. Constraints breed invention.
Three weeks to the first working version. A free AWS tier. No specialised hardware. No lab. The only option was to look at what already existed, two billion smartphones, each carrying sensors the world had never thought to combine this way, and ask a different question. Not: what equipment do I need? But: what is already in everyone's pocket, waiting to be listened to?
6. Democratisation is not just a word.
An X-ray machine costs as much as a house. A smartphone costs as much as a week's groceries. By putting material intelligence into devices people already carry, we are not just building a product. We are redistributing a capability that has been locked inside institutions for a century. The woman in the market, the pharmacist in a remote village, the customs officer at a border crossing, they all gain a new sense. That is the kind of change that matters.
Beyond the technical lessons, I also see a clear path for this to become a sustainable platform. The technology opens multiple paths: enterprise API access for logistics companies needing real-time package verification, a freemium consumer app with premium features for power users, or per-scan micropayments for occasional use. The infrastructure scales to support all three simultaneously.
Beyond the technical lessons, I also see a clear path for this to become a sustainable platform. The technology opens multiple paths: enterprise API access for logistics companies needing real-time package verification, a freemium consumer app with premium features for power users, or per-scan micropayments for occasional use. The infrastructure scales to support all three simultaneously.
Built for the AWS Hackathon 2025, EMEA region, with gratitude for the chance to build something that matters.
Kiro, the AI that helped code, debug, and dream, deserves a share of the credit. The rest is human stubbornness, human curiosity, and the refusal to accept that hidden must stay hidden.
Humanity spent thousands of years guessing what was in the box.
Institutions built machines that could see through it.
Now ordinary people might be able to listen to it.
Institutions built machines that could see through it.
Now ordinary people might be able to listen to it.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article