AIdeas Finalist: Beyond The Box
A multi-modal sensing system that leverages smartphone magnetometers and BLE signals to classify materials inside closed containers, combining physical signal analysis with a cloud-based learning pipeline.
App Category: Commercial Solutions
Tags: #aideas-2025 #commercial-solutions #EMEA
My Vision
A sealed container represents one of humanity's oldest blind spots. For millennia, we have tapped, shaken, and guessed. Beyond The Box replaces uncertainty with measurable physics. Using only two smartphones and their native magnetometer, Bluetooth Low Energy radio, and accelerometer, the system classifies the material inside a closed container without opening it. No X-ray. No custom hardware. No lab.
The system rests on a fundamental physical premise. Different materials alter electromagnetic fields in distinct, quantifiable ways. Water molecules absorb energy specifically at 2.45 GHz, the exact frequency band in which Bluetooth Low Energy operates. Ferrous metals do something more dramatic: the iron atoms inside them are permanently magnetised at the microscopic level, and when an external field is present, those magnetic domains align and amplify, inducing a localised field distortion that the magnetometer reads as a clear anomaly against the Earth's ambient background. Non-ferrous conductors such as aluminium reflect BLE modestly without disturbing the magnetometer. Air introduces no measurable attenuation or magnetic anomaly. These are not learned correlations. They are invariant physical laws, and the system is engineered to listen to what those laws already declare.
Why This Matters
The inability to verify container contents without breaching them imposes real costs across global supply chains. Counterfeit pharmaceuticals, misdeclared hazardous materials, and fraudulent luxury goods all exploit this opacity.
Industrial solutions exist. Computed tomography scanners, handheld X-ray fluorescence analysers from manufacturers such as Olympus and Bruker, and RF density meters from Rohde and Schwarz routinely exceed tens of thousands of dollars. They demand trained operators and dedicated facilities. None of them adapt. None of them scale to a delivery driver, a rural pharmacist, or a border inspector with a thousand parcels and a gut feeling.
Two billion people already carry the necessary hardware. Beyond The Box repurposes that existing sensor suite into a material classification capability that operates at the edge, scales to zero cost between uses, and improves with every scan contributed to its training pipeline. No comparable consumer-grade, multi-modal, continuously learning solution exists. The differentiator is not marginal accuracy improvement. It is the elimination of hardware cost and a learning loop that strengthens with every verified scan.
How I Built This
Physical Configuration and Baseline Calibration
One smartphone operates as a BLE emitter, the other as a receiver, placed on opposing sides of the target object. Distance is measured from the receiver to the container's near surface, not phone to phone. This distinction matters because the material effect is applied locally at the object boundary, after which the signal continues to attenuate under inverse-square propagation. A mandatory half-second calibration window executes before every active scan. During this interval, the receiver records the ambient magnetic field vector and the free-space BLE reference level. All subsequent active scan features are computed relative to this environment-specific baseline, effectively removing the influence of local magnetic variation and background RF noise.
The system supports three scan modes. Stationary mode holds the receiver still against a small object and accounts for all current training scans. Swipe mode sweeps across large objects to build a spatial signal profile, intended for mixed-material or oversized containers. Liquid mode uses a slow vertical sweep to detect fill level by exploiting the air-water gradient in partially filled containers. The pipeline supports all three; swipe and liquid mode will mature with additional data.
Signal Processing and Feature Engineering
Each three-second stationary scan yields a multivariate time series of timestamp, velocity, RSSI, and triaxial magnetometer readings. The extraction pipeline, defined in a single feature engineering file that serves as the immutable source of truth for both training and inference, computes 64 engineered features across several categories.
The baseline group captures the ambient environment recorded during calibration. Delta features measure the material's effect as the difference between the active scan and that baseline. The magnetic magnitude group includes statistics such as mean, standard deviation, interquartile range, skewness, kurtosis, and Shannon entropy of the orientation-independent field strength. Gradient features capture how rapidly the field changes across the scan window. Per-axis variance features describe how the disturbance distributes across the X, Y, and Z axes without relying on absolute orientation. A settled-window group re-computes the core physics statistics using only the final 70 percent of each stationary scan. Three distance-normalised features complete the set, correcting for the inverse-cube decay law that governs magnetic dipole fields so that a ferrous object at 30 centimetres produces a comparable feature value to the same object at 5 centimetres.
BLE spectral decomposition provides a second axis of separation. The Fast Fourier Transform of the RSSI time series reveals that water's smooth absorption characteristic produces a low-variance, low-entropy frequency profile. Ferrous metals, by contrast, generate high-variance, high-entropy scatter as multipath interference compounds the magnetic disturbance. BLE is the star signal in this system. The magnetometer confirms ferrous presence, but BLE spectral features are what separate liquid from air and provide a secondary signal when magnetometer readings are ambiguous. The same decomposition applied to the magnetic time series yields ten frequency-domain features across both sensors.
A SHAP-validated subset of the 64 features is selected for model training, ensuring only the most discriminative signals reach the classifier.
NLP Engine
The system includes a natural language processing engine operating on a 794-entry material dictionary spanning ferrous, non-ferrous, liquid, container, and empty-state categories.
Matching uses a three-layer approach. Priority phrases are checked first, so that a specific term like "soda can" resolves to non-ferrous before any general metal match can fire. N-gram generation then breaks the input into one, two, and three-word chunks, preferring longer matches. Fuzzy matching at an 80 percent character similarity threshold handles typos and colloquialisms. Contradiction detection scans for negation and correction keywords and suppresses the corresponding flag when found, so that "not a knife" does not set the ferrous flag.
The output is five binary flags. These flags are deliberately excluded from the feature vector consumed by XGBoost. They operate only within the confidence tiering logic, where high-confidence predictions return immediately, moderate-confidence predictions use NLP agreement as a tiebreaker, and low-confidence predictions either fall back to NLP entirely or prompt the user to rescan. This separation ensures the model's predictions are grounded exclusively in physical sensor data.
Data Quality Through User Interface Design
The mobile application enforces clean data collection at the point of capture. A colour-coded scan mode system provides immediate visual feedback when the operator's physical motion mismatches the expected sensor behaviour. Haptic pulses signal start and stop events, allowing the user to maintain visual focus on the target container. Real-time velocity monitoring aborts scans that exceed strict kinematic thresholds, preventing accelerometer artefacts from contaminating the training pool. The accelerometer's standard deviation across the scan window is converted into a scan stability score that the pipeline uses to weight data quality. After each scan, the user assigns a ground truth rating based on their physical verification of the contents. The pipeline excludes any scan rated below a defined threshold. These interface decisions are machine learning safeguards implemented in user experience form.
AWS Architecture and the Data Flywheel
The system is deployed across two decoupled channels. Channel 1 handles live inference. Flutter transmits the raw sensor stream directly to a Lambda Function URL, bypassing API Gateway and removing a network hop. Lambda executes the feature extraction logic and calls a SageMaker Serverless endpoint hosting the XGBoost model. The endpoint returns four class probabilities, after which Lambda applies the confidence tiering framework. The entire inference path completes in under a second.
Channel 2 manages human-in-the-loop retraining. When a user verifies scan contents, the app publishes the confirmed data via MQTT over TLS to AWS IoT Core. An IoT rule routes the payload into an S3 quarantine bucket organised by material class. After human review, approved scans are promoted to the training corpus and SageMaker retrains the model.
This architecture scales to zero cost between requests. Lambda, SageMaker Serverless, and IoT Core all incur charges only when actively invoked. No idle infrastructure runs continuously. The free tier accommodates substantial development and pilot usage, and the system expands seamlessly to production volumes without architectural modification.
Machine Learning Model and Validation
The classifier is an XGBoost multiclass model trained on 1,181 verified real-world scans distributed across four categories. Air and empty containers account for 270 samples, liquid for 278, ferrous metal for 313, and non-ferrous metal for 320. Objects were scanned at distances from 3 to 30 centimetres, both directly and enclosed within cardboard cartons of varying dimensions. The training pipeline employs a stratified 70/15/15 split with strictly real data, no synthetic oversampling, Bayesian hyperparameter optimisation, and isotonic probability calibration to align reported confidence with empirical accuracy.
On a held-out test set of 178 real scans, the model achieves 83.7 percent overall accuracy. Air recall stands at 95 percent, liquid at 95 percent, ferrous at 74 percent, and non-ferrous at 73 percent. The ferrous and non-ferrous boundaries at longer ranges represent the current performance ceiling, and additional close-range non-ferrous training data is the clearest path to improvement.
The system performs best on single-material contents inside non-metallic packaging, the large majority of real-world use cases. Two boundary conditions are notable. A fully metallic container approaches Faraday cage behaviour, its own signature dominating the contents. Complex mixed-material contents produce blended signals the current model is not trained to decompose. Swipe mode addresses this, but requires training data that does not yet exist at scale.
Demo
The following video demonstrates a live end-to-end scan, from baseline calibration through material prediction, using two standard smartphones and a sealed container.
The alternative link is https://youtube.com/shorts/fwk9l-QrORQ?feature=share
What I Learned
The most resonant feedback from the judging panel concerned the absence of quantitative validation. The original submission lacked accuracy metrics, sample sizes, or independent testing results. That critique was entirely fair. The revised system now stands on a documented 83.7 percent test accuracy across 1,181 real scans, with per-class recall and F1 scores publicly reported. Validation is no longer implied. It is demonstrated.
A second insight emerged from the failure of a hierarchical architecture. Isolating magnetometer features for ferrous detection was physically well motivated, yet the flat model outperformed it. The ferrous class contains weak-signal examples the magnetometer alone cannot resolve. BLE scatter provides a complementary signal. Architecture must follow data, not theory.
The democratisation angle remains central. Industrial material detection has been confined to capital-intensive hardware for decades. Repurposing sensors already in billions of devices and wrapping them in a cloud-native learning loop makes this capability accessible for the first time. Commercialisation paths are straightforward: enterprise API access for logistics verification, a freemium consumer tier with premium analytics, and per-scan micropayments for occasional users all coexist within the same infrastructure.
A provisional patent has been filed covering the methodology. The filing is intentionally broad, protecting the general approach to smartphone-based multi-modal material sensing rather than any single implementation detail.
At 1,181 scans, Beyond The Box is a rigorously validated prototype. At ten thousand scans, its performance at the non-ferrous boundary will tighten materially. At a million scans, contributed organically by users across diverse environments, the training distribution will capture variation that no laboratory could replicate. The value of the system compounds with every verified scan. The architecture was designed from the first commit to harness that compounding effect. What began as a hackathon project is now a demonstrably functional, patent-backed, and commercially viable sensing platform built entirely on hardware that humanity already carries in its pockets.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article