Redacting PII on Drone Footage: SAM 3 vs Zero Shot YOLO
A hands on comparison of open vocabulary segmentation (Meta's SAM 3) against a zero shot YOLO pipeline for PII redaction on low altitude drone video, with per frame timings on Apple silicon.
The focus of this article is a set of experiments on how well current segmentation and object-detection models can redact personally identifiable information (PII) from drone footage: people, license plates and items such as phone numbers on a billboard. The test was to see how an open-vocabulary segmentation model could performed with no training and how it would compare to the YOLO family.
During the experiments I built a small command-line tool around Meta's SAM 3 that finds and redacts
person and license plate base on a text prompt input, wrote a zero-shot YOLO baseline with the same interface, and ran both on ground-level photos and on about 3,300 frames of low-altitude drone video. On ground-level photos the two are close. On aerial footage, where a person is 40 to 60 pixels tall, SAM 3 wins on recall at roughly 2.5 to 4x the per frame cost. The most useful finding was not a number: adding the prompt "phone number" redacted a printed phone number on a billboard that neither pipeline's default prompts touched. That is the argument for keeping an open-vocabulary model somewhere in a redaction system, even if the model at the edge ends up being a fine-tuned YOLO.Everything below ran on a MacBook Pro (Apple M5 Pro) through PyTorch MPS . No cloud GPUs, no training. All numbers are from my own logs and should be read as a laptop-scale comparison, not a published benchmark.
Why I wanted to know this
Survey drones capture frames that end up in front of a lot of people: analysts, contractors, sometimes the public. Those frames need PII stripped first. The obvious classes are faces or people and license plates, but PII in the wild is messier than two classes: a phone number on a sign, a company name on a van, a whiteboard in a window. Before proposing an architecture I wanted a measured answer to both questions from the introduction: is zero-training good enough, and what does it cost against the edge-friendly alternative.
The tool
redact.py is a single Python file of about 500 lines on top of ultralytics , torch and OpenCV. Per image it:- Downscales to
--max-size(default 2048 px on the long edge) for inference. - Runs
SAM3SemanticPredictorwith the text prompts (defaultpersonandlicense plate). - Unions every returned instance mask back at full resolution.
- Dilates the mask by
--marginpixels so hair, hands and plate frames are covered. - Applies one of four redaction modes:
blur,pixelate,filloralpha(a transparent PNG for downstream compositing).
The interesting part is how little code the detection step is. SAM 3 takes a list of strings and returns instance masks:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
def build_predictor(model_path, conf, device):
from ultralytics.models.sam import SAM3SemanticPredictor
predictor = SAM3SemanticPredictor(overrides={
"conf": conf, "task": "segment", "mode": "predict",
"model": str(model_path), "device": device,
"save": False, "verbose": False,
})
predictor.setup_model()
return predictor
def infer_mask(predictor, image_bgr, prompts, max_size):
h, w = image_bgr.shape[:2]
scale = min(1.0, max_size / max(h, w))
small = cv2.resize(image_bgr, (round(w * scale), round(h * scale))) if scale < 1 else image_bgr
predictor.set_image(small_as_path) # the predictor wants a file path
results = predictor(text=list(prompts))
return collect_union_mask(results, h, w) # OR all instance masks, upscaled to (h, w)Redaction is masked NumPy indexing:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
def redact(image_bgr, mask, mode, strength, fill_color):
sel = mask > 0
out = image_bgr.copy()
if mode == "fill":
out[sel] = fill_color
elif mode == "blur":
k = max(3, strength) | 1
out[sel] = cv2.GaussianBlur(image_bgr, (k, k), 0)[sel]
elif mode == "pixelate":
h, w = image_bgr.shape[:2]
f = max(4, strength)
small = cv2.resize(image_bgr, (w // f, h // f), interpolation=cv2.INTER_LINEAR)
out[sel] = cv2.resize(small, (w, h), interpolation=cv2.INTER_NEAREST)[sel]
return outTwo extras turned out to matter in practice. Some drone payloads add black letterbox bars to every frame, which confuse the model, so a crop flag trims them for inference and pastes them back afterwards. And an interactive
--select-objects mode offers a menu of presets plus free text, which is the best live demo: type sign or whiteboard and watch it work.The zero-shot YOLO baseline
To have something to compare against, I wrote a stand-in with the same command-line flags that swaps the detector for:
- YOLOE-11L-seg , Ultralytics' open-vocabulary segmentation model, prompted with
person. - A public YOLO11 license-plate detector from Hugging Face (
morsetechlab/yolov11-license-plate-detection), because YOLOE could not find plates from a text prompt at all: 1 of 6 plates atconf=0.05with four synonyms.
Plate boxes are rasterized into the same mask and go through the same
redact() function, so the only variable is detection. One hack was needed: the plate model confidently flagged a perforated ceiling beam in a car park as two plates covering 10% of the frame, so I added a --plate-max-area filter (default 2% of the frame; real plates were all under 0.4%).An important caveat for everything below: this is zero-shot YOLO. Nobody trained it on faces or plates. A YOLO11 fine-tuned on face and plate datasets would close much of the recall gap you are about to see. The comparison is "what you get for free" on both sides, not a ceiling for YOLO.
Four photos: a multi story car park with five visible plates, a street scene with two plates, a bus stop with seven pedestrians and a control image.
| Zero-shot YOLO | SAM 3 | |
|---|---|---|
| Plates found (car park) | 3 of 5 readable, plus 2 false positives on the ceiling until the area filter | 5 of 5, including two partial plates; 0 false positives |
| Pedestrians (bus stop) | 7 of 7 | 7 of 7, including a partial figure at the frame edge |
| Time per image | 0.3 to 0.9 s | 1.23 s (6.0 s one-time model load) |
At this scale the two are close once the YOLO side has a dedicated plate model. SAM 3's edge is that it found plates from the words
license plate alone.Four photos: a multi story car park with five visible plates, a street scene with two plates, a bus stop with seven pedestrians and a control image.
| Zero-shot YOLO | SAM 3 | |
|---|---|---|
| Plates found (car park) | 3 of 5 readable, plus 2 false positives on the ceiling until the area filter | 5 of 5, including two partial plates; 0 false positives |
| Pedestrians (bus stop) | 7 of 7 | 7 of 7, including a partial figure at the frame edge |
| Time per image | 0.3 to 0.9 s | 1.23 s (6.0 s one-time model load) |
At this scale the two are close once the YOLO side has a dedicated plate model. SAM 3's edge is that it found plates from the words
license plate alone.What it looks like on footage
The following images and clips are free-license stock footage from Mixkit , ran through the same pipeline as the experiments. 972 frames in total. Same prompts, same dark red fill.
| Clip | What it tests | Frames | Zero-shot YOLO | SAM 3 |
|---|---|---|---|---|
| Crosswalk, top-down, roughly 20 m | Small pedestrians, plates on passing cars | 432 | 0.8 person masks and 0.7 plates per frame, 0.42 s per frame | 2.3 instances per frame, 0.9 s per frame |
| Beach promenade, oblique, roughly 60 m | Dense crowd at tiny scale | 300 | 23.0 person masks per frame | 23.2 instances per frame |
| Highway median path, oblique, roughly 15 m | People on a path, plates in moving traffic | 240 | 8.0 person masks and 3.3 plates per frame | 13.9 instances per frame |
Crosswalk from about 20 m. Both pipelines cover all six pedestrians and both plates in this frame. Over the full clip, SAM 3 keeps the plate on each passing car covered while the plate model flickers on and off.
Beach from about 60 m. Near parity, and a useful limit case: at this altitude a sunbather is a few pixels, and both models cover only the people on the promenade and the largest figures on the sand.
Highway median from about 15 m. The one frame where the baseline wins something: the dedicated plate model catches the plate on the white car at left, which SAM 3 misses at this size, while SAM 3 picks up more of the people further along the path.
The billboard & prompting non people items
In the VisDrone billboard clip, both pipelines covered the crosswalk crowd and the model's face on an OPPO advert. Neither touched the phone number "2323 8855" printed on the billboard, because
person and license plate do not describe it.So I re-ran SAM 3 with one extra prompt:
1
2
python redact.py billboard/ -o out/ --mode fill --fill-color 0,0,139 \
--prompts "person" "license plate" "phone number"It redacted exactly the number line and the small-print contact line below it, and nothing else. Prompt wording matters, though:
"text" returned zero hits, "billboard" blacked out every sign in the frame, and "advertisement text" took the whole blue screen. Adding a PII class is a word, but you still have to find the right word and check the result.To show the same mechanism on footage I can publish, here is a street-level Times Square still (not drone footage, but dense with signage) with the default prompts, then with
"brand logo" added:
Default prompts. Both cover pedestrians and taxi plates. SAM 3 also treats a face on a billboard as a person, which is arguably correct for a redaction tool.
Adding
"brand logo" as a third prompt. It gets Kodak, Samsung, the Corona labels and a car badge, and misses the Coca-Cola script. Same lesson as the phone number: one word gets you most of a new class, and you still have to look at the output.The YOLO path has no equivalent. Redacting phone numbers or logos there means collecting a dataset and training a detector.
Benchmark
All timings on an Apple M5 Pro via MPS, batch size 1. During the five-clip run both models sometimes shared the GPU, so treat these as upper bounds.
| Zero-shot YOLO (YOLOE-11L-seg + plate model) | SAM 3 | |
|---|---|---|
| Weights on disk | 68 MB + 39 MB (+ 572 MB CLIP text encoder) | 3.2 GB |
| Model load | about 2 s | 6.0 to 6.5 s one-time (9.4 s with three prompts) |
| 1344 px frames | 0.21 to 0.25 s per frame | 0.8 to 1.0 s per frame |
| 2688 px frames | 0.47 to 0.83 s per frame | 0.86 to 0.97 s per frame |
| 3840 px (4K) frames, inference at 2048 px | 0.34 s per frame | 0.77 s per frame |
| Three prompts, 2688 px | n/a | 1.46 s per frame |
| Five-clip run total (1,234 frames) | about 4 min | 20.0 min |
| Recall on small people (under 60 px) | Drops mid-distance, seated, lying and occluded people | Consistent |
| Plates at about 25 px | Flickers frame to frame | Steady |
| False positives | Ceiling beam as plate (fixed with an area filter) | None observed |
| Adding a PII class | Train a new detector | Add a prompt |
Rule of thumb: a one-minute 4K drone clip at 30 fps is about 10 minutes of SAM 3 or about 4 minutes of zero-shot YOLO on a laptop. Neither is real time without a GPU server, and SAM 3 at 3.2 GB is never going to run on a drone.
Where this points
The benchmark argues for a two-tier design rather than a winner:
- Edge tier: a YOLO11 or YOLO-seg fine-tuned on face and plate datasets, compiled for whatever the drone or ground station actually carries. For on-drone compute, a low-power NPU such as a Hailo-8 (about 26 TOPS at a few watts) is a better fit than a GPU. The compile step is real work: INT8 quantisation needs a representative calibration set, and NMS typically has to run on the host CPU.
- Cloud tier: SAM 3, or whatever the best open-vocabulary segmenter is next quarter, as the quality pass and the place where new PII classes get added by prompt. Batch jobs from S3 onto GPU instances, with no real-time requirement.
The question that decides the model story is where compute runs: on the drone, on the ground station, or nowhere until upload. That is a conversation with whoever operates the drones, not a benchmark result.
Reproducing it
The whole tool is the two snippets above plus argument parsing and file handling. To rebuild it:
1
2
3
4
5
6
7
8
uv venv --python 3.12 .venv && source .venv/bin/activate
pip install ultralytics opencv-python numpy timm
# accept the licence and download sam3.pt from huggingface.co/facebook/sam3 into models/
python redact.py photos/ -o out/ --mode pixelate
python redact.py photos/ -o out/ --mode fill --fill-color 0,0,139 \
--prompts "person" "license plate" "phone number"
python redact.py photo.jpg -o out/ --select-objects # interactive prompt menuThe three stock clips used in the figures are Mixkit videos 61 (crosswalk), 4456 (beach) and 2021 (highway median), all under the Mixkit Stock Video Free License.
For the YOLO baseline, load
yoloe-11l-seg.pt with model.set_classes(["person"]), add any YOLO plate detector you trust, union the masks and reuse the same redact().If you try this on your own footage, I would like to hear what prompts worked and which ones surprised you. "Phone number" working while "text" returned nothing is the kind of thing you only learn by running it.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article