Simulated cortical vision on EC2 behind CloudFront
I built the stimulation step of a simulated cortical visual prosthesis and ran it on EC2 behind CloudFront. What broke, how I closed it, and the numbers.
Series: Toward a world without monitors and keyboards (3 articles)
- 3Simulated cortical vision on EC2 behind CloudFront This article
A cortical visual prosthesis has to turn an image into currents for each electrode. I built that step in PyTorch, along with a simulator that shows what the stimulation would look like, and ran it on AWS so I could try it with a webcam from a browser. Browsers only allow camera access over HTTPS or on localhost, which is why the service sits behind Amazon CloudFront.
This post covers the AWS side: the stack, how I kept the origin closed, and what broke on the way. The results come at the end. Everything here is a simulation. I did not stimulate anything, and I tested on synthetic scenes plus my own laptop camera.
What the service does
The browser captures one camera frame at a time and posts it to the server. The server returns a PNG of the simulated percept, with stats in a response header (mean current, share of electrodes above threshold, inference time). You can switch between 100, 1,000 and 10,000 electrodes, three encoders, calibration on or off, and a mean-current limit.
Server-side inference on a t3.large took 13 to 22 ms for 100 electrodes, 19 to 22 ms for 1,000 and 80 to 100 ms for 10,000, across a few measurements.
The stack
- Amazon EC2: one t3.large on Amazon Linux 2023 in the default VPC, running FastAPI and PyTorch (CPU)
- Amazon CloudFront: caching disabled, with an HTTP origin on port 8080
- Amazon S3: a private bucket holding the app archive, which the instance downloads at boot
- AWS Systems Manager: pushes new code to the instance without SSH
- AWS CloudFormation: creates and deletes all of it
I used a CPU instance on purpose. The On-Demand quota for G and VT instances was 0 in this account, and the model does not need a GPU. It has about 107,000 parameters, and the simulator is a matrix product.
Keeping the origin closed
The security group allows port 8080 only from the CloudFront origin-facing managed prefix list. That list covers every CloudFront distribution, including other people's, so anyone could point their own distribution at my instance's DNS name and pass the security group.
I added two checks in the app. CloudFront attaches a secret custom header (x-origin-verify) to every request it forwards, and the app rejects requests without it. API calls also need an access key in a header. The key travels in the URL fragment, which the browser never sends to the server, and the page removes it from the address bar.
Other settings:
- The service runs as a systemd DynamicUser with a read-only file system, and refuses to start if the access key is missing
- Images are limited to 2 MB and 4 megapixels, and at most two requests are processed at once
- The instance requires IMDSv2 with a hop limit of 1, and the EBS volume is encrypted
One gap remains: the leg from CloudFront to EC2 is plain HTTP, so camera frames cross it unencrypted. A custom domain with an Application Load Balancer (Elastic Load Balancing) and an AWS Certificate Manager (ACM) certificate would close it. I did not do that, and the page says so.
What broke
The first deployment returned 504 from CloudFront. The instance was running, so I read its console output:
1
ERROR: Could not find a version that satisfies the requirement flit_core<4,>=3.11 (from versions: none)Amazon Linux 2023 ships pip 22.3.1. It tried to build typing_extensions from source, then looked for the build dependency on the PyTorch CPU index, which does not carry it. I upgraded pip first and pinned the CPU build of torch with an extra index. User data runs only on first boot, so I renamed the instance's logical ID in the template to make CloudFormation replace it.
Two smaller problems followed. The first request for 10,000 electrodes took 6.5 seconds because the simulator was built lazily, so I now build all three at startup. After an update, my browser kept serving the old page from its cache, which broke the key handling until I sent a no-store header for the page.
I also ran two independent review passes over the code and template, one on security and one on my own claims and rules. They found a decompression-bomb path (a small PNG that expands to a huge image), the service running as root, and an access key that disabled authentication when empty. All three are fixed.
What the numbers say
I compared encoders on 120 synthetic scenes, using a simulated patient that was not used in training. The metric is edge correlation between the percept and the scene, which tracks fine detail.
| Electrodes | Simple sampling, uncalibrated | Simple sampling, calibrated | Learned encoder, calibrated |
|---|---|---|---|
| 100 | 0.030 | 0.077 | 0.274 |
| 1,000 | 0.085 | 0.270 | 0.540 |
| 10,000 | 0.150 | 0.470 | 0.658 |
Calibration mattered as much as the learned encoder. Calibration here means estimating each electrode's threshold by asking a simulated patient whether they saw a spot, 21 questions per electrode. It cut the threshold error from about 51% to about 6%. SSIM did not rise with electrode count (0.603, 0.568, 0.512 for the learned encoder), because it mostly scores large regions.
The limits: the evaluation uses synthetic scenes only, and the simulated patient comes from the same model that trained the encoder, so I have not shown that it transfers to a different patient model. The simulator has no timing and no interaction between neighbouring electrodes.
Teardown
One script deletes the stack and the tagged bucket. I checked afterwards that no stack, instance, distribution, bucket, security group or role was left.
I confirmed the demo with the built-in laptop camera, locally and through CloudFront. In both, the image improved as I raised the electrode count.
This is the output side. The input side, turning neural activity into text, is in my earlier post on rebuilding a brain-to-text decoder with Amazon Bedrock.
Series: Toward a world without monitors and keyboards (3 articles)
- 3Simulated cortical vision on EC2 behind CloudFront This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article