AWS Builder Center

From Local Sandbox to AWS EC2: Building CodeShield for Secure Untrusted Code Execution

What happens when your application needs to execute code it doesn't fully trust? I built CodeShield, a distributed execution layer that combines Kafka, Docker, PostgreSQL, Redis, and NSJail, and deployed it on Amazon EC2 to explore the challenges of securely running untrusted code in the cloud.

From Local Sandbox to AWS EC2: Building CodeShield for Secure Untrusted Code Execution

Running code is easy.
Running code that you don't trust is a completely different problem.
A coding platform, assessment system, developer tool, or AI-powered application may need to execute programs submitted or generated dynamically. If that code runs directly inside the application's environment, a simple infinite loop can consume resources, while filesystem access, process creation, or network access can introduce much more serious risks.
That problem led me to build CodeShield — a secure execution layer for untrusted code.
The project started as a distributed code execution system running locally. During the AWS First Commit hackathon, I focused on taking that architecture beyond my development machine and deploying it on Amazon EC2.
The interesting part wasn't just getting the application online. It was understanding what changes when a system that depends on containers, workers, Kafka, databases, and sandboxing has to operate on an actual cloud machine.



The Problem

Executing untrusted code creates several challenges at the same time.
A submitted program might:
  • Run indefinitely
  • Consume excessive CPU
  • Consume excessive memory
  • Generate huge amounts of output
  • Create too many processes
  • Attempt to access files outside its workspace
  • Attempt to access the network
  • Crash unexpectedly
  • Fail during compilation
For a traditional web application, the server normally executes application code that the developer controls.
For a code execution platform, the situation is different.
The platform is intentionally accepting code that it does not control.
That means the execution environment needs to be treated as a security boundary.
The goal of CodeShield is therefore simple:
Move untrusted execution away from the application process and into a controlled, isolated execution environment.



What is CodeShield?

CodeShield is a distributed execution platform designed to provide a controlled environment for running untrusted programs.
At a high level, the system separates the API that receives execution requests from the workers that actually execute code.
A simplified flow looks like this:
Client
│
▼
CodeShield API
│
▼
Job Queue
│
▼
Execution Worker
│
▼
NSJail Sandbox
│
├── Compile
├── Execute
└── Evaluate
│
▼
Execution Result

The separation is important.
The API should not be responsible for directly executing arbitrary user programs.
Instead, execution becomes a background workload that can be processed by dedicated workers.



Architecture

The main components of CodeShield are:
  • Node.js API
  • Apache Kafka
  • Execution workers
  • NSJail
  • PostgreSQL
  • Redis
  • Docker
  • Amazon EC2
The API accepts an execution request and performs the necessary validation and application-level processing.
Execution jobs are then passed through the messaging layer.
Workers consume jobs and perform the actual compilation and execution inside the sandbox.
A simplified architecture is:

This architecture creates a useful separation between:
  1. Request handling
  1. Job distribution
  1. Code execution
  1. Persistent application data
  1. Fast-access state/cache



Why Kafka?

Code execution can be unpredictable.
One submission might finish almost instantly, while another might hit a timeout or consume resources until it is terminated.
Putting execution directly inside the API request lifecycle would tightly couple request handling with potentially expensive workloads.
Kafka allows CodeShield to treat execution as a distributed job.
The conceptual flow becomes:
HTTP Request
│
▼
API
│
▼
Kafka Topic
│
▼
Worker
│
▼
Sandbox

This separation means the API and execution workers have different responsibilities.
It also provides a foundation for adding additional workers as execution demand increases.
I deliberately kept the architecture simple rather than trying to introduce a large orchestration layer before it was necessary.



The Security Boundary: NSJail

The most important component of CodeShield is the sandbox.
For this project, I used NSJail to create a restricted environment for executing untrusted programs.
The goal isn't to make an arbitrary program magically "safe."
The goal is to restrict what the program is allowed to do and limit the resources available to it.
Depending on the configured execution profile, the sandbox can be used to control things such as:
  • Processes
  • Execution time
  • Memory/resources
  • Filesystem visibility
  • System calls
  • Networking
  • Process creation
The exact restrictions are defined by the sandbox configuration rather than by the application simply trusting the submitted program.
This distinction is important.
A timeout implemented in application code is useful, but it is only one layer.
A secure execution system needs multiple boundaries between the submitted program and the host environment.



Compilation and Execution

CodeShield separates the execution pipeline into several stages.
Submission
│
▼
Validation
│
▼
Queue
│
▼
Worker
│
▼
Sandbox Workspace
│
▼
Compilation
│
▼
Execution
│
▼
Evaluation
│
▼
Cleanup
│
▼
Result

The compiler handles the compilation stage.
The runner is responsible for executing the generated program while handling execution behavior such as timeouts and process termination.
The evaluator determines the final execution result based on the expected execution behavior.
Keeping these responsibilities separated made it easier to test individual parts of the execution engine.



Why Docker?

CodeShield consists of multiple services and dependencies.
Running everything manually on every development machine would make the environment difficult to reproduce.
Docker provides a consistent runtime environment for the application and its supporting services.
The local architecture includes services such as:
CodeShield API
CodeShield Worker
PostgreSQL
Redis
Kafka
MongoDB / supporting services

The exact services depend on the deployment configuration.
Docker Compose makes it possible to bring the environment up as a group rather than configuring every dependency manually.
More importantly, the same containerized application environment can be moved from local development to the EC2 server.
That reduced the gap between:
"It works on my machine"

and:
"It is actually running on a server."




Why Amazon EC2?

For CodeShield, I wanted direct control over the machine running the execution infrastructure.
That made Amazon EC2 a natural fit.
The execution workers need a Linux environment where I can control:
  • Processes
  • Containers
  • CPU and memory resources
  • Networking
  • Docker
  • NSJail
  • Application configuration
Instead of hiding the underlying machine behind a higher-level deployment abstraction, EC2 allowed me to work directly with the server.
That turned the deployment itself into part of the learning experience.



Deploying CodeShield to EC2

The deployment process involved several steps.

1. Create the EC2 instance

I created an EC2 instance and configured the required access.
SSH was used to administer the server.
ssh -i "your-key.pem" ubuntu@YOUR_EC2_IP

The exact username depends on the operating system image used for the instance.



2. Prepare the environment

After connecting to the instance, I prepared the Linux environment and installed the required container tooling.
The goal was to make the EC2 environment capable of running the same containerized services used during development.



3. Configure Docker Compose

The production deployment uses Docker Compose to run the CodeShield services.
Instead of manually starting every dependency, the services can be started together through the deployment configuration.
Conceptually:
docker compose -f docker-compose.prod.yml up -d --build

The actual command depends on the final production configuration.



4. Configure environment variables

The application requires environment-specific configuration for things such as:
  • Database connections
  • Redis
  • Kafka
  • Authentication
  • Application configuration
Secrets are kept outside the source code and provided through environment configuration.



5. Configure Security Groups

One of the most important parts of the deployment was understanding that not every service should be publicly accessible.
The public interface should expose only what external clients need.
Internal services such as:
PostgreSQL
Redis
Kafka
Zookeeper

should not be exposed to the public internet unnecessarily.
This was one of the differences I noticed between running the system locally and operating it on a cloud server.
Locally, everything is inside my development environment.
On EC2, network exposure becomes an actual security decision.



AWS Architecture

The deployed architecture can be represented as:

EC2 provides the compute environment.
Docker provides service packaging.
Kafka provides asynchronous job distribution.
The workers perform execution.
NSJail provides the sandboxing layer.
The databases provide persistence and supporting application state.



Testing the Security Boundary

One of the most useful parts of building CodeShield was testing what happens when the submitted program behaves badly.
Instead of testing only:
cout << "Hello World";

I also considered execution scenarios such as:

Infinite loops

while (true) {}

The execution should not be allowed to run indefinitely.

Resource-intensive programs

Programs that attempt to consume excessive resources need to be constrained by the execution environment.

Process abuse

Programs that attempt to create excessive numbers of processes should be restricted by the sandbox configuration.

Filesystem access

The executed program should only see the filesystem that the sandbox makes available to it.

Compilation failures

Invalid programs should produce a controlled compilation result instead of crashing the worker.
These tests helped me think about code execution as a security problem rather than simply a compiler problem.



Redis and PostgreSQL

CodeShield uses PostgreSQL for persistent application data.
Prisma is used as the database access layer.
Redis is used where fast-access state or caching is useful.
This gives the system a separation between:
Persistent data → PostgreSQL

Fast-access state/cache → Redis

The exact responsibility of each store is defined by the application rather than treating every piece of data as a database record.



What I Learned Deploying to AWS

The biggest lesson from this project was that local development hides a lot of infrastructure problems.
Locally, I could run multiple services with predictable networking.
On EC2, I had to think about:
  • SSH access
  • Security Groups
  • Public versus private ports
  • Environment variables
  • Docker networking
  • Service dependencies
  • Logs
  • Process management
  • Resource usage
  • Production configuration
A distributed application doesn't become a production system simply because it has multiple services.
Those services still have to be operated somewhere.
EC2 made that distinction very visible.



What I Would Improve

CodeShield is still a project under active development.
There are several areas I would explore next:
  • More execution languages
  • Stronger isolation configurations
  • Better observability
  • More detailed execution metrics
  • Worker scaling
  • Better failure recovery
  • More granular execution policies
  • Production-grade infrastructure automation
I would also like to explore how the same execution layer could support applications that need to execute dynamically generated or AI-generated code.
That is an increasingly interesting use case because AI systems can generate programs dynamically, creating the same fundamental trust problem: the application needs to execute code without automatically trusting it.



The Bigger Idea

The interesting part of CodeShield isn't the compiler.
Compilers already exist.
The interesting problem is the infrastructure around the compiler.
A secure execution platform needs to answer questions such as:
Where does the code run?
What can it access?
How much CPU and memory can it consume?
What happens if it never exits?
How do we isolate execution from the API?
How do we distribute thousands of execution jobs?
How do we operate all of this on a real server?
CodeShield is my attempt to build that execution layer as a distributed system.



Final Architecture

The resulting system looks roughly like this:
┌──────────────────┐
│ Client │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ CodeShield API │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ Kafka │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ Execution Worker │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ NSJail │
│ Sandbox │
└────────┬─────────┘
│
┌──────────┼──────────┐
▼ ▼ ▼
Compiler Runner Evaluator
│
▼
┌──────────────────┐
│ Execution Result │
└──────────────────┘

Supporting infrastructure
┌──────────┐ ┌──────────┐
│PostgreSQL│ │ Redis │
└──────────┘ └──────────┘

Hosted on
┌─────────────────────┐
│ Amazon EC2 │
└─────────────────────┘




Conclusion

Building CodeShield taught me that secure code execution is much more than compiling and running a program.
It requires isolation, resource control, asynchronous processing, failure handling, reproducible environments, and careful infrastructure decisions.
Moving the project from my local machine to Amazon EC2 also changed how I thought about deployment.
The architecture might look clean on a diagram, but making all the services communicate correctly on a real server is where many of the practical engineering problems appear.
That was the most valuable part of the project for me.
I started with a code execution engine.
I ended up learning much more about distributed systems, sandboxing, containers, Linux infrastructure, networking, and cloud deployment along the way.
CodeShield is still evolving, but the core idea is simple:
Don't trust the code you're executing. Build the infrastructure so you don't have to.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article