AWS Builder Center
From Pixels to Polygons: Building a 2D-to-3D Converter Without an AI Model

From Pixels to Polygons: Building a 2D-to-3D Converter Without an AI Model

A practical look at how classical computer vision and computational geometry can turn a 2D image into a 3D mesh, and how the prototype could evolve into a scalable AWS-based rendering pipeline.

From Pixels to Polygons: Building a 2D-to-3D Converter Without an AI Model

I started this project with a fairly simple question: how much useful 3D geometry can I get from a single 2D image without sending the image to an external image-to-3D AI service?
There are plenty of systems that can do impressive single-image 3D reconstruction, but that also makes it easy to hide what is actually happening between the image and the final model. I wanted to work through that middle part myself. Instead of asking a model to predict the missing geometry, I built a smaller pipeline around classical computer vision and computational geometry: take an image, find its useful shapes, turn those shapes into polygons, and give those polygons depth.
The result is not a system that can look at a photograph of a chair and reconstruct the chair from every angle. It is doing something much more specific, and in some ways more interesting for this experiment. It takes the visible 2D shape and turns it into an extruded 3D object.
That distinction became important pretty quickly.

The prototype

The actual conversion backend is a small FastAPI application written in Python. Its main dependencies are OpenCV for image processing, NumPy for handling image data, Shapely for polygon geometry, and Trimesh for creating and exporting the final mesh.
The pipeline is essentially:
image → grayscale → blur → threshold → contours → polygons → validation → scaling → extrusion → OBJ
The frontend is a separate React/Vite application built around a workspace where the user can draw or upload an image and work through the conversion flow.
There is an important distinction in the current state of the project: the Python backend contains the actual image-to-mesh conversion pipeline, while parts of the frontend are still prototype UI. The current frontend's "Generate 3D" action uses a simulated generation delay, and its canvas viewer renders a procedurally generated demonstration shape rather than loading the OBJ returned by the backend. The export controls are also currently UI simulations. I think that is worth stating because the project is more useful when the actual implemented pieces are separated from the pieces that are still being built.
The interesting part for me is the backend anyway, because that is where the 2D image actually becomes geometry.


It starts with pixels, not geometry

When an image reaches the  /api/generate-3d  endpoint, the backend reads the uploaded file into memory and decodes it with OpenCV in grayscale.
That first conversion removes something we don't really need for this particular reconstruction method: color.
The algorithm is not trying to understand whether a pixel is blue, red, or green. It is trying to decide where the useful shape is and where the background is. A grayscale image gives us a single intensity value per pixel, which is enough for the thresholding step that follows.
Before thresholding, the image is passed through a Gaussian blur using a 5×5 kernel.
At first glance, blurring sounds like it would make the result worse. After all, we eventually want the boundary of an object, so why deliberately soften the image? In this case the blur is there to reduce small variations that could otherwise turn into unwanted edges during thresholding. A photograph or scanned drawing can contain tiny changes in intensity that have nothing to do with the actual shape we want to reconstruct.
The next step is a binary threshold:
blur = cv2.GaussianBlur(img, (5, 5), 0)

_, thresh = cv2.threshold(
blur, 127, 255, cv2.THRESH_BINARY_INV
)

Pixels are divided around an intensity value of 127, and the result is inverted. The important outcome is that the continuous grayscale image becomes a much simpler representation: foreground-like regions and background-like regions.
This is the first major conceptual jump in the pipeline. We have stopped treating the input as an image that needs to be "understood" and started treating it as a map from which boundaries can be extracted.

Finding the shape

Once the thresholded image exists, OpenCV's contour detection is used:
contours, _ = cv2.findContours(
thresh,
cv2.RETR_EXTERNAL,
cv2.CHAIN_APPROX_SIMPLE
)

A contour is essentially a sequence of points describing the boundary of a connected region.
This is where the image starts turning into something that a geometry library can work with.
The choice of  RETR_EXTERNAL  is significant. The backend asks OpenCV for the outermost contours rather than constructing a full contour hierarchy containing internal boundaries and nested contours.
That works well for simple silhouettes. If the input is a clean shape on a contrasting background, the outer contour gives us exactly the boundary we need to start building a solid object.
It also immediately tells us one of the limitations of the current implementation.
Consider a shape like a letter "O". The hole in the middle is meaningful geometry. With a full contour hierarchy, the outer boundary and inner boundary could be treated separately and used to create a mesh with a hole. The current implementation only retrieves external contours, so it does not explicitly preserve that kind of nested interior boundary.
That is not a bug that needs to be hidden. It is simply a consequence of choosing a simpler contour extraction strategy for the first version.

Not every contour deserves to become a model

Contour detection is deliberately generous. It can find regions that are technically boundaries but are not useful objects.
A small speck of noise, for example, can become its own contour.
The backend therefore checks every detected contour before passing it into the geometry stage.
First, the contour is squeezed into a simpler array representation and rejected if it does not contain at least three usable points:
if len(cnt.shape) != 2 or cnt.shape[0] < 3:
continue

That makes sense geometrically. Two points give us a line. We need at least three points to describe an area.
The remaining contour points are converted into coordinate pairs and passed to Shapely:
coords = [(float(x), float(y)) for x, y in cnt]
poly = Polygon(coords).buffer(0)

The  buffer(0)  operation is an interesting little piece of the pipeline. It is commonly used as a geometry cleanup technique for certain invalid polygon configurations. Instead of assuming that every contour OpenCV produces is immediately a valid polygon, the backend attempts to clean the geometry before checking it.
The polygon is then rejected if it is still invalid or if its area is below 50:
if not poly.is_valid or poly.area < 50:
continue

That area threshold is doing a simple but useful job. A tiny contour is unlikely to be a meaningful object in the final model, so throwing it away prevents a lot of insignificant geometry from reaching the extrusion stage.
This is also where the difference between computer vision and geometry becomes clear. OpenCV is helping answer "where are the boundaries?" Shapely is helping answer "does this boundary form usable geometry?"

From image coordinates to model coordinates

At this point, the polygon still exists in image space.
Its coordinates are measured in pixels. An image might contain points such as  (810, 180)  or  (450, 320) . Those values make sense inside the image, but they are not particularly convenient as the coordinate system for a small 3D model.
The backend therefore normalizes the polygon by dividing its coordinates by 100:
poly = Polygon(
[(x / 100.0, y / 100.0) for x, y in poly.exterior.coords]
)

This is not a physical measurement conversion. The image does not suddenly become 8.1 centimetres wide because a coordinate became  8.1 . It is simply a scale normalization that brings the geometry into a smaller coordinate range before extrusion.
There is another subtle detail here: the implementation uses  poly.exterior.coords .
That means the extrusion stage is based on the polygon's exterior ring. Internal holes are not being carried through this conversion stage. Again, that makes the current converter much better suited to solid silhouettes than shapes where interior cutouts are essential.

The step that actually makes it 3D

This was the part I was most interested in when building the prototype, because the phrase "2D to 3D" can make the process sound much more complicated than it actually is in this implementation.
The polygon already describes a complete 2D shape. We do not need to invent an entirely new surface representation. We can simply give that shape thickness.
The backend does this with Trimesh:
mesh = trimesh.creation.extrude_polygon(
poly,
height=1.0
)

Conceptually, imagine the polygon lying flat on an XY plane.
Now make a second copy of that same boundary at another Z coordinate. The first polygon forms one end of the object, the second polygon forms the other end, and the corresponding edges between them become the side surfaces.
The flat outline has become a volume.
If the original polygon looks like this:
______
| |
| |
| |
|______|

extrusion effectively turns it into something closer to:
______
/ /|
/______/ |
| | |
| | /
|______|/

The important point is that the algorithm is not predicting what the back of an object should look like. It is copying the known 2D boundary and assigning a fixed depth of  1.0 .
That is why I would describe the result as 2D silhouette extrusion, rather than general-purpose single-image 3D reconstruction.
For a logo, icon, simple diagram, flat silhouette, or other shape where a constant thickness makes sense, this can be perfectly reasonable. For a photograph of a real object, it leaves out almost everything that makes the object genuinely three-dimensional: perspective, hidden surfaces, changing depth, material properties, and structural information that simply isn't present in the silhouette.

Multiple contours become multiple meshes

The backend does not stop after finding one contour.
Every valid contour is converted into its own extruded mesh and added to a list:
meshes = []

for cnt in contours:
...
mesh = trimesh.creation.extrude_polygon(poly, height=1.0)
meshes.append(mesh)

At the end, the individual meshes are combined:
final_mesh = trimesh.util.concatenate(meshes)

This is useful when an image contains several disconnected foreground regions. Instead of forcing everything into one polygon, each valid external contour can produce its own solid and the resulting meshes can be concatenated into the final object.
It also creates an interesting edge case. If disconnected regions are actually parts of one logical object, the current system has no semantic understanding that they belong together. They simply remain separate pieces of geometry.
That is one of the places where the limitations of a classical image-processing pipeline become obvious. The system knows about pixels, contours, areas and polygons. It does not know that two shapes "belong to the same object" in the way a more sophisticated recognition system might.

Exporting the result

Once the meshes are combined, the backend generates a UUID and writes the result into an  outputs  directory as an OBJ file.
model_id = str(uuid.uuid4())

model_path = os.path.join(
OUTPUT_DIR,
f"{model_id}.obj"
)

final_mesh.export(model_path)

The API then returns the generated model ID and a URL for retrieving it.
There is also a separate endpoint that checks whether the OBJ exists and serves it back as a file response.
So the actual backend flow is quite straightforward:
upload image → process image → build mesh → save OBJ → return model ID
There is no external image-to-3D API sitting in the middle of this process.

What the 3D model actually means

This is probably the most important caveat of the project.
A single 2D image does not contain enough information to reconstruct arbitrary 3D geometry perfectly. If I take a photograph of a mug, the image tells me something about the visible outline, but it does not directly tell me the exact geometry on the opposite side of the mug or how deep every part of it is.
The current project deliberately avoids pretending otherwise.
Its depth model is simple: take the extracted 2D polygon and extrude it by a fixed height.
That means the quality of the result depends heavily on the input.
A clean black silhouette against a light background is a very different problem from a photograph of an object sitting on a cluttered desk. The first can produce a useful contour with relatively little work. The second introduces shadows, textures, reflections, background objects and ambiguous boundaries, all of which can end up influencing thresholding and contour extraction.
For this reason, I see the current approach as being most naturally suited to things like logos, icons, simple illustrations, silhouettes, flat objects and other inputs where the outline itself carries most of the information we care about.

The difficult cases are where the interesting engineering starts

The clean examples are actually the easy part.
A noisy background can produce unwanted contours. Poor contrast can make the foreground and background difficult to separate with a fixed threshold of 127. Very small regions can disappear because of the area filter. Complex outlines can produce a large number of points. Self-intersecting or otherwise problematic contours can fail polygon validation.
There is also a fundamental ambiguity around depth.
Even if the contour is extracted perfectly, the algorithm still has to decide how much depth to give the shape. At the moment that decision is fixed at  1.0 . That is convenient because it makes the geometry deterministic, but it also means every part of a shape receives the same thickness.
A future version could make that much more interesting without introducing an AI model. Better segmentation could make the first stage more reliable. Contour hierarchy could preserve holes and nested regions. Polygon decomposition could make complex shapes easier to triangulate and manipulate. Perspective correction could help when the input is photographed at an angle.
Depth itself could eventually become more than a single constant. Multiple depth layers could be assigned to different regions, or a user could specify thickness interactively. Mesh cleanup, smoothing and decimation could help control unnecessary geometry. UV coordinates and materials could make the exported model more useful in a conventional 3D workflow.
These are extensions to the geometry pipeline, not things the current implementation already does.

The browser side is a separate piece of the puzzle

The frontend is built with React and Vite and provides a workspace for drawing or uploading input, confirming it, starting the generation flow and adjusting the appearance of the preview.
The current viewer is deliberately lightweight. Instead of relying on a full WebGL scene, the  ThreeDViewer  component uses an HTML canvas and performs its own basic projection and lighting calculations. It generates an extruded demonstration shape, rotates it, calculates face normals using cross products, applies a simple lighting model and uses a painter's algorithm to order faces by depth before drawing them.
That part of the project taught me something slightly different from the backend. Rendering a 3D object does not necessarily require a massive rendering engine to understand the underlying idea. At a basic level, the viewer is taking 3D coordinates, rotating them, projecting them into 2D and filling triangles.
The current viewer, however, is not yet wired to the OBJ generated by the FastAPI backend. Likewise, the frontend export controls currently simulate the export action rather than downloading the backend-generated model. The backend's implemented export format is OBJ.
That separation is useful to keep in mind when looking at the project as a whole. The conversion algorithm exists; the product layer around it is still evolving.


Where a local pipeline starts becoming a systems problem

Running the converter locally is relatively simple.
A request arrives, the image is processed, the mesh is generated and the result is saved. For a prototype, that is enough.
The problem changes when the number of users changes.
Imagine several people uploading images at the same time. Some images could be larger than others. Some could take longer to process. A future version might add more expensive segmentation, multiple depth layers or additional mesh processing. If the API server performs all of that work synchronously, each request is tied to the machine doing the computation.
That is where the AWS part of the project starts to make sense.
I would not change the actual computer-vision pipeline just because it is moving to the cloud. The more useful change would be to separate request handling from rendering work.
Instead of:
User → API → process image → return model
the system could become:
User → API → create job → queue → rendering worker → store model → update status → user retrieves model
The distinction looks small on paper, but it changes how the system behaves under load.

Taking the converter to AWS

In a possible AWS version, the uploaded image and generated OBJ would live in Amazon S3. That solves a very practical problem: the generated model needs to remain available after the machine that created it finishes processing. The worker should not have to act as permanent file storage.
API Gateway could sit in front of the application-facing API, while a lightweight Lambda function could handle operations such as creating a conversion job, validating a request or checking job status. The important part is that Lambda would not need to perform the heavy image-processing work itself.
The conversion request could be placed into SQS.
That queue is useful specifically because image-to-mesh generation is a job rather than a normal instant API response. If five users upload images together, they do not all have to fight for the same processing request at exactly the same moment. Jobs can wait in the queue until a worker is available.
The worker itself is where I would keep the existing Python pipeline. A containerized environment such as ECS or AWS Batch would make more sense here than trying to force OpenCV, Shapely and Trimesh processing into a tiny request handler. The container could receive a job, download the source image from S3, run the same contour and extrusion pipeline, write the OBJ back to S3 and report the result.
For job metadata, DynamoDB could store something simple such as the job ID, current status, input location, output location and error information. The frontend would then have something concrete to ask for rather than waiting on a long HTTP connection.
If the workflow eventually becomes more complicated, Step Functions could coordinate stages such as preprocessing, reconstruction, validation and post-processing. I would only introduce it once there are actually multiple workflow stages that benefit from that orchestration. A simple queue and worker does not need an orchestration service just for the sake of having one.
CloudWatch would be useful for the operational side. If a worker starts failing on a particular class of image, or jobs begin accumulating in the queue, those are problems that become much harder to see once the system is running remotely. Logs and metrics would make those failures observable.
CloudFront could also make sense for delivering the frontend and, depending on how the generated assets are exposed, distributing model files more efficiently. Again, that becomes useful when there is an actual delivery problem to solve rather than because every cloud architecture diagram needs a CDN.

One image's journey through the cloud version

Take the same simple silhouette that works with the local converter.
The user uploads it through the frontend. Instead of sending the entire processing job through one long-running request, the API stores the input image in S3 and creates a job record containing its ID and status.
The job is placed into SQS.
A processing worker picks it up. The worker runs essentially the same pipeline I built locally: OpenCV reads the image, converts it to grayscale, applies Gaussian blur, thresholds it, extracts external contours, filters unusable contours, converts them into Shapely polygons, validates them, normalizes their coordinates and passes them to Trimesh for extrusion.
The worker exports the OBJ and puts it into S3.
The job record changes from something like  processing  to  completed  and points to the generated model.
The frontend does not need to know where the worker ran, how long the Python process took or which machine handled the request. It only needs the job status and, when processing is complete, a way to retrieve the resulting model.
That is the part of the cloud architecture I find most relevant to this project. The cloud services are not replacing the computer-vision algorithm. They are giving that algorithm a better environment to run when the number of conversions grows.

What I would work on next

The most obvious next step is improving the connection between the backend and frontend so that the generated OBJ becomes the actual model displayed in the browser instead of the current demonstration geometry.
After that, I would spend more time on the image-processing side.
A fixed threshold is a useful starting point, but it is also one of the first things that will struggle with real-world inputs. Adaptive thresholding, better background removal, morphological operations and more deliberate preprocessing could make the contour stage much more reliable.
Contour hierarchy is another major improvement. Supporting inner contours would allow the system to preserve holes rather than treating only the outside boundary as meaningful.
The geometry stage could also become smarter. Instead of directly extruding every accepted polygon, complex shapes could be decomposed into simpler regions, validated more aggressively and cleaned before extrusion. Mesh decimation could then reduce unnecessary geometry after generation.
Depth is probably the biggest opportunity. A constant extrusion height is enough to prove the concept, but variable depth would make the result much more expressive. Even without AI, there are ways to derive additional depth information from structured inputs, user annotations, perspective cues or multiple layers.
Finally, the export side could become more complete. The frontend currently presents GLB, OBJ and STL as possible formats, but the backend implementation currently exports OBJ. Adding actual GLB/glTF and STL generation would make those controls correspond to real output formats rather than just interface options.

What I learned from building it

The part I expected to be difficult was the 3D generation.
It turned out that extrusion itself was one of the easier parts.
Once a clean polygon exists, turning it into a solid is conceptually straightforward. The harder problem is getting a polygon that is actually worth extruding. A few pixels of noise can become a contour. A contour can become an invalid polygon. A valid polygon can still represent the wrong part of the image.
That changed how I think about the phrase "2D to 3D." The interesting work is not simply the final call that creates a mesh. It is the chain of decisions that happens before that call: what counts as foreground, which contours matter, which points should survive, whether the resulting geometry is valid, and what assumptions we are willing to make about depth.
It also made the AWS part feel much more natural.
The cloud architecture is not the project. The conversion pipeline is.
But once that pipeline exists, it becomes reasonable to ask what happens when the local prototype has to process hundreds or thousands of jobs instead of one image at a time. At that point, storing files separately, putting jobs into a queue and running the actual geometry pipeline in dedicated workers starts solving real problems.

Closing

The prototype started with a simple idea: take something that exists only as pixels and see whether I could turn its visible shape into actual 3D geometry without depending on an external AI reconstruction service.

The answer is yes, within a fairly specific boundary.
The current system can extract external shapes from suitable images, turn those contours into valid polygons, normalize them and extrude them into 3D meshes. It does not recover hidden geometry, understand arbitrary objects or magically infer depth that was never present in the image.
That limitation is actually what makes the project interesting to me. There is a lot that can be done before needing a model to predict what the image does not contain.
The next challenge is no longer just making one image become one mesh. It is making the pipeline better at difficult images, connecting the generated geometry to the browser properly, and eventually separating the rendering job from the API so the same idea can operate as a real cloud-based service.
The pixels are only the starting point. The interesting part is everything that happens between the pixels and the mesh. # buildatpce 
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article