AWS Builder Center
Docker Series Part 2: Image Layers and Union Filesystems

Docker Series Part 2: Image Layers and Union Filesystems

Why your image is 25x bigger than your code, how to write a Dockerfile that doesn't waste it, and how a registry can take down a fleet at 3 AM.

Cloud Infrastructure Engineer
A team's Python service image is 2.1 GB. Their application code is 80 MB. Someone asks on Slack: "Why is our image 25x bigger than our code?" The first answer is "Python is heavy." The next is "base images are big." Neither is wrong, but neither is the real story either. The real story is that Docker images are built from layers, one per instruction in the Dockerfile, and each layer is permanent. Installing build tools, then deleting them in a later layer, does not reduce the image size; the deleted files are still sitting in the earlier layer, shipped with every pull. The 2 GB image is the sum of every tool that was ever installed during the build, even if the final  RUN  command deleted them.
This part of the series covers the three things that actually control image size and reliability in production: how layers stack, how to write a Dockerfile that doesn't fight its own cache, and how registries — tags, digests, rate limits — decide whether your deploy even starts.


Part 1: Image Layers and Union Filesystems

Understanding image layers is the difference between "my image is slow and big, and I do not know why" and "I can cut this image from 1.2 GB to 80 MB with three specific changes." This lesson explains how OverlayFS stacks layers, how Docker uses that to make image pulls fast, and the specific patterns that control image size in practice.
A container image isn't one file. It's a manifest (lists the layers and their order), a config (command, env, ports), and a stack of layers — each a tarball of the files one Dockerfile instruction added, changed, or deleted.
docker history nginx:1.25-alpine --no-trunc
# RUN /bin/sh -c set -x && ... 13.3MB
# ADD file:1d2e2f0c0... in / 7.38MB ← base alpine
Layers are immutable and content-addressed: identical bytes get the same SHA-256 digest, so  docker pull  skips layers you already have. That's also why you can't "modify" a layer — only stack a new one on top.
At runtime, Docker uses OverlayFS to merge the image's read-only layers with a writable "upper" layer unique to that container. Reads walk the stack top-down; writes get copied into the upper layer only (copy-on-write). Every container gets its own upper layer, but they all share one copy of the underlying image — which is why starting a new container is nearly free.
The part that surprises people: image size is the sum of every layer, forever. This Dockerfile looks like it cleans up after itself, but doesn't:
FROM ubuntu:22.04
RUN apt-get install -y build-essential # +500MB, layer 2
RUN ./compile-my-app.sh # +50MB, layer 3
RUN apt-get remove -y build-essential # marks 500MB deleted — doesn't reclaim it
The 500 MB is still in layer 2, which still ships on every pull. Two real fixes:
# Fix 1: one RUN, so intermediate files never become a separate layer
RUN apt-get install -y build-essential && \
./compile-my-app.sh && \
apt-get remove -y build-essential && \
rm -rf /var/lib/apt/lists/*

# Fix 2 (better): multi-stage — covered below
The other big lever is a  .dockerignore  file — without one,  COPY . /app  happily drags your  .git  folder,  node_modules , local databases, and log files into a layer. One team's images went from 3 GB to 120 MB with a 10-line  .dockerignore . The   dive   tool is worth running on any image over 500 MB — it shows exactly which layer added what, and how much is waste.
WAR STORY
A team's images were 3 GB. Their app was 80 MB. Root cause: no  .dockerignore , and the project root had a  .git  directory (900 MB),  node_modules/  (1.2 GB), a local SQLite database (200 MB), and accumulated build logs (300 MB). A single  COPY . /app  instruction pulled all of that into a layer. A 10-line  .dockerignore  cut the image to 120 MB. Total engineering effort: 15 minutes. Days of build time saved across CI per week: many.

Part 2: Writing a Dockerfile That Doesn't Suck

A developer opens a ticket: "Our CI takes 8 minutes per build because Docker reinstalls all Python dependencies every time." The Dockerfile starts with  FROM python:3.11 , then  COPY . /app , then  RUN pip install -r /app/requirements.txt . The order looks natural — you copy your code, then install its deps. But it is destroying the layer cache. Every code change invalidates the  COPY . /app  layer, which invalidates the  RUN pip install  layer, which reinstalls every package from scratch. A two-line reordering drops build time from 8 minutes to 15 seconds.
Writing a good Dockerfile is not mysterious. It is about understanding what each instruction does to the layer cache, and ordering things so that slow, stable layers come first and fast, changing layers come last. This lesson is the patterns that actually matter: cache-friendly ordering, COPY vs ADD, multi-stage builds, and the handful of flags that collapse image size 10×.
The rule that explains almost every Dockerfile optimization: layers that change rarely go first; layers that change often go last. The cache invalidates at the first changed instruction and every layer after it rebuilds. This ordering mistake alone turns 15-second builds into 8-minute ones:
# Bad: any code change reinstalls every dependency
COPY . .
RUN pip install -r requirements.txt

# Good: install only reruns when requirements.txt changes
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
A few other patterns that matter more than they look:
  •  COPY , not  ADD .  ADD  has magic behavior (extracts tarballs, fetches URLs) that caches poorly and skips checksum verification. Use  COPY , and an explicit  RUN curl -fsSL ... && sha256sum -c  when you actually need a download.
  • Multi-stage builds are the single biggest size win available — often 5–10×, for free. Compile with a full toolchain in one stage, then copy only the built artifact into a minimal final stage:
FROM golang:1.21 AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /out/app ./cmd/server

FROM gcr.io/distroless/static-debian12
COPY --from=build /out/app /app
ENTRYPOINT ["/app"]
Same pattern applies to Node, Python, and Java. A Go image that would be ~900 MB without multi-stage lands around ~20 MB with it.
  • Pick the right base.  scratch / distroless  for static binaries,  slim  or  alpine  for services, full distros only in build stages. Alpine's musl libc occasionally trips up DNS resolution and precompiled Python wheels —  slim  is the safer default when that bites.
  • Exec form only:  CMD ["python", "main.py"] , never  CMD python main.py . Shell form runs your process under  /bin/sh -c , so  SIGTERM  hits the shell, not your app, and graceful shutdown never fires.
  • BuildKit cache mounts ( RUN --mount=type=cache,target=/root/.cache/pip ) persist package-manager caches across builds without baking them into the image — CI gets dramatically faster without any image bloat.
WAR STORY
An engineering team's service images averaged 1.4 GB. A 2-hour audit with  dive  and a rewritten Dockerfile brought the average to 180 MB. Specific wins: (1) multi-stage to drop build tools (-400 MB); (2)  .dockerignore  to exclude  .git  (-900 MB); (3) moving  pip install  before  COPY . .  (build time from 6 min → 30 s on code-only changes); (4) distroless base for compiled services (-200 MB each). None of this was novel, it was the standard patterns applied consistently. Most teams underinvest in Dockerfile quality because it does not break anything visible; the cost shows up as slow CI and bloated registry bills.

Part 3: Image Registries

At 3 AM, a Kubernetes cluster suddenly starts failing health checks across every service.  kubectl describe pod  shows:  ErrImagePull: toomanyrequests: You have reached your pull rate limit . The cluster's node pools were rotating, each new node pulling base images from Docker Hub, and the team's anonymous pull quota was exhausted. Nothing was broken in the code. Nothing was wrong with the images. The entire outage was caused by a registry's unauthenticated rate limit and a team that did not know it existed.
Registries feel like a commodity until they are in the critical path, which they always are, the moment an orchestrator needs a fresh image and cannot get one. This lesson covers the parts of registries that actually matter in production: tags vs. digests, rate limits, private registries and authentication, and the specific failure modes that take down fleets. If you only remember one thing:  nginx:latest  in production is a recipe for unpredictable deployments, and digest pinning is the fix.

What a Registry Is

An image registry is an HTTP server that stores and serves OCI images. The API is defined by the OCI Distribution Spec, which every major registry implements:
  • Docker Hub (hub.docker.com) — the default, most popular, free for public images, paid for private and high-throughput.
  • GitHub Container Registry (GHCR) —  ghcr.io , free for public and private images for GitHub users.
  • Amazon ECR — AWS-native, deep IAM integration, tight with ECS / EKS.
  • Google Artifact Registry (GAR) — GCP equivalent, replaces the older GCR.
  • Azure Container Registry (ACR) — Azure equivalent.
  • Quay.io — Red Hat's, popular for open-source projects.
  • Harbor — self-hosted, the standard open-source private registry.
  • GitLab Container Registry — integrated with GitLab CI.
Because they all implement OCI Distribution, the push/pull protocol is identical:  docker pull  works the same way against any of them,  docker push  works the same way (with auth), and images move between registries with  docker tag  +  docker push .
The distinction that matters most: tags are mutable, digests are immutable.
docker pull nginx:1.25 # today: sha256:abc123...
# ...a month later, same tag...
docker pull nginx:1.25 # sha256:def456... ← different image
 :latest  is the worst offender — it's not "the newest version," just whatever the publisher last pointed it at, and every pull might silently grab a different image. For anything production, pin to a digest:
docker pull nginx@sha256:eb05700fe7baa6890b74278e39b66b2ed1326831f9ec3ed4bdc6361a4ac2f333

Tag Conventions That Save You

Pick a tagging scheme and enforce it. The good ones:
Tag
Meaning
When it is safe
myapp:1.2.3
Semantic version, pinned
Production (but tag could still be force-repushed)
myapp:1.2.3-abc1234
Semver + short git SHA
Production, excellent
myapp:abc1234567890
Full git SHA
CI artifacts, canonical
myapp:sha-YYYYMMDD-HHMMSS-sha
Timestamp + SHA
Avoid, too long
myapp@sha256:eb05...
Immutable digest
Production, gold standard
myapp:latest
Whatever was last pushed
Local dev only
myapp:dev / myapp:staging
Environment rolling tags
Accept that they drift
The best combo: CI builds with the full git SHA ( myapp:abc1234567890 ), then on deploy looks up the digest and writes that into the Kubernetes manifest. So the deployed manifest references  image: myapp@sha256:... , while humans still work with tags.
# Resolve a tag to a digest
docker buildx imagetools inspect myorg/myapp:v1.2.3
# Name: docker.io/myorg/myapp:v1.2.3
# MediaType: application/vnd.oci.image.index.v1+json
# Digest: sha256:eb05700fe7baa6890b74278e39b66b2ed1326831f9ec3ed4bdc6361a4ac2f333
A few other things that only bite you in production:
Rate limits are real. Docker Hub caps anonymous pulls at 100/6h per IP — a node pool rotating behind one NAT gateway can blow through that in minutes. Fix, in order of effort: authenticate pulls, run a pull-through cache, or mirror base images into your own registry (ECR/GAR/ACR have no external rate limit on your own images).
  • Multi-arch matters on mixed fleets. One tag can point to an image index covering  linux/amd64  and  linux/arm64 ; without it, an ARM node (Graviton, Apple Silicon) hits  exec format error .
  • Storage isn't self-cleaning on self-hosted registries — set a retention policy (keep tagged releases, prune untagged pushes after 30 days) before it grows into a real bill.
  • Signing (Cosign/Sigstore) is becoming the baseline for knowing what actually shipped — admission controllers can reject anything unsigned.
  • Rate limits are real. Docker Hub caps anonymous pulls at 100/6h per IP — a node pool rotating behind one NAT gateway can blow through that in minutes. Fix, in order of effort: authenticate pulls, run a pull-through cache, or mirror base images into your own registry (ECR/GAR/ACR have no external rate limit on your own images).
  • Auth is short-lived on cloud registries. ECR tokens expire after 12 hours; a pod that starts after that window fails to pull unless workload identity or a credential provider refreshes it automatically.
WAR STORY
A team deployed 40 new nodes to a Kubernetes cluster during an incident, hoping to scale out. Every new node failed  ImagePull  on their base nginx image. The cluster that was supposed to scale up to meet load instead stayed at its old capacity because the new nodes could not start. Root cause: Docker Hub rate limits on anonymous pulls from a shared NAT gateway IP. Fix: authenticate pulls + mirror frequently-pulled base images to the company's Harbor registry. Once they did this, ImagePullBackOff incidents dropped from weekly to zero.

Takeaways

Image size is a layer-accounting problem, not a mystery: order your Dockerfile so slow, stable steps come first; use multi-stage builds to keep build tools out of the final image; and never let a mutable tag decide what runs in production. Ship your images the same way you ship code — pinned, cache-aware, and reproducible.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article