
Docker Series Part 2: Image Layers and Union Filesystems
Why your image is 25x bigger than your code, how to write a Dockerfile that doesn't waste it, and how a registry can take down a fleet at 3 AM.
A team's Python service image is 2.1 GB. Their application code is 80 MB. Someone asks on Slack: "Why is our image 25x bigger than our code?" The first answer is "Python is heavy." The next is "base images are big." Neither is wrong, but neither is the real story either. The real story is that Docker images are built from layers, one per instruction in the Dockerfile, and each layer is permanent. Installing build tools, then deleting them in a later layer, does not reduce the image size; the deleted files are still sitting in the earlier layer, shipped with every pull. The 2 GB image is the sum of every tool that was ever installed during the build, even if the finalRUNcommand deleted them.
This part of the series covers the three things that actually control image size and reliability in production: how layers stack, how to write a Dockerfile that doesn't fight its own cache, and how registries — tags, digests, rate limits — decide whether your deploy even starts.

Part 1: Image Layers and Union Filesystems
Understanding image layers is the difference between "my image is slow and big, and I do not know why" and "I can cut this image from 1.2 GB to 80 MB with three specific changes." This lesson explains how OverlayFS stacks layers, how Docker uses that to make image pulls fast, and the specific patterns that control image size in practice.
A container image isn't one file. It's a manifest (lists the layers and their order), a config (command, env, ports), and a stack of layers — each a tarball of the files one Dockerfile instruction added, changed, or deleted.
docker history nginx:1.25-alpine --no-trunc# RUN /bin/sh -c set -x && ... 13.3MB# ADD file:1d2e2f0c0... in / 7.38MB ← base alpineLayers are immutable and content-addressed: identical bytes get the same SHA-256 digest, so
docker pull skips layers you already have. That's also why you can't "modify" a layer — only stack a new one on top.At runtime, Docker uses OverlayFS to merge the image's read-only layers with a writable "upper" layer unique to that container. Reads walk the stack top-down; writes get copied into the upper layer only (copy-on-write). Every container gets its own upper layer, but they all share one copy of the underlying image — which is why starting a new container is nearly free.
The part that surprises people: image size is the sum of every layer, forever. This Dockerfile looks like it cleans up after itself, but doesn't:
FROM ubuntu:22.04RUN apt-get install -y build-essential # +500MB, layer 2RUN ./compile-my-app.sh # +50MB, layer 3RUN apt-get remove -y build-essential # marks 500MB deleted — doesn't reclaim itThe 500 MB is still in layer 2, which still ships on every pull. Two real fixes:
# Fix 1: one RUN, so intermediate files never become a separate layerRUN apt-get install -y build-essential && \ ./compile-my-app.sh && \ apt-get remove -y build-essential && \ rm -rf /var/lib/apt/lists/*
# Fix 2 (better): multi-stage — covered belowThe other big lever is a
.dockerignore file — without one, COPY . /app happily drags your .git folder, node_modules , local databases, and log files into a layer. One team's images went from 3 GB to 120 MB with a 10-line .dockerignore . The dive tool is worth running on any image over 500 MB — it shows exactly which layer added what, and how much is waste.WAR STORY
A team's images were 3 GB. Their app was 80 MB. Root cause: no.dockerignore, and the project root had a.gitdirectory (900 MB),node_modules/(1.2 GB), a local SQLite database (200 MB), and accumulated build logs (300 MB). A singleCOPY . /appinstruction pulled all of that into a layer. A 10-line.dockerignorecut the image to 120 MB. Total engineering effort: 15 minutes. Days of build time saved across CI per week: many.
Part 2: Writing a Dockerfile That Doesn't Suck
A developer opens a ticket: "Our CI takes 8 minutes per build because Docker reinstalls all Python dependencies every time." The Dockerfile starts with
FROM python:3.11 , then COPY . /app , then RUN pip install -r /app/requirements.txt . The order looks natural — you copy your code, then install its deps. But it is destroying the layer cache. Every code change invalidates the COPY . /app layer, which invalidates the RUN pip install layer, which reinstalls every package from scratch. A two-line reordering drops build time from 8 minutes to 15 seconds.Writing a good Dockerfile is not mysterious. It is about understanding what each instruction does to the layer cache, and ordering things so that slow, stable layers come first and fast, changing layers come last. This lesson is the patterns that actually matter: cache-friendly ordering, COPY vs ADD, multi-stage builds, and the handful of flags that collapse image size 10×.
The rule that explains almost every Dockerfile optimization: layers that change rarely go first; layers that change often go last. The cache invalidates at the first changed instruction and every layer after it rebuilds. This ordering mistake alone turns 15-second builds into 8-minute ones:
# Bad: any code change reinstalls every dependencyCOPY . .RUN pip install -r requirements.txt
# Good: install only reruns when requirements.txt changesCOPY requirements.txt .RUN pip install --no-cache-dir -r requirements.txtCOPY . .A few other patterns that matter more than they look:
-
COPY, notADD.ADDhas magic behavior (extracts tarballs, fetches URLs) that caches poorly and skips checksum verification. UseCOPY, and an explicitRUN curl -fsSL ... && sha256sum -cwhen you actually need a download.
- Multi-stage builds are the single biggest size win available — often 5–10×, for free. Compile with a full toolchain in one stage, then copy only the built artifact into a minimal final stage:
FROM golang:1.21 AS build WORKDIR /src COPY . . RUN CGO_ENABLED=0 go build -o /out/app ./cmd/server
FROM gcr.io/distroless/static-debian12 COPY --from=build /out/app /app ENTRYPOINT ["/app"]Same pattern applies to Node, Python, and Java. A Go image that would be ~900 MB without multi-stage lands around ~20 MB with it.
- Pick the right base.
scratch/distrolessfor static binaries,slimoralpinefor services, full distros only in build stages. Alpine's musl libc occasionally trips up DNS resolution and precompiled Python wheels —slimis the safer default when that bites.
- Exec form only:
CMD ["python", "main.py"], neverCMD python main.py. Shell form runs your process under/bin/sh -c, soSIGTERMhits the shell, not your app, and graceful shutdown never fires.
- BuildKit cache mounts (
RUN --mount=type=cache,target=/root/.cache/pip) persist package-manager caches across builds without baking them into the image — CI gets dramatically faster without any image bloat.
WAR STORY
An engineering team's service images averaged 1.4 GB. A 2-hour audit withdiveand a rewritten Dockerfile brought the average to 180 MB. Specific wins: (1) multi-stage to drop build tools (-400 MB); (2).dockerignoreto exclude.git(-900 MB); (3) movingpip installbeforeCOPY . .(build time from 6 min → 30 s on code-only changes); (4) distroless base for compiled services (-200 MB each). None of this was novel, it was the standard patterns applied consistently. Most teams underinvest in Dockerfile quality because it does not break anything visible; the cost shows up as slow CI and bloated registry bills.
Part 3: Image Registries
At 3 AM, a Kubernetes cluster suddenly starts failing health checks across every service.
kubectl describe pod shows: ErrImagePull: toomanyrequests: You have reached your pull rate limit . The cluster's node pools were rotating, each new node pulling base images from Docker Hub, and the team's anonymous pull quota was exhausted. Nothing was broken in the code. Nothing was wrong with the images. The entire outage was caused by a registry's unauthenticated rate limit and a team that did not know it existed.Registries feel like a commodity until they are in the critical path, which they always are, the moment an orchestrator needs a fresh image and cannot get one. This lesson covers the parts of registries that actually matter in production: tags vs. digests, rate limits, private registries and authentication, and the specific failure modes that take down fleets. If you only remember one thing:
nginx:latest in production is a recipe for unpredictable deployments, and digest pinning is the fix.What a Registry Is
An image registry is an HTTP server that stores and serves OCI images. The API is defined by the OCI Distribution Spec, which every major registry implements:
- Docker Hub (hub.docker.com) — the default, most popular, free for public images, paid for private and high-throughput.
- GitHub Container Registry (GHCR) —
ghcr.io, free for public and private images for GitHub users.
- Amazon ECR — AWS-native, deep IAM integration, tight with ECS / EKS.
- Google Artifact Registry (GAR) — GCP equivalent, replaces the older GCR.
- Azure Container Registry (ACR) — Azure equivalent.
- Quay.io — Red Hat's, popular for open-source projects.
- Harbor — self-hosted, the standard open-source private registry.
- GitLab Container Registry — integrated with GitLab CI.
Because they all implement OCI Distribution, the push/pull protocol is identical:
docker pull works the same way against any of them, docker push works the same way (with auth), and images move between registries with docker tag + docker push .The distinction that matters most: tags are mutable, digests are immutable.
docker pull nginx:1.25 # today: sha256:abc123...# ...a month later, same tag...docker pull nginx:1.25 # sha256:def456... ← different image:latest is the worst offender — it's not "the newest version," just whatever the publisher last pointed it at, and every pull might silently grab a different image. For anything production, pin to a digest:docker pull nginx@sha256:eb05700fe7baa6890b74278e39b66b2ed1326831f9ec3ed4bdc6361a4ac2f333Tag Conventions That Save You
Pick a tagging scheme and enforce it. The good ones:
Tag | Meaning | When it is safe |
|---|---|---|
myapp:1.2.3 | Semantic version, pinned | Production (but tag could still be force-repushed) |
myapp:1.2.3-abc1234 | Semver + short git SHA | Production, excellent |
myapp:abc1234567890 | Full git SHA | CI artifacts, canonical |
myapp:sha-YYYYMMDD-HHMMSS-sha | Timestamp + SHA | Avoid, too long |
myapp@sha256:eb05... | Immutable digest | Production, gold standard |
myapp:latest | Whatever was last pushed | Local dev only |
myapp:dev / myapp:staging | Environment rolling tags | Accept that they drift |
The best combo: CI builds with the full git SHA (
myapp:abc1234567890 ), then on deploy looks up the digest and writes that into the Kubernetes manifest. So the deployed manifest references image: myapp@sha256:... , while humans still work with tags.# Resolve a tag to a digestdocker buildx imagetools inspect myorg/myapp:v1.2.3# Name: docker.io/myorg/myapp:v1.2.3# MediaType: application/vnd.oci.image.index.v1+json# Digest: sha256:eb05700fe7baa6890b74278e39b66b2ed1326831f9ec3ed4bdc6361a4ac2f333A few other things that only bite you in production:
Rate limits are real. Docker Hub caps anonymous pulls at 100/6h per IP — a node pool rotating behind one NAT gateway can blow through that in minutes. Fix, in order of effort: authenticate pulls, run a pull-through cache, or mirror base images into your own registry (ECR/GAR/ACR have no external rate limit on your own images).
- Multi-arch matters on mixed fleets. One tag can point to an image index covering
linux/amd64andlinux/arm64; without it, an ARM node (Graviton, Apple Silicon) hitsexec format error.
- Storage isn't self-cleaning on self-hosted registries — set a retention policy (keep tagged releases, prune untagged pushes after 30 days) before it grows into a real bill.
- Signing (Cosign/Sigstore) is becoming the baseline for knowing what actually shipped — admission controllers can reject anything unsigned.
- Rate limits are real. Docker Hub caps anonymous pulls at 100/6h per IP — a node pool rotating behind one NAT gateway can blow through that in minutes. Fix, in order of effort: authenticate pulls, run a pull-through cache, or mirror base images into your own registry (ECR/GAR/ACR have no external rate limit on your own images).
- Auth is short-lived on cloud registries. ECR tokens expire after 12 hours; a pod that starts after that window fails to pull unless workload identity or a credential provider refreshes it automatically.
WAR STORY
A team deployed 40 new nodes to a Kubernetes cluster during an incident, hoping to scale out. Every new node failedImagePullon their base nginx image. The cluster that was supposed to scale up to meet load instead stayed at its old capacity because the new nodes could not start. Root cause: Docker Hub rate limits on anonymous pulls from a shared NAT gateway IP. Fix: authenticate pulls + mirror frequently-pulled base images to the company's Harbor registry. Once they did this, ImagePullBackOff incidents dropped from weekly to zero.
Takeaways
Image size is a layer-accounting problem, not a mystery: order your Dockerfile so slow, stable steps come first; use multi-stage builds to keep build tools out of the final image; and never let a mutable tag decide what runs in production. Ship your images the same way you ship code — pinned, cache-aware, and reproducible.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article