Why your container takes 30s to restart and exits with 137
This article describes a situation that often happens when a pod deliberately doesn't support a SIGTERM call.
Why your container takes 30 seconds to restart (and exits with 137)
A badly configured liveness probe, a PID 1 that ignores SIGTERM, and how the two combine.
When Kubernetes needs to stop a container (a liveness probe failed, the pod was deleted, a rollout is happening), the kubelet sends a SIGTERM to the container's main process, the one running as PID 1. If that process has no handler for SIGTERM, it ignores it, so the kubelet waits for the grace period (30 seconds by default) and then sends a SIGKILL.
That's bad in two ways. First, every restart triggered by a probe is delayed by those 30 seconds. Second, a badly configured liveness probe makes it worse: if the probe fails while the application is still starting, the kubelet has already decided to kill it. Since the process ignores the SIGTERM, it keeps starting during the grace period and may even begin serving requests, but the SIGKILL comes anyway, because the kubelet doesn't check the container's health again. The result is an application that starts, serves for a few seconds, and gets killed in the middle of its work.
Demonstration
I ran into this in my lab while testing probes on a FastAPI app with 2 replicas on a kind cluster.
A badly configured liveness probe
1
2
3
4
5
6
7
8
9
10
11
12
13
livenessProbe:
httpGet:
path: /health
port: 8000
periodSeconds: 2
failureThreshold: 1 # a single failure restarts the container
# no initialDelaySeconds, no startupProbe
readinessProbe:
httpGet:
path: /health
port: 8000
periodSeconds: 3
failureThreshold: 3The liveness probe starts checking the moment the container starts, and one failure is enough to restart it. There's no startup probe to give the app time to boot.
No SIGTERM handler on PID 1 (during boot)
1
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]This is already the right form (the exec form), so uvicorn is PID 1 and gets signals directly. But Python doesn't install a SIGTERM handler by default, and uvicorn only installs its own after it has loaded. Until then, PID 1 has no handler.
The consequence
One of the two replicas:
1
2
3
4
5
6
7
8
9
10
11
$ kubectl get pods -l app=api -w
NAME READY STATUS RESTARTS AGE
api-7ff5f85699-xw25s 1/1 Running 0 6s
api-7ff5f85699-xw25s 0/1 Running 1 (1s ago) 34s
api-7ff5f85699-xw25s 1/1 Running 1 (5s ago) 38s
api-7ff5f85699-xw25s 0/1 Running 2 (1s ago) 66s
api-7ff5f85699-xw25s 1/1 Running 2 (5s ago) 70s
api-7ff5f85699-xw25s 0/1 Running 3 (1s ago) 98s
api-7ff5f85699-xw25s 1/1 Running 3 (5s ago) 102s
api-7ff5f85699-xw25s 0/1 Running 4 (1s ago) 2m10s
api-7ff5f85699-xw25s 1/1 Running 4 (5s ago) 2m14sThe container restarts every ~32 seconds, and after each restart it's Ready (
1/1) in about 4 seconds, serving traffic normally until the next kill. Each cycle looks like this:1
2
3
4
5
6
t=0s container starts; Python is still loading, uvicorn isn't listening yet
t≈1s liveness: GET /health → connection refused → 1 failure → restart
kubelet sends SIGTERM → PID 1 has no handler yet → dropped
t≈4s uvicorn finishes starting → readiness passes → 1/1, serving requests
t≈31s the 30-second grace period ends → SIGKILL → container dies (exit code 137)
t≈32s the container starts again → back to t=0Both replicas were created at the same time, so they died together, and my load generator saw about 2 seconds of failed requests in every cycle.
For comparison, when I deleted one of these pods after it had fully started, it ended as
Completed (exit code 0): uvicorn's handler was already installed, so it shut down cleanly.Why?
Every signal has a default action, and for SIGTERM it's "terminate". If a process doesn't install a handler, the kernel applies the default action and the process dies.
PID 1 is the exception. On Linux, PID 1 is the init process, and if it died from a stray signal the whole system would go down. So the kernel protects it: a signal that PID 1 has no handler for is simply dropped. Only SIGKILL and SIGSTOP can't be ignored, which is why the SIGKILL works after 30 seconds.
Each container has its own PID namespace, so its main process becomes the PID 1 of that namespace and inherits this protection, even though it was never written to be an init process. In my case, the SIGTERM arrived during boot, before uvicorn had installed its handler, and the kernel dropped it.
And the liveness probe is what started it all: it fired while the app was still booting, so every cycle repeats the same story.
If your Dockerfile uses the shell form (CMD uvicorn main:app ...), it's even worse. PID 1 is/bin/sh, which has no handler and doesn't forward signals to its children, so your app never gets the SIGTERM, not even after it has started. Every stop takes 30 seconds and ends with a SIGKILL.
Solution
There are two problems, so there are two fixes.
1. Don't let your app be PID 1: use tini
tini is a tiny init process made for containers. It runs as PID 1, starts your app as its child, forwards the signals it receives, and exits with the app's exit code.
1
2
3
4
5
6
7
FROM python:3.10-slim
RUN apt-get update \
&& apt-get install -y --no-install-recommends tini \
&& rm -rf /var/lib/apt/lists/*
# ... rest of the Dockerfile ...
ENTRYPOINT ["/usr/bin/tini", "--"]
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]Now uvicorn is a regular process, so the special rule no longer applies: a SIGTERM during boot kills it immediately (the default action), and after boot its own handler shuts it down cleanly. No more 30-second wait.
Lowering
terminationGracePeriodSeconds is not a fix: it only shortens the wait, still ends in a SIGKILL, and takes time away from clean shutdowns.2. Give the app time to start: add a startup probe
1
2
3
4
5
6
7
8
9
10
11
12
startupProbe:
httpGet:
path: /health
port: 8000
periodSeconds: 2
failureThreshold: 30 # up to 60 seconds to start
livenessProbe:
httpGet:
path: /health
port: 8000
periodSeconds: 10
failureThreshold: 3While the startup probe is running, the liveness and readiness probes are disabled. Once it passes, the liveness probe takes over. This was the real cause of the loop: a liveness probe with no startup probe, checking from second zero and restarting on the first failure. It's a common mistake, and now it's easy to recognize: restarts at a fixed interval of about 30 seconds plus your boot time.
On AWS
The same happens on Amazon EKS, since it's the same kubelet behavior, and on Amazon ECS, where the default stop timeout before the SIGKILL is also 30 seconds. On ECS you don't even need to change the image: setting
initProcessEnabled: true in the task definition's linuxParameters runs an init process as PID 1 for you, just like tini.Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article