Graceful Shutdown for Spring Boot Services on Amazon ECS
How to stop dropping in-flight requests during ECS deployments by aligning Spring Boot's shutdown phase, the ECS stop timeout and ALB deregistration delay.
The problem
Every time we rolled out a new task definition for one of our Spring Boot services on ECS Fargate, a handful of requests failed with 502s or connection resets. Nothing was wrong with the new version: the old tasks were simply being killed while they were still serving traffic. The fix was not a single setting but getting three timeouts to agree with each other.
What happens when ECS stops a task
When ECS replaces a task during a deployment, the sequence is roughly:
- The task is deregistered from the ALB target group, and the target enters the "draining" state for the deregistration delay (300 seconds by default).
- ECS sends SIGTERM to the container.
- If the container has not exited after the stop timeout (30 seconds by default), ECS sends SIGKILL.
The catch is that deregistration and SIGTERM happen at nearly the same time. Without graceful shutdown, Spring Boot reacts to SIGTERM by closing the embedded web server immediately, so requests that were already in flight, or that the ALB still routes during the first moments of draining, get cut off.
Step 1: Enable graceful shutdown in Spring Boot
Since Spring Boot 2.3 the embedded server can stop accepting new requests and wait for active ones to finish. In application.yml:
1
2
3
4
5
6
server:
shutdown: graceful
spring:
lifecycle:
timeout-per-shutdown-phase: 20sWith this, on SIGTERM the server stops accepting new connections, lets in-flight requests complete for up to 20 seconds, and only then lets the application context close (connection pools, SQS listeners, schedulers and so on).
Step 2: Give ECS enough time with stopTimeout
The application's shutdown budget must fit inside the container's stop timeout, otherwise SIGKILL arrives first. Set stopTimeout in the task definition a little above the Spring timeout to leave room for context cleanup (the maximum on Fargate is 120 seconds):
1
2
3
4
5
6
7
"containerDefinitions": [
{
"name": "orders-service",
"image": "...",
"stopTimeout": 30
}
]Also make sure the JVM actually receives SIGTERM. If your Dockerfile uses a shell-form ENTRYPOINT (for example ENTRYPOINT java -jar app.jar), the shell becomes PID 1 and may not forward the signal. Use the exec form instead:
1
ENTRYPOINT ["java", "-jar", "/app/app.jar"]Step 3: Tune the ALB deregistration delay
The default 300-second deregistration delay mostly just slows down deployments. Once the application drains its own requests, a delay slightly longer than your slowest request is enough. Setting it to around 30 seconds keeps rollouts fast while still letting the load balancer finish existing connections.
Step 4: Fail readiness early
If you use Spring Boot Actuator health checks for the target group, enable the readiness and liveness probes (management.endpoint.health.probes.enabled: true) and point the ALB health check at /actuator/health/readiness. During graceful shutdown Spring publishes a ReadinessState.REFUSING_TRAFFIC event, so the readiness endpoint starts reporting the instance as out of service while it finishes its remaining work.
Putting the numbers together
- timeout-per-shutdown-phase: 20s (time for in-flight requests to finish)
- ECS stopTimeout: 30s (greater than the Spring timeout plus cleanup)
- ALB deregistration delay: about 30s (longer than your slowest request)
After aligning these three values our deployments stopped producing 502s entirely. The general rule: each layer's timeout should be longer than the layer inside it, so the application always finishes before the platform loses patience.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article