The ECS Service Was Healthy. Users Still Got 502s.

A practical guide to zero-downtime ECS deployments with Application Load Balancer, covering health checks, deregistration delay, connection draining, SIGTERM, Spring Boot graceful shutdown, deployment settings, and rollback.

Ashish kumar jha

Software Devloper

Published Oct 6, 2026

Last modified Oct 6, 2026

The ECS Service Was Healthy. Users Still Got 502s.

The ECS deployment completed successfully.

The tasks are RUNNING.

The target group says the new tasks are healthy.

And users are still getting:

1
502 Bad Gateway

This is one of the most confusing deployment failures in containerized applications.

The mistake is thinking:

"The task is healthy, therefore the task is safe to receive or stop traffic."

Those are different questions.

A health check answers something like:

1
"Can this target answer my health endpoint?"

A graceful deployment needs to answer:

1
2
"Can this target stop receiving new work
without interrupting the work it is already processing?"

That second problem involves several moving parts:

1
2
3
4
5
6
7
8
9
10
11
ECS
+
Application Load Balancer
+
Target Group
+
Container lifecycle
+
Application shutdown
+
Long-running requests

This article explains how those pieces interact and how to build deployments that do not create a short window of 5xx errors during task replacement.

The Incident

Imagine this architecture:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
                 Internet
                    |
                    v
             +-------------+
             |     ALB     |
             +------+------+
                    |
             Target Group
          +---------+---------+
          |         |         |
          v         v         v
       ECS Task  ECS Task  ECS Task
          |         |         |
          +---------+---------+
                    |
                    v
               Spring Boot

A deployment begins.

ECS starts new tasks:

1
2
Old tasks:  A B C
New tasks:  D E F

Everything looks fine.

Then ECS starts stopping the old tasks.

What happens to a request that is already running on Task A?

That is where the interesting part begins.

Health Is Not the Same as Shutdown

Suppose the load balancer checks:

1
GET /actuator/health

and receives:

1
200 OK

The target is healthy.

But health checks do not automatically mean:

1
No request is currently running.

A task could be:

1
2
3
Healthy
+
processing a 12-second checkout request

Now the task is deregistered.

If the application is killed immediately, the in-flight request can fail.

The user sees:

1
502

The application may have been perfectly healthy five seconds earlier.

What Happens During Deregistration?

When a target is deregistered from an Application Load Balancer target group, the load balancer stops routing new requests to the target while allowing in-flight requests to drain.

AWS calls this connection draining.

The target enters a draining state while existing requests are finishing. citeturn522517search8

The load balancer's default deregistration delay is currently:

1
300 seconds

but that does not mean every service should use 300 seconds.

AWS specifically recommends tuning the value to the workload, and warns against very short delays for long-lived requests such as streaming or slow uploads. citeturn830715search1

The Complete Shutdown Sequence

A healthy deployment should conceptually look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
                ECS
                 |
                 v
         Deregister target
                 |
                 v
        ALB stops new traffic
                 |
                 v
        Existing requests drain
                 |
                 v
         Application receives
             SIGTERM
                 |
                 v
       Spring Boot stops accepting
            new work
                 |
                 v
       Existing work completes
                 |
                 v
          Process exits

The order matters.

If the process exits before traffic has drained, users can see errors.

ECS Uses SIGTERM Before SIGKILL

For a Linux container, ECS sends a stop signal to the container and, if it does not exit within the configured stop timeout, ECS forcibly terminates it.

The default stop timeout is 30 seconds for ECS container stopping behavior, and the timeout can be configured. citeturn830715search6turn830715search1

The important lifecycle is:

1
2
3
4
5
6
7
8
SIGTERM
   |
   v
Application should shut down gracefully
   |
   | if it does not exit
   v
SIGKILL

Therefore:

Your application needs to know what to do when SIGTERM arrives.

Spring Boot Should Shut Down Gracefully

For a Spring Boot service, enable graceful shutdown:

1
2
3
4
5
6
server:
  shutdown: graceful

spring:
  lifecycle:
    timeout-per-shutdown-phase: 30s

The exact timeout should match the maximum time you allow the application to finish its in-flight work.

The goal is:

1
2
3
4
5
6
7
8
9
10
11
12
13
SIGTERM
   |
   v
Stop accepting new requests
   |
   v
Finish in-flight requests
   |
   v
Close resources
   |
   v
Exit

This is much safer than:

1
2
3
4
SIGTERM
   |
   v
Immediate process termination

Why You Can Still Get 502s

Even with graceful shutdown configured, you can still have problems if these timeouts are inconsistent.

Consider:

1
2
ALB deregistration delay = 5 sec
Spring shutdown timeout  = 30 sec

Suppose an existing request needs 12 seconds.

The ALB may stop waiting before the application finishes the request.

Or consider:

1
2
3
ALB deregistration delay = 30 sec
ECS stop timeout         = 10 sec
Spring shutdown timeout  = 30 sec

The application may be forcibly killed after 10 seconds.

The system therefore has a timeout hierarchy.

A useful rule is:

1
2
3
4
5
Traffic drain window
        >=
Application graceful shutdown budget
        >=
Expected in-flight request completion

The exact values depend on the workload.

The Hidden Problem: Keep-Alive Connections

HTTP clients frequently reuse connections.

That means:

1
2
3
4
5
6
Client
  |
  +---- TCP connection ----> ALB
                              |
                              v
                           ECS task

When a target is being removed, the load balancer needs to handle these existing connections carefully.

AWS notes that ECS monitors the load balancer and waits for the target to reach an unused state before stopping the task in the relevant draining workflow. citeturn830715search1

This is why aggressive shutdown settings can create transient errors even when your application code itself is correct.

Health Checks Need to Represent Readiness

A bad health check is:

1
2
3
GET /health

always returns 200

A better health check represents whether the application should receive traffic.

For example:

1
2
3
4
Application started
Database reachable
Critical dependencies initialized
Application ready

Then the ALB can use a meaningful endpoint.

Be careful, though:

A readiness check should not become so expensive that the health checker itself creates load.

A health endpoint should be lightweight and deterministic.

Deployment Configuration Matters

For an ECS rolling deployment, two settings are especially important:

1
2
minimumHealthyPercent
maximumPercent

minimumHealthyPercent controls how many tasks must remain healthy during deployment.

maximumPercent controls how many tasks can exist during deployment relative to the desired count.

AWS documents the default values for a replica service as:

1
2
minimumHealthyPercent = 100%
maximumPercent        = 200%

for the rolling update deployment type, subject to service configuration and launch/deployment strategy. citeturn830715search0turn830715search3

For a service with:

1
desiredCount = 4

a common rolling deployment configuration can therefore allow:

1
2
3
4 old
+
4 new

during the transition, assuming the cluster has capacity.

This is useful because the new tasks can become healthy before the old tasks are removed.

Deployment Circuit Breaker

A deployment should not continue indefinitely when the new revision cannot become healthy.

ECS supports a deployment circuit breaker that can mark a deployment as failed if the service cannot reach a steady state.

With rollback enabled, ECS can roll back to the last successfully completed deployment. citeturn830715search3

Conceptually:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Old version
     |
     v
Deploy new version
     |
     v
Health checks fail
     |
     v
Deployment fails
     |
     v
Rollback
     |
     v
Known-good version

This is much safer than discovering after five minutes that every new task is unhealthy.

A Practical ECS Deployment Timeline

Suppose:

1
Desired tasks = 4

Before deployment:

1
A B C D

During deployment:

1
2
3
A B C D
    +
E F G H

New tasks become healthy:

1
E F G H = healthy

Then ECS can gradually remove:

1
2
3
4
A
B
C
D

The key is that traffic should be shifted away from an old task before that task is actually terminated.

What Causes 502 Specifically?

Application Load Balancers can return 502 responses for several target-side conditions.

Common examples include:

1
2
3
4
Target closed connection unexpectedly
TLS handshake problems
Invalid target response
Deregistration/termination during an active request

ALB troubleshooting documentation also distinguishes 502, 503, and 504 conditions.

For example:

1
2
3
502 -> target-side response/connection problem
503 -> no usable targets
504 -> target did not respond within the relevant timeout

The exact cause should be confirmed from ALB logs and CloudWatch metrics rather than inferred from the status code alone. citeturn522517search10

The Metrics I Watch First

When troubleshooting deployment-related 502s, inspect these together.

ALB

1
2
3
4
5
6
HTTPCode_ELB_5XX_Count
HTTPCode_Target_5XX_Count
TargetConnectionErrorCount
TargetResponseTime
HealthyHostCount
UnHealthyHostCount

ECS

1
2
3
4
5
RunningTaskCount
DesiredTaskCount
Deployment status
Task stop/start events
Task health

Application

1
2
3
4
5
6
HTTP request count
5xx rate
request duration
shutdown logs
active requests
database connection usage

The most useful question is:

Did the ALB lose its target while the application still believed it was serving traffic?

That can reveal a lifecycle mismatch very quickly.

Logging the Shutdown Lifecycle

A production Spring Boot service should make shutdown visible.

Useful log events include:

1
2
3
4
5
6
Application shutdown requested
Stopping server
Waiting for in-flight requests
Closing datasource
Closing message consumers
Application shutdown complete

Without these logs, deployment failures become much harder to reason about.

A deployment should produce an observable sequence such as:

1
2
3
4
5
15:00:01 new task healthy
15:00:02 old task begins draining
15:00:02 old task receives SIGTERM
15:00:03 request count decreases
15:00:04 old task exits

This is far more useful than simply seeing:

1
Task stopped

Graceful Shutdown Is Not Only About HTTP

Your application may also have:

1
2
3
4
5
6
7
Kafka consumers
SQS consumers
scheduled jobs
background executors
database transactions
file processing
outbound HTTP calls

When shutdown begins, these components need a defined lifecycle.

For example:

1
2
3
4
5
6
7
8
9
10
11
12
13
SIGTERM
   |
   +--> stop accepting HTTP requests
   |
   +--> stop polling new messages
   |
   +--> finish active message processing
   |
   +--> finish/commit required DB work
   |
   +--> close resources
   |
   +--> exit

Otherwise the HTTP layer may be graceful while a background worker is killed halfway through a business operation.

A Useful Timeout Hierarchy

A service with load balancer + application + database might look like:

1
2
3
4
5
6
7
8
9
Client timeout
        >
ALB request budget
        >
Application request timeout
        >
Downstream timeout
        >
Database query timeout

During shutdown, you also need:

1
2
3
4
5
Deregistration delay
        >=
Expected request drain time
        >=
Graceful shutdown budget

The exact hierarchy depends on the architecture.

The key idea is:

Infrastructure timeouts should not terminate work before the application has had a reasonable opportunity to finish it.

Long-Lived Requests Need Special Treatment

Not every HTTP service has requests that finish in 100 ms.

You may have:

1
2
3
4
5
6
File uploads
Streaming responses
Report generation
Server-sent events
Long polling
WebSocket traffic

For these workloads, blindly setting:

1
deregistration_delay.timeout_seconds = 5

can be dangerous.

AWS specifically warns against very short deregistration delays for long-lived requests. citeturn830715search1

The right value should come from the workload.

A Better Spring Boot ECS Configuration

A practical starting point:

1
2
3
4
5
6
server:
  shutdown: graceful

spring:
  lifecycle:
    timeout-per-shutdown-phase: 30s

Then align the infrastructure:

1
2
3
ALB deregistration delay
        >=
application shutdown budget

and ensure the ECS stop timeout gives the application enough time to actually finish that shutdown phase.

Do not blindly copy 30s everywhere.

Measure:

1
2
3
4
P95 request duration
P99 request duration
Maximum expected request duration
Shutdown cleanup duration

Then choose the values deliberately.

Testing Zero-Downtime Deployments

A deployment is not zero-downtime because nobody saw an error in one manual test.

Test continuously.

For example:

1
2
3
4
5
while true; do
  curl -s -o /dev/null \
       -w "%{http_code}\n" \
       https://api.example.com/orders
done

Then deploy a new ECS revision.

Collect:

1
2
3
4
5
HTTP 2xx
HTTP 5xx
latency
target health
task transitions

A good deployment should show:

1
2xx 2xx 2xx 2xx 2xx ...

not:

1
2xx 2xx 502 502 502 2xx ...

Failure Scenarios Worth Testing

Do not test only the happy path.

Test:

1
2
3
4
5
6
7
8
9
10
1. Normal rolling deployment
2. Deployment while traffic is high
3. Long-running request during deployment
4. Slow database request during shutdown
5. Task killed during in-flight request
6. New task passes health check slowly
7. New revision never becomes healthy
8. Deployment rollback
9. Scale-out during deployment
10. Scale-in during deployment

These tests reveal whether your lifecycle settings actually work together.

Common Anti-Patterns

Anti-pattern 1: Kill the container immediately

1
2
3
4
5
6
7
Task stopped
   |
   v
Process killed
   |
   v
Active requests fail

Fix:

1
2
3
Drain traffic
+
graceful shutdown

Anti-pattern 2: Make ALB deregistration delay tiny

This may make deployments faster, but it can terminate long-running requests prematurely.

Fast deployment is not the same as safe deployment.

Anti-pattern 3: Health check only tests process liveness

A process can be alive but not ready for production traffic.

The health check should represent actual readiness for the workload.

Anti-pattern 4: Ignore rollback

If new tasks cannot reach a stable state, continuing the deployment can turn a small release problem into a full outage.

Use deployment failure detection and rollback where appropriate. citeturn830715search3

Production Deployment Checklist

Before calling an ECS service "zero-downtime ready":

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[ ] ALB health check is meaningful
[ ] New tasks become healthy before old tasks stop
[ ] minimumHealthyPercent is intentional
[ ] maximumPercent is intentional
[ ] Deployment circuit breaker is enabled where appropriate
[ ] Automatic rollback is configured where appropriate
[ ] ALB deregistration delay matches request duration
[ ] Spring Boot graceful shutdown is enabled
[ ] ECS stop timeout is long enough
[ ] Shutdown lifecycle is logged
[ ] Background consumers stop accepting new work
[ ] In-flight work can finish safely
[ ] Long-running endpoints were considered
[ ] Continuous deployment testing has been performed
[ ] ALB 5xx metrics are monitored

The Bigger Lesson

An ECS task being:

1
RUNNING

does not mean:

1
Safe to kill.

A target being:

1
HEALTHY

does not mean:

1
No in-flight requests.

And a deployment being:

1
SUCCESSFUL

does not automatically mean:

1
No requests were interrupted.

A robust deployment treats application lifecycle and infrastructure lifecycle as one system:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
                ECS
                 |
                 v
         Start new tasks
                 |
                 v
         ALB health checks
                 |
                 v
        New targets become healthy
                 |
                 v
      Deregister old targets
                 |
                 v
          Drain connections
                 |
                 v
       Application receives SIGTERM
                 |
                 v
       Finish in-flight requests
                 |
                 v
           Process exits

That is what zero-downtime deployment really means.

Not:

"The new container started."

But:

"Traffic moved safely from the old version to the new version without interrupting active work."

Final Takeaways

1
2
3
4
5
6
7
8
9
10
1. Health checks and graceful shutdown solve different problems.
2. ALB deregistration delay controls traffic draining.
3. ECS uses SIGTERM before forceful termination.
4. Spring Boot should be configured for graceful shutdown.
5. Shutdown and load-balancer timeouts must be aligned.
6. Long-running requests require special treatment.
7. minimumHealthyPercent and maximumPercent shape deployment safety.
8. Deployment circuit breakers reduce prolonged bad releases.
9. Monitor ALB 5xx metrics during deployments.
10. Test the deployment while real traffic is flowing.

The most useful question during a deployment incident is:

"Who stopped this target, and was the request still running when it happened?"

Once you trace that lifecycle, many mysterious 502s stop being mysterious.

# amazon-ecs # auto-scaling # java

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

The ECS Service Was Healthy. Users Still Got 502s.

Ashish kumar jha

The ECS Service Was Healthy. Users Still Got 502s.

The Incident

Health Is Not the Same as Shutdown

What Happens During Deregistration?

The Complete Shutdown Sequence

ECS Uses SIGTERM Before SIGKILL

Spring Boot Should Shut Down Gracefully

Why You Can Still Get 502s

The Hidden Problem: Keep-Alive Connections

Health Checks Need to Represent Readiness

Deployment Configuration Matters

Deployment Circuit Breaker

A Practical ECS Deployment Timeline

What Causes 502 Specifically?

The Metrics I Watch First

ALB

ECS

Application

Logging the Shutdown Lifecycle

Graceful Shutdown Is Not Only About HTTP

A Useful Timeout Hierarchy

Long-Lived Requests Need Special Treatment

A Better Spring Boot ECS Configuration

Testing Zero-Downtime Deployments

Failure Scenarios Worth Testing

Common Anti-Patterns

Anti-pattern 1: Kill the container immediately

Anti-pattern 2: Make ALB deregistration delay tiny

Anti-pattern 3: Health check only tests process liveness

Anti-pattern 4: Ignore rollback

Production Deployment Checklist

The Bigger Lesson

Final Takeaways