AWS Builder Center

Builder Center

When Kubernetes Health Checks Lie: Readiness, Liveness, Startup Probes and Real Service Availability

Why healthy Pods can still deliver broken services, and how to engineer health checks that reflect production reality.

Pradeep Kandepaneni

DevOps Cloud Platform & Site Reliability Engineer | AWS Certified Solutions Architect – Professional

When Kubernetes Health Checks Lie: Readiness, Liveness, Startup Probes and Real Service Availability

A Kubernetes Deployment reports all replicas available. Every Pod is Running. Readiness probes are passing. The cluster dashboard is green.
Customers are still receiving HTTP 503 responses.
This is not necessarily a Kubernetes bug. It is a gap between what Kubernetes considers healthy and what users experience as available.
Kubernetes health probes answer specific operational questions. They do not establish end-to-end service reliability.
The problem begins when platform teams treat those answers as interchangeable.
A liveness probe cannot prove that an application can serve requests. A readiness probe cannot prove that an entire business transaction succeeds. A startup probe cannot guarantee that an application has completed every dependency initialization.
These mechanisms are valuable precisely because their responsibilities are narrow.
The engineering challenge is to preserve those boundaries while building a stronger definition of availability around them.

1. Three probes, three different decisions

Kubernetes provides three probe types:
Probe
Question
Kubernetes action
Startup
Has the application completed the startup condition?
Suppress liveness and readiness evaluation until startup succeeds; restart the container if the startup probe repeatedly fails
Readiness
Should this Pod receive traffic through Kubernetes Service routing?
Mark the Pod unready and remove it from eligible Service endpoints
Liveness
Is this container in a condition that requires restarting?
Restart the container after the configured failure threshold
The distinction matters.
Readiness is a traffic-routing decision.
Liveness is a process-recovery decision.
Startup is a startup-phase protection mechanism.
They should not share one endpoint merely because exposing three endpoints requires additional engineering.
Consider an application that loses access to a database for 20 seconds.
Should Kubernetes restart it?
Not necessarily.
Should Kubernetes continue routing requests to it?
That depends on whether it can still serve useful traffic.
Should the startup probe fail?
If the application has already started successfully, startup probing is no longer the relevant mechanism.
The same dependency failure can require different actions depending on application state and operational intent.

2. Why HTTP 200 does not mean healthy

A common health endpoint looks like this:
Plain text
@app.get("/health")
def health():
    return {"status": "ok"}
The endpoint responds successfully whenever the HTTP server is running.
That proves a limited fact: the process can respond to that particular request.
It does not prove:
  • The application can authenticate users.
  • Its database queries are succeeding.
  • Its message producer can publish events.
  • Its worker pool has available capacity.
  • Its critical downstream dependencies are reachable.
  • Its business transactions complete within acceptable latency.
A Pod can pass this probe while the service is effectively unavailable.
The opposite is also possible.
A health endpoint may fail because an optional dependency is unavailable even though the application continues serving its most important functionality.
A useful health check must therefore answer a precise question rather than attempt to summarize the entire system with one Boolean.

3. Readiness should represent traffic eligibility

Readiness is commonly the most consequential application probe because it influences whether traffic is sent to a Pod through Kubernetes Service routing.
Consider a checkout service.
It requires local configuration, a functioning request handler, and an initialized connection pool.
Those conditions may be appropriate readiness prerequisites.
But suppose its analytics service is unavailable.
If analytics is asynchronous and noncritical, marking checkout unready could be the wrong response.
The checkout service may still process orders successfully.
A dependency-aware readiness policy should distinguish critical and optional dependencies.
Plain text
checkout-api
    |
    +--> HTTP handler          Critical
    |
    +--> Database pool         Critical
    |
    +--> Payment provider      Context-dependent
    |
    +--> Analytics publisher   Optional
A practical readiness contract might be:
Ready when the application can accept its intended class of traffic without immediately failing due to a local condition.
That is deliberately narrower than guaranteeing that every downstream operation will succeed.
For example, readiness might return HTTP 503 when the application has exhausted its request-processing capacity or is still initializing required resources.
It should not automatically return 503 whenever any dependency reports an error.
The distinction prevents local failures from becoming cluster-wide routing events.

4. The dependency amplification trap

Suppose a service has 20 replicas.
Every replica uses the same database.
The database becomes temporarily unavailable.
Each readiness probe checks the database directly.
Plain text
Database unavailable
        |
        v
20 readiness probes fail
        |
        v
20 Pods become unready
        |
        v
No eligible Service endpoints
        |
        v
All incoming requests fail
The database failure was already serious.
The readiness policy can make recovery harder by removing all application endpoints simultaneously.
Now imagine the database recovers.
Every replica attempts to reconnect at approximately the same time.
The database experiences a connection storm.
Readiness begins flapping.
The service may oscillate between zero and many eligible endpoints.
This is why dependency checks require careful design.
Consider using bounded timeouts, cached local dependency state, connection-pool health, and application-level circuit breakers rather than running an expensive dependency transaction on every probe.
But cached health introduces another tradeoff: it can be stale.
A readiness check should be cheap, predictable, and locally actionable. It should not become a distributed transaction.

5. Liveness should detect irrecoverable local failure

Liveness is often overused.
Consider:
Plain text
livenessProbe:
  httpGet:
    path: /health
    port: 8080
  periodSeconds: 10
  failureThreshold: 3
If  /health  performs a database query, a database outage may cause the liveness probe to fail.
Kubernetes then restarts the container.
The replacement container starts, attempts to connect to the same unavailable database, and fails again.
Plain text
Database outage
      |
      v
Liveness failure
      |
      v
Container restart
      |
      v
Cold startup
      |
      v
Database still unavailable
      |
      v
Another restart
Instead of recovering the database dependency, Kubernetes creates a restart loop.
This can also destroy useful diagnostic state, increase connection attempts, and consume startup resources.
Liveness should generally detect conditions where restarting the local process is a reasonable recovery action.
Examples might include an unrecoverable application deadlock or a critical internal execution loop that has stopped making progress.
Even then, the probe must be designed carefully.
An HTTP server may continue answering  /live  while background workers are deadlocked.
Conversely, a temporarily saturated request handler may fail a probe even though the application would recover without a restart.
Liveness is a recovery mechanism, not an availability metric.

6. Startup probes prevent premature recovery actions

Some applications require significant startup time.
They may load large configuration sets, initialize caches, perform JVM warm-up, or establish connections before becoming operational.
Without a startup probe, aggressive liveness settings can repeatedly terminate an application before initialization completes.
A startup probe creates a protected startup window.
Plain text
startupProbe:
  httpGet:
    path: /startup
    port: 8080
  periodSeconds: 5
  failureThreshold: 24

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  periodSeconds: 5
  failureThreshold: 2

livenessProbe:
  httpGet:
    path: /live
    port: 8080
  periodSeconds: 10
  failureThreshold: 3
The startup configuration allows approximately two minutes for the startup condition to succeed before the failure threshold triggers a restart, subject to probe scheduling and execution timing.
While the startup probe has not succeeded, Kubernetes does not execute the configured liveness and readiness probes.
Once startup succeeds, those probes take over.
This is preferable to giving liveness an excessively large initial delay because it allows startup completion to be detected dynamically.
However, startup probes should not conceal indefinitely hung initialization.
Their failure budget must reflect a realistic upper bound for startup under expected operating conditions.

7. A production-oriented probe design

For a typical HTTP application, the endpoints can have distinct responsibilities.
Plain text
/live
   |
   +--> Is the local process capable
        of continuing execution?

/ready
   |
   +--> Is the application initialized?
   +--> Can it accept intended traffic?
   +--> Is required local capacity available?

/startup
   |
   +--> Has required initialization completed?
A corresponding Deployment configuration could be:
Plain text
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
spec:
  replicas: 4
  selector:
    matchLabels:
      app: checkout-api
  template:
    metadata:
      labels:
        app: checkout-api
    spec:
      terminationGracePeriodSeconds: 45
      containers:
        - name: checkout-api
          image: example.com/checkout-api:v2
          ports:
            - containerPort: 8080

          startupProbe:
            httpGet:
              path: /startup
              port: 8080
            periodSeconds: 5
            failureThreshold: 24

          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 2

          livenessProbe:
            httpGet:
              path: /live
              port: 8080
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3
This is an illustrative baseline, not a universal production configuration.
The endpoint implementations, probe intervals, failure thresholds, and termination behavior must be validated against actual application behavior.
In particular, do not assume that passing  /ready  means the application has adequate throughput capacity under peak load.
Readiness determines eligibility, not performance guarantees.

8. Probe timing is part of system behavior

Probe configuration introduces detection latency.
For example:
Plain text
Readiness interval: 5 seconds
Failure threshold: 2
Timeout: 2 seconds
A failure will not necessarily be detected immediately.
The actual transition depends on probe scheduling, response timing, and the configured threshold.
A rough planning model is:
Plain text
Detection latency
    ≈
probe scheduling delay
+
time needed to accumulate failures
That matters during a rollout.
A Pod may begin returning application errors before Kubernetes marks it unready.
The load balancer may continue sending requests during the detection interval.
Likewise, recovery requires successful readiness evaluation before traffic eligibility is restored.
Probe settings therefore trade off:
  • Failure detection speed
  • Sensitivity to transient failures
  • Probe overhead
  • Readiness flapping
  • Recovery latency
Very aggressive probes can turn short-lived resource contention into unnecessary traffic removal.
Very conservative probes can leave unhealthy Pods eligible for too long.
The correct settings should be derived from failure tolerance and measured application behavior.

9. Kubernetes readiness is not the same as load-balancer readiness

On Amazon EKS, another important boundary exists between Kubernetes endpoint readiness and external load-balancer target health.
For example:
Plain text
Client
   |
   v
AWS Application Load Balancer
   |
   v
Target Group
   |
   v
Kubernetes Pod
A Pod may become Kubernetes Ready before the ALB has completed its own target health evaluation.
The reverse transition also has propagation timing.
During rolling deployments, this distinction can matter.
A new Pod may be eligible for Kubernetes routing while the external load balancer has not yet established that it can receive traffic.
The AWS Load Balancer Controller supports Pod readiness gates for supported configurations, allowing load-balancer target health to participate in Kubernetes Pod readiness during rollouts.
But readiness gates require appropriate controller configuration and should not be assumed to exist automatically in every EKS deployment.
The important architectural principle is that each traffic-routing layer has its own health state.
Plain text
Application readiness
         |
         v
Kubernetes Pod readiness
         |
         v
EndpointSlice eligibility
         |
         v
Load-balancer target health
         |
         v
User request success
These states are related.
They are not identical.

10. Termination behavior is another health boundary

A Pod becoming unready does not instantly guarantee that no traffic can reach it.
Endpoint updates, load-balancer deregistration, existing connections, and application shutdown all have their own timing.
Consider a rolling deployment.
Plain text
Old Pod receives termination signal
             |
             v
Kubernetes begins endpoint updates
             |
             v
External routing state converges
             |
             v
Existing requests finish
             |
             v
Container exits
If the application exits immediately after receiving SIGTERM, in-flight requests may fail.
A robust shutdown strategy should coordinate application-level graceful termination with the platform's termination grace period and external routing behavior.
This often requires:
  • Stop accepting new work when shutdown begins.
  • Allow active requests to complete within a bounded interval.
  • Stop or drain background consumers appropriately.
  • Release resources safely.
  • Exit before the configured termination deadline.
For asynchronous workers, termination also needs to account for message acknowledgement semantics.
Killing a worker during message processing can create redelivery or partial side effects.
Health and lifecycle engineering cannot be separated completely.

11. Probe success does not establish an SLO

This is the central distinction.
Kubernetes might report:
Plain text
Ready replicas: 10/10
The business service might report:
Plain text
Successful transactions: 92%
Both observations can be accurate.
The Pods are eligible to receive traffic.
The application is failing 8% of transactions.
An SLO must be based on service behavior rather than inferred from Kubernetes object state.
For a request-driven service, useful SLIs might include:
Plain text
Availability:
successful eligible requests / total eligible requests

Latency:
proportion of eligible requests completed
within the defined latency threshold

Correctness:
proportion of transactions producing
the expected result
These definitions need precise eligibility rules and business context.
For example, an intentional HTTP 4xx response caused by invalid client input should not automatically count as service unavailability.
A payment authorization declined by a bank may be a valid business result rather than an application failure.
A health endpoint cannot encode all of these distinctions.
That is why real availability needs application telemetry, not just Kubernetes probes.

12. Build an availability model across layers

A more complete model separates four levels.
Layer
What it establishes
What it cannot establish
Process
Container is running and responsive to specific checks
Business correctness
Pod
Kubernetes considers the workload eligible for routing
External reachability
Service path
Network and routing path can reach the application
Transaction success
User journey
A representative business operation succeeds
Every possible workflow
The monitoring architecture should reflect those boundaries.
Plain text
                User Journey
                     |
                     v
           Synthetic Transactions
                     |
                     v
              Application SLIs
                     |
                     v
             Service Routing
                     |
                     v
          Kubernetes Readiness
                     |
                     v
             Container Health
This is not a hierarchy where lower-layer success guarantees upper-layer success.
It is a dependency map for diagnosis.
When user-facing availability drops, engineers should be able to determine whether the failure originated in the application, routing layer, Pod lifecycle, or underlying infrastructure.

13. Test probes through fault injection

Health-check configurations should be tested against failure scenarios, not approved solely by code review.
A useful validation matrix includes:
Injected failure
Expected behavior
Slow application startup
Startup probe protects initialization
Local process deadlock
Liveness eventually triggers recovery if its endpoint can detect the condition
Temporary database outage
Avoid unnecessary restart storms
Optional analytics outage
Critical application traffic remains eligible
Request-handler saturation
Readiness behavior follows the defined capacity policy
Pod termination
In-flight work drains safely
External load-balancer target delay
Rollout avoids premature traffic exposure
Downstream business failure
Application SLI detects the failure even if probes pass
These tests expose a subtle but important property: probe behavior is part of the application's failure-management design.
A probe should have an expected response for every relevant failure class.
If engineers cannot explain why a particular failure should trigger restart, traffic removal, or neither, the health contract is underspecified.

14. The operational tradeoff

There is no universally perfect readiness or liveness endpoint.
A shallow readiness check may leave partially degraded Pods eligible for traffic.
An overly deep readiness check may cause a shared dependency outage to remove every Pod from routing.
A shallow liveness check may miss internal deadlocks.
An overly aggressive liveness check may create restart storms during recoverable conditions.
The objective is not maximum sensitivity.
It is selecting the right automated action for each failure mode.
That requires understanding:
Plain text
Failure detected
       |
       v
Can local restart fix it?
       |
       +--> Yes: consider liveness
       |
       +--> No
             |
             v
Can this Pod serve intended traffic?
             |
             +--> No: readiness failure
             |
             +--> Yes: remain ready
                       and expose degradation
                       through telemetry
Even this decision tree is simplified.
A shared dependency outage may require circuit breaking, graceful degradation, or traffic shaping rather than making every Pod unready.
The important point is to choose the action intentionally.

15. What senior platform teams should standardize

The strongest platform standard is not a mandatory  /health  endpoint copied into every application.
It is a documented health contract.
For each service, define:
Startup contract: What initialization must complete, and how long may it reasonably take?
Readiness contract: What local conditions make the Pod unsuitable for intended traffic?
Liveness contract: Which local failures justify restarting the process?
Termination contract: How does the application drain requests, consumers, and other work?
Availability contract: Which externally observable transactions define whether the service is actually working?
Then test those contracts during rollout, dependency failure, saturation, and recovery.
Platform teams can standardize instrumentation, probe configuration patterns, and deployment guardrails.
Application teams must still own the domain-specific meaning of availability.
A generic platform endpoint cannot know whether an order was placed correctly.

Conclusion: Green Pods are not the objective

Kubernetes health checks are necessary infrastructure controls.
They are not business availability guarantees.
Readiness controls traffic eligibility.
Liveness controls restart behavior.
Startup probes protect initialization.
External load balancers introduce additional routing health states.
SLOs measure whether users receive the service they were promised.
Treating all of these as one health signal creates false confidence and sometimes makes incidents worse.
The better engineering question is not:
"Are all Pods healthy?"
It is:
"Which failure conditions can Kubernetes recover automatically, which require traffic management, and which must be detected through user-facing reliability signals?"
A production-ready health strategy connects those answers without confusing them.
A green Kubernetes dashboard is useful evidence. It is not proof that the service works.


Authoritative references

  1.  Kubernetes: Configure Liveness, Readiness and Startup Probes 
  1.  Kubernetes: Pod Lifecycle 
  1.  Kubernetes: EndpointSlices 
  1.  AWS Load Balancer Controller: Pod Readiness Gate 
Technical note: The probe behaviors and Kubernetes examples are based on documented Kubernetes semantics. Application-specific thresholds and failure responses are illustrative and require validation in the target environment.
# amazon-eks# kubernetes# devops# reliability
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article