AWS Builder Center

Builder Center

The Database Was Healthy. The Application Ran Out of Connections.

A practical deep dive into database connection pool exhaustion in Spring Boot on AWS, covering HikariCP, Amazon RDS, RDS Proxy, timeouts, metrics, load testing, and production-safe tuning.

Ashish kumar jha

Software Devloper

The Database Was Healthy. The Application Ran Out of Connections.

Your Amazon RDS database can be healthy, CPU can be normal, storage can have plenty of space, and queries can still start failing.
The reason is often not the database itself.
It is the connection pool between the application and the database.
This failure is especially confusing in Spring Boot applications because the first visible symptom is usually something like:
1
2
3
Unable to acquire JDBC Connection
Connection is not available, request timed out
HikariPool - Connection is not available
The database may look completely normal from the RDS console.
That is the trap.
The application might have already exhausted its own pool of JDBC connections while the database is still capable of accepting more work.
This article explains how that happens, why simply increasing the pool size can make the incident worse, where Amazon RDS Proxy fits, and how to design and test a production-safe connection strategy.

The Incident

Imagine an API running as an ECS service.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Internet
|
v
+-------------+
| ALB |
+------+------+
|
+---------+---------+
| | |
v v v
+-------+ +-------+ +-------+
| ECS 1 | | ECS 2 | | ECS 3 |
| API | | API | | API |
+---+---+ +---+---+ +---+---+
| | |
+---------+---------+
|
v
+-----------+
| RDS |
+-----------+
Suppose each ECS task has:
1
2
3
4
spring:
datasource:
hikari:
maximum-pool-size: 20
With 10 application tasks, that can mean:
1
10 tasks × 20 connections = 200 possible DB connections
Now imagine a traffic spike.
Requests increase.
Some database queries become slower.
Connections stay occupied longer.
The application keeps accepting requests.
More requests wait for connections.
Eventually the pool cannot provide a connection before the configured timeout.
The API starts returning 500s or latency increases dramatically.
Meanwhile:
1
2
3
4
5
RDS CPU -> normal
RDS storage -> normal
Application CPU -> maybe normal
Application -> connection timeout
Users -> errors
This is a classic example of a bottleneck existing between two otherwise healthy components.

First Principle: A Connection Pool Is a Capacity Limit

A JDBC connection is an expensive resource.
When a request needs the database, the application generally:
1
2
3
4
5
6
7
8
9
10
11
12
13
Request
|
v
Borrow connection
|
v
Execute SQL
|
v
Commit / rollback
|
v
Return connection
The important point is that a connection is not unlimited.
If a pool contains 20 connections, at most roughly 20 database operations can hold those connections concurrently from that application instance.
The remaining requests wait.
Conceptually:
1
2
3
4
5
6
7
8
9
10
HikariCP
+-------------+
Request --->| C1 |
Request --->| C2 |
Request --->| C3 |
Request --->| ... |
Request --->| C20 |
Request --->| |
Request --->| WAITING |
+-------------+
A pool is therefore not just an implementation detail.
It is a concurrency control mechanism.
That means a larger pool is not automatically better.

Why Increasing maximumPoolSize Can Make Things Worse

A common reaction to connection timeouts is:
1
2
3
4
spring:
datasource:
hikari:
maximum-pool-size: 100
The error disappears temporarily.
Then the database becomes slower.
Why?
Because you changed the amount of work the application is allowed to send to the database.
For example:
1
2
3
4
5
6
7
Before:

10 ECS tasks × 20 = 200 connections

After:

10 ECS tasks × 100 = 1000 connections
You did not magically create a database capable of processing 1000 concurrent queries.
You simply allowed the application to create much more pressure.
That can increase:
  • CPU contention
  • memory consumption
  • lock contention
  • context switching
  • query queueing
  • database connection overhead
  • latency
  • timeout frequency
The right question is not:
"How many connections can I create?"
It is:
"How many concurrent database operations can the database process efficiently for this workload?"

The Most Important Number Is Not the Pool Size

Suppose your database can sustainably support approximately 300 useful concurrent connections for the workload.
Your application has:
1
15 ECS tasks
A naive configuration might use:
1
maximumPoolSize = 50
That creates:
1
15 × 50 = 750 possible connections
That can be far above the useful database concurrency.
A safer starting point is to reason backwards:
1
2
3
4
5
6
7
8
9
10
Database capacity
|
v
Workload concurrency
|
v
Total connection budget
|
v
Connections per application instance
For example:
1
2
3
4
Total desired connection budget = 180
Application instances = 15

Approximate per-instance pool = 12
This is only a starting point.
There is no universal formula that says every Spring Boot service should use 10, 20, or 50 connections.
Pool sizing depends on:
  • query latency
  • database CPU
  • database engine
  • transaction duration
  • number of instances
  • workload mix
  • read/write ratio
  • lock contention
  • connection establishment cost
  • peak concurrency
The pool should be sized from measurement, not copied from another service.

What Actually Causes Pool Exhaustion?

Connection pool exhaustion usually comes from one of a few patterns.

1. Queries are slow

Suppose a query normally takes:
1
20 ms
and suddenly takes:
1
800 ms
The same number of requests now occupies connections for much longer.
The pool effectively becomes smaller.
Example:
1
2
3
4
5
6
7
8
9
10
20 connections
20 ms average query time

=> high turnover

20 connections
800 ms average query time

=> much slower turnover
=> requests start waiting
This is why connection-pool incidents often appear together with a database query regression.

2. Transactions are too long

A connection may remain occupied for the duration of a transaction.
Consider:
1
2
3
4
5
6
7
8
9
10
11
@Transactional
public void processOrder(OrderRequest request) {

Order order = repository.save(...);

callExternalPaymentService();

updateInventory();

publishEvent();
}
The external payment call is unrelated to the database, but the transaction may still keep database resources occupied while the HTTP call runs.
That can be disastrous under concurrency.
A better design is often:
1
2
3
4
5
6
7
8
9
10
Validate
|
v
Short DB transaction
|
v
Commit
|
v
External call / event
Transaction boundaries should be designed deliberately.
Do not hold database resources while waiting on an unrelated network service unless there is a strong consistency reason.

3. Connections are leaked

A connection leak happens when code borrows a connection and does not return it correctly.
JPA and Spring-managed repositories normally handle this for you.
Problems become more likely with:
  • manual JDBC code
  • custom data-source usage
  • poorly managed streams
  • long-lived transactions
  • incorrect async patterns
If you use manual JDBC, use try-with-resources:
1
2
3
4
5
6
try (Connection connection = dataSource.getConnection();
PreparedStatement statement = connection.prepareStatement(sql);
ResultSet resultSet = statement.executeQuery()) {

// process result
}
The important rule is:
Every borrowed connection must have a reliable lifecycle.

HikariCP in Spring Boot

Spring Boot commonly uses HikariCP for JDBC connection pooling.
A basic production configuration might look like:
1
2
3
4
5
6
7
8
9
10
11
12
13
spring:
datasource:
url: jdbc:postgresql://${DB_HOST}:5432/orders
username: ${DB_USERNAME}
password: ${DB_PASSWORD}

hikari:
maximum-pool-size: 20
minimum-idle: 5
connection-timeout: 3000
validation-timeout: 1000
idle-timeout: 600000
max-lifetime: 1800000
The exact values are workload-specific.
The important concepts are:

maximum-pool-size

Maximum number of connections in the application pool.
This is your main concurrency control.

minimum-idle

How many idle connections the pool attempts to maintain.
Increasing this can make startup faster for workloads with steady traffic, but it also means more resources remain allocated.

connection-timeout

How long the application waits for a connection from the pool.
This is not the same as the database query timeout.
For example:
1
2
3
4
5
Request
|
+--> wait for pool connection
|
+--> execute SQL
You have separate waiting phases.

max-lifetime

Controls how long a pooled connection is kept before being retired.
The value should be selected with the database/network environment in mind so the application does not keep connections beyond infrastructure limits.

Connection Timeout vs Query Timeout

These are often confused.
Imagine:
1
2
3
4
5
6
7
8
9
10
11
12
13
Request arrives
|
v
Waiting for connection 2 seconds
|
v
Connection acquired
|
v
SQL query 10 seconds
|
v
Response
A pool timeout cannot fix the slow query.
Likewise, reducing the query timeout does not increase the number of connections available.
You need to reason about them independently:
1
2
3
4
5
6
7
8
9
Pool acquisition timeout
+
Connection establishment timeout
+
Query / statement timeout
+
Transaction timeout
+
HTTP request timeout
These timeouts should be designed as a hierarchy.
A downstream operation should not be allowed to consume the entire timeout budget of the caller indefinitely.

A Better Failure Model

A useful mental model is:
1
2
3
4
5
6
7
8
9
10
11
12
13
Incoming requests
|
v
Application concurrency
|
v
Connection pool
|
v
Database concurrency
|
v
Storage / CPU / locks / IO
Every layer has a capacity.
If the database can only process a certain amount of concurrent work efficiently, letting the application queue an unlimited number of database operations does not improve throughput.
It usually increases latency.

Where Amazon RDS Proxy Fits

Amazon RDS Proxy is useful when connection management itself becomes a scalability problem.
Conceptually:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
+----------------+
Clients ------------>| Spring Boot |
+--------+-------+
|
v
+-------------+
| RDS Proxy |
| connection |
| pooling |
+------+------+
|
v
+-------------+
| RDS |
+-------------+
RDS Proxy can pool and reuse connections to the database.
That is especially useful for workloads with:
  • many short-lived connections
  • bursty traffic
  • serverless workloads
  • unpredictable connection spikes
  • many application clients
It can help prevent the database from being overwhelmed by connection churn.
But RDS Proxy is not a replacement for application-level pool tuning.
It does not magically fix:
  • slow SQL
  • missing indexes
  • long transactions
  • lock contention
  • connection leaks
  • bad concurrency design
It also has its own behaviors such as connection pinning that can reduce multiplexing efficiency.
So the right question is:
"Do I have a connection-management problem that RDS Proxy solves?"
Not:
"My database is slow. Should I add RDS Proxy?"

A Practical Architecture

For a typical ECS + Spring Boot + RDS workload:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
Internet
|
v
+-------------+
| ALB |
+------+------+
|
+------------+-------------+
| | |
v v v
ECS Task ECS Task ECS Task
Hikari Hikari Hikari
| | |
+-------------+-------------+
|
v
+-----------+
| RDS Proxy |
+-----+-----+
|
v
+-------+
| RDS |
+-------+
Whether the proxy is appropriate depends on the workload.
For a smaller service with stable traffic, direct RDS connectivity with a carefully sized HikariCP pool may be enough.
For bursty workloads with many clients, RDS Proxy can be valuable.

Production Observability

Do not wait for users to report:
1
500 Internal Server Error
Monitor both application and database signals.

Application metrics

Watch:
1
2
3
4
5
6
Hikari active connections
Hikari idle connections
Hikari pending threads
Connection acquisition latency
Request latency
HTTP 5xx
One especially useful signal is:
1
Hikari pending threads
If this climbs during an incident, requests are waiting for pool capacity.

RDS metrics

Depending on the engine and workload, inspect:
1
2
3
4
5
6
7
DatabaseConnections
CPUUtilization
FreeableMemory
ReadLatency
WriteLatency
DiskQueueDepth
Database load / Performance Insights
The most important lesson is:
Never look at only one metric.
For example:
1
2
3
4
DatabaseConnections -> high
CPU -> normal
Application pending -> high
Query latency -> high
That tells a very different story from:
1
2
3
DatabaseConnections -> high
CPU -> 95%
Query latency -> high
In the second case, the database itself may be the primary bottleneck.

A Useful Calculation During an Incident

Suppose:
1
2
3
4
5
6
ECS tasks = 12
Maximum pool size = 25

Potential connections
= 12 × 25
= 300
Then a deployment scales the service to 24 tasks:
1
24 × 25 = 600
Nothing about the database changed.
But your potential connection demand doubled.
This is one reason horizontal scaling and database scaling are not independent decisions.
Scaling the application can increase database pressure.
That is an architectural coupling worth documenting.

Do Not Forget Retries

Retries can make pool exhaustion worse.
Imagine:
1
2
3
4
5
6
7
8
9
Request
|
+--> DB timeout
|
+--> retry
|
+--> DB timeout
|
+--> retry
If every application instance retries aggressively, you can create a feedback loop:
1
2
3
4
5
6
7
8
9
10
11
12
13
Database latency increases
|
v
Requests timeout
|
v
Clients retry
|
v
More DB requests
|
v
Database latency increases again
AWS recommends controlling retry calls with bounded retries, exponential backoff, and jitter, and avoiding retries for non-idempotent operations. citeturn687207search0
Retries should be part of capacity planning.

Load Testing the Pool

A configuration is not production-ready because the happy-path test passed.
Test the following:
1
2
3
4
5
6
7
8
1. Normal traffic
2. Peak traffic
3. Slow query
4. Database throttling
5. Connection failure
6. Application scale-out
7. Application scale-in
8. Retry storm
One very useful experiment is:
1
2
3
4
5
6
7
8
9
10
Start with:
maximumPoolSize = 10

Run load

Observe:
- throughput
- latency
- pool saturation
- database CPU
Then compare:
1
2
3
maximumPoolSize = 20
maximumPoolSize = 30
maximumPoolSize = 40
The goal is not the biggest number.
The goal is the point where:
1
2
3
Throughput stops improving
OR
Latency starts increasing sharply
That is often a better indicator of useful concurrency.

A Production-Safe Starting Strategy

For a Spring Boot service on AWS:
1
2
3
4
5
6
7
8
9
10
1. Measure database query latency.
2. Measure peak application concurrency.
3. Determine database connection capacity.
4. Set a bounded per-instance connection pool.
5. Multiply pool size by maximum application instances.
6. Verify the resulting total connection budget.
7. Configure reasonable acquisition/query/transaction timeouts.
8. Monitor pool saturation.
9. Test scale-out behavior.
10. Add RDS Proxy only when its connection-management benefits match the workload.
Do not start with:
1
"Let's set maximumPoolSize to 100."
Start with:
1
"How much database concurrency can this system actually sustain?"

The Incident Checklist

When users report:
1
2
API is timing out
Database looks healthy
Check in this order:
1
2
3
4
5
6
7
8
9
10
11
[ ] Hikari active connections
[ ] Hikari pending threads
[ ] Pool acquisition timeout
[ ] Query latency
[ ] Long transactions
[ ] Connection leaks
[ ] DB connection count
[ ] DB CPU / IO / locks
[ ] Recent deployment or scale-out
[ ] Retry volume
[ ] Number of application instances
A useful question is:
"Did the application become more concurrent without the database becoming more capable?"
That question catches many incidents quickly.

The Bigger Lesson

A database connection pool is a bounded resource, not just a configuration property.
The real architecture is:
1
2
3
4
5
6
7
8
9
10
11
12
13
Clients
|
v
API concurrency
|
v
Connection pool
|
v
Database concurrency
|
v
Database capacity
A reliable system keeps these limits aligned.
If the application can create more concurrent database work than the database can process efficiently, scaling the application can amplify the problem instead of fixing it.
And if the application pool is too small, requests can fail even while the database still has capacity.
The goal is not to maximize connections.
The goal is to maximize useful throughput while keeping latency and failure rates predictable.

Final Takeaways

1
2
3
4
5
6
7
8
9
10
1. Connection pool size is a concurrency limit.
2. More connections do not automatically mean more throughput.
3. Application scale multiplies total possible DB connections.
4. Slow queries and long transactions make pools exhaust faster.
5. Pool timeout and query timeout are different controls.
6. Retries can amplify connection pressure.
7. RDS Proxy helps with connection management, not bad SQL.
8. Measure pool saturation, not just database CPU.
9. Test scale-out and failure scenarios.
10. Size the pool from workload evidence, not from a copied number.
The database being "healthy" does not mean the application has enough database capacity.
Sometimes the database is fine.
The queue of requests waiting to use it is the actual incident.
# amazon-rds# java# spring-boot# reliability
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article