Rotating database credentials with AWS Secrets Manager: what the Cloud Quest lab taught me
Moving a database password into Secrets Manager is only half the fix. How rotation works, single vs alternating users, and what to alert on.
The Cloud Quest Security lab on secrets management starts from something I have seen in real code more times than I'd like: an application that connects to its database with a username and password sitting in its configuration. The fix in the lab is to move that credential into AWS Secrets Manager and have the application fetch it at runtime. That part is quick. The lab also points at rotation, and working out what rotation actually does took me longer. That is the part worth writing down.
Storing the secret is only half the fix
Once the password lives in Secrets Manager, the application asks for it through the API instead of reading it from a file. That takes it out of the codebase, but the value itself stays the same until someone changes it. If it leaked before the move, it is still leaked. Rotation is what makes a leaked value expire on its own.
How rotation works
Rotation is run by a Lambda function. For RDS and a few other databases, Secrets Manager can create that function for you from an AWS template. Every rotation goes through four steps:
- createSecret: generate a new password and store it as a new version labelled AWSPENDING.
- setSecret: change the password on the database itself.
- testSecret: log in with the new value to prove it works.
- finishSecret: move the AWSCURRENT label to the new version. The old one becomes AWSPREVIOUS.
The labels were the part that clicked for me. The application always asks for AWSCURRENT, so it only ever receives a password that has passed the test step. If a step fails, the label stays where it was and the rotation is marked as failed, which is why the alert further down matters.
Single user or alternating users
With the single user strategy, the function changes the password of the same database user the application logs in with. For a short window the application can still be holding the old password after the database has already changed it. The alternating users strategy keeps two database users and switches between them, so the previous credential stays valid for one more cycle. It needs a separate admin secret that is allowed to change passwords, but for anything busy I would pick it.
Two things I would watch for
Networking. The rotation Lambda has to reach two things: the database, and the Secrets Manager API. If the function runs in private subnets with no NAT gateway and no Secrets Manager VPC endpoint, it can talk to the database but not to Secrets Manager, and rotation fails on a timeout that does not explain itself.
Caching. An application that fetches the secret once at startup and keeps it forever will break the first time rotation runs. The fix is to cache the value with a refresh interval, and to fetch it again whenever the database rejects a login.
What I would do in production
- Turn on rotation for every database secret and match the schedule to the risk. Secrets Manager can rotate as often as every four hours, but monthly is a reasonable baseline for most workloads.
- Alert on rotation failures. Secrets Manager records a RotationFailed event in CloudTrail, and an EventBridge rule on it means a broken rotation gets noticed before the next one is due.
- From the SOC side, watch GetSecretValue in CloudTrail. A secret read by a role that has never read it before is worth a look.
- Keep the secret's resource policy and its KMS key tight. Rotation limits how long a stolen value stays useful, but it does not decide who can read the current one.
Why it matters
Before this lab I thought of rotation as a compliance checkbox. It is really a recovery control: it puts a time limit on any credential that escapes, without anyone having to notice the leak first. Combined with tight access to the secret itself, that is a much better position than a password that lives in a config file for years.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article