I edited EC2 user data, rebooted three times, and nothing changed
I fixed a bug in my user data script on a stopped instance and started it again. The fix never ran, because user data scripts only run on the first boot by default.
I was using user data to install nginx and drop a config file onto a practice instance. The script had a bug, so I stopped the instance, edited the user data in the console and started it again.
Nothing changed. I rebooted. Nothing. I rebooted one more time, as if the third time would be different, before I finally went and read the docs.
On Amazon Linux and most common AMIs, user data is handled by cloud-init, and by default shell scripts in user data run once per instance, on the first boot. The EC2 user data documentation says so pretty plainly. Editing user data on an existing instance changes what's stored, but cloud-init already considers that instance set up and doesn't run it again.
The most useful thing I learned from all this is where the output goes:
```bash
sudo cat /var/log/cloud-init-output.log
```
That log showed the exact line where my original script failed. It was a typo in a package name. I'd been guessing for twenty minutes when the answer was sitting in a log file the whole time.
My options from there were: launch a new instance with the fixed script, run the fixed script by hand over Session Manager, or configure cloud-init to run scripts on every boot. For a practice instance, launching a new one was the cleanest. Running it by hand works for a quick test, but then the instance doesn't match its launch config anymore, and that bites you later.
Now I test user data by launching fresh instances instead of editing old ones, and I try to write the scripts so they're safe to run twice. It also made launch templates click for me. If instances are disposable and user data only runs at launch, then the fix always belongs in the template, not on the server.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article