AWS Builder Center

My EventBridge Scheduler timer fired and nothing happened

We built Deadman, a safety net for AWS firewall changes: it applies a security-group change and undoes it on a timer unless a person confirms. The timer is a one-time EventBridge Scheduler schedule that invokes a revert Lambda.

For hours, the revert never ran. The schedule was created, fired, and deleted itself, and the revert function's log group did not exist, so there was nothing to read. The cause: my SAM template never set a FunctionName, so CloudFormation named the function with a random suffix, while the schedule pointed at the plain name. Scheduler deleted itself without any visible error.
What fixed it: setting an explicit FunctionName on the revert function, and building its ARN with !Sub. That also broke a circular dependency I had created by referencing the function's ARN from other resources.
Other things that cost me time: a Lambda authorizer that returned the wrong shape produced only a generic "Internal Server Error" from API Gateway; and after redeploying the website to S3, CloudFront kept serving the old page until I created an invalidation. CloudWatch Logs then showed the real timing: the timer had fired about 28 seconds after the requested time.
If you use Scheduler to trigger Lambda: name your functions explicitly, check the target ARN against the real function, and invoke the function once by hand so its log group exists before you rely on the timer.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article