AWS Builder Center

Builder Center

5 AWS Services, 5 Different Approaches to Idempotence

Comparing idempotence in CloudFormation UpdateStack, AWS Backup StartBackupJob, and EC2, RDS and Aurora Start/Stop

Paul Marcelin

If your application uses EventBridge, SQS queues, AWS Lambda functions, Step Functions, or other constructs from the realm of modern, "loosely-coupled" software architecture, you can make it faster, simpler, and more reliable by attending to idempotence. There is less overhead, less checking, once you can guarantee that repeating an operation won't change the result. If the AWS operations that underly your application are idempotent, it is even easier to make your own code idempotent.
In the process of writing and refining  github.com/sqlxpert/lights-off-aws  , a tool to cut costs by temporarily stopping or deleting expensive resources based on cron schedules in resource tags or stack tags, I've become all too familiar with the different ways in which 5 core AWS services approach idempotence. The  services were launched at different times , and it shows.
Operations
Idempotence Mechanism
Exception, Error Code
Token (Restrictions)
EC2:



 StartInstances   StopInstances 
Built-in


RDS:



 StartDBInstance   StopDBInstance 
Not idempotent
Invalid…StateFault Invalid…State

Aurora:



 StartDBCluster   StopDBCluster 
Check error message after
Invalid…StateFault Invalid…StateFault

CloudFormation:



 UpdateStack 
Set token before

ClientRequestToken (≤128 letters, numbers, hyphens)
AWS Backup:



 StartBackupJob 
Set token before

IdempotencyToken (no restrictions)

1. EC2

EC2 is the oldest of the 5 services covered.  StartInstances  and  StopInstances  are idempotent. If I try to start a compute instance that is already running, my request succeeds. The dynamic response mentions that the instance was already running at the exact time of my request, in case I need to know.
JSON
"StartingInstances": [
    {
        "InstanceId": "[redacted]",
        "CurrentState": {
            "Code": 16,
            "Name": "running"
        },
        "PreviousState": {
            "Code": 16,
            "Name": "running"
        }
    }
]

2. RDS

RDS was built on EC2, but  StartDBInstance  and  StopDBInstance  are not idempotent. If I try to start a database that is already running, I get an error. The exception name is  InvalidDBInstanceStateFault  but the error code is  InvalidDBInstanceState  — a bug waiting to happen! The only thing the long, mostly static error message doesn't tell me is that the database was already running ( available ) at the exact time of my request. I don't know whether to ignore the error (because my start request is indeed a harmless repeat) or take it seriously (in case the database is in a bad state and cannot be started). Keep in mind that relying on a separate status check would create a  race condition .
plaintext
An error occurred (InvalidDBInstanceState) when calling the StartDBInstance
operation: Instance [redacted] cannot be started as it is not in one of the
following statuses: 'stopped, inaccessible-encryption-credentials-recoverable,
incompatible-network (only valid for non-SqlServer instances)'.

3. Aurora

Aurora is RDS's new and improved cousin.  StartDBCluster  and  StopDBCluster  feature a consistent exception name and error code,  InvalidDBClusterStateFault  . More importantly, the dynamic error message mentions that the database was already running ( available ) at the exact time of my request. Knowingly ignoring the error achieves idempotence after the fact.
plaintext
An error occurred (InvalidDBClusterStateFault) when calling the StartDBCluster
operation: DbCluster [redacted] is in available state but expected it to be
one of stopped, inaccessible-encryption-credentials-recoverable.

4. CloudFormation

CloudFormation predates Aurora. If I add a fixed token,  UpdateStack  is idempotent: repeated requests succeed and CloudFormation acts only on the first request.  ClientRequestToken  is limited to 128 alphanumeric characters and hyphens. Because idempotence often applies during a known time interval, and because CloudFormation displays the token in the AWS Console (in the stack events list), a human-readable "extended"  ISO 8601-standard time  string would be ideal. Instead, we have to settle for the cryptic " basic " form,  T1510Z  instead of  T15:10Z , for example.
Arbitrary lexical restrictions that make token values harder for humans to interpret are frustrating, but there is also a semantic problem lurking below this approach to idempotence. What  UpdateStack  idempotence case cannot be resolved with a token? Feel free to discuss in the comments. Hint: review the EC2  StartInstances  example, above.

5. AWS Backup

AWS Backup is the newest of the 5 services covered.  StartBackupJob  follows the same approach as CloudFormation  UpdateStack  , but the token is named  IdempotencyToken  and no specific limits are placed on length or permitted characters.

Differences, and Why We Care

These 5 AWS services take different approaches to idempotence. EC2, the oldest of the 5, and AWS Backup, the newest, handle idempotence very well. RDS  StartDBInstance  and  StopDBInstance  are not idempotent. The Aurora equivalents,  StartDBCluster  and  StopDBCluster  , put extra work on the user — and CloudFormation  UpdateStack  does, too, though there's no warning in the  UpdateStack documentation .
If we consider EC2  StartInstances  /  StopInstances  , Aurora  StartDBCluster  /  StopDBCluster  and AWS Backup  StartBackupJob  as examples of three basic approaches to idempotence, which two approaches are semantically equivalent? Would any one approach be sufficient to make all of the operations covered in this article idempotent? If not, what makes one operation or pair of operations special? Feel free to discuss your ideas in the comments.
Inconsistencies are expensive! As AWS users, we have to discover them, sometimes by trial and error, then write extra code to work around them, and then fix the extra bugs that result. Although it's impractical to change existing, widely-used APIs, I hope AWS will standardize its approaches to idempotence in new services and features. I suggest ways to prevent inconsistent software designs and implementations within our own organizations, in "How Can We Avoid Repeating this Pattern?", the second half of  a less technical version of this article, on LinkedIn    
# amazon-ec2# amazon-rds# aws-backup# reliability# serverless
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article