AWS Builder Center

Builder Center

The Hidden Contracts of AWS Systems: Time, Duplicates, and Ordering

The Hidden Contracts of AWS Systems: Time, Duplicates, and Ordering

Modern AWS applications rarely fail because a single service stops working. They fail when assumptions about timing, duplicate events, ordering, and state quietly break. This article explores how to make those hidden contracts explicit when designing event-driven systems on AWS.

1
2
3
4
5
6
7
8
9
10
Cloud architecture diagrams usually show boxes and arrows:

text
Client
↓
API
↓
Lambda
↓
Database
The diagram looks simple.
But production systems are rarely simple.
A request can arrive twice.
An event can arrive late.
Two events can arrive in a different order than expected.
A service can complete an operation while the caller is still waiting for the response.
These situations expose something that architecture diagrams often hide:
Every distributed system has contracts about time, duplication, and ordering.
When those contracts are implicit, failures become difficult to understand.
When they are explicit, the system becomes much easier to design, operate, and recover.

1. The First Hidden Contract: Time

Imagine an application that receives an order.
The API stores the order and then sends an event for further processing.
A developer may naturally think:
1
2
3
4
5
Order created
↓
Event published
↓
Worker processes event
But distributed systems don't work in such a perfectly synchronized way.
The worker might process the event several seconds later.
It might process it after another event.
The database might contain a newer state by the time the event is consumed.
This means an event should not automatically be treated as a statement about the application's current state.
Instead, an event should describe something that happened.
For example:
1
2
3
4
5
6
{
"eventType": "OrderCreated",
"orderId": "ORD-1024",
"eventId": "evt-7812",
"createdAt": "2026-10-08T12:30:00Z"
}
The timestamp and event ID are not decorative fields.
They provide context about when the event was created and which occurrence it represents.
This distinction becomes extremely useful when events arrive late.

2. Current State and Historical Events Are Different

Consider this sequence:
1
2
3
12:00 → OrderCreated
12:01 → OrderCancelled
12:03 → OrderCreated event is processed
At 12:03, the database might correctly say:
1
Order = CANCELLED
But the event being processed says:
1
OrderCreated
That does not necessarily mean the event is wrong.
It means the event describes something that happened in the past.
This is one reason event-driven systems benefit from treating events as immutable facts rather than constantly changing state.
A useful mental model is:
1
2
3
4
5
Event
"What happened?"

Database state
"What is true now?"
Those are two different questions.
AWS event-driven architectures commonly use services such as Amazon EventBridge, Amazon SQS, and Amazon SNS to move events between components. The important design decision is not simply choosing a service—it is deciding what information a consumer can safely assume about an event.

3. The Second Hidden Contract: Duplicates

Now consider a different problem.
A worker receives:
1
PaymentRequested
It calls a payment provider.
The provider successfully processes the payment.
But before the worker receives the response, the network connection fails.
From the worker's perspective:
1
Did the payment succeed?
It doesn't know.
If it simply retries, the provider may receive the request twice.
That creates a dangerous situation:
1
2
3
4
5
6
7
8
9
Request
↓
Payment succeeds
↓
Response lost
↓
Retry
↓
Payment succeeds again
The problem isn't necessarily that the infrastructure failed.
The problem is that the system didn't define what should happen when an operation is repeated.
This is where idempotency becomes important.
An operation is idempotent when performing it multiple times produces the same intended result as performing it once.
For example:
1
Request ID: PAY-12345
The application can use that identifier to recognize that the logical operation has already been processed.
A simplified flow becomes:
1
2
3
4
5
6
7
8
9
10
11
12
Request
↓
Check operation ID
↓
Already processed?
├── Yes → Return existing result
│
└── No
↓
Perform operation
↓
Store result
This turns an uncertain retry into a controlled operation.

4. Why Message IDs Are Not Always Enough

A common mistake is assuming that the messaging system's message ID automatically solves idempotency.
It doesn't necessarily solve the business problem.
Suppose a customer clicks:
1
"Create Report"
twice.
The application could generate two different messages:
1
2
Message A → Create report
Message B → Create report
From the messaging system's perspective, these are different messages.
But from the business perspective, they might represent the same request.
Therefore, the idempotency key should often represent business intent, not simply the transport-level message.
For example:
1
customerId + requestId
or:
1
orderId + operationType
The exact key depends on the application.
The important question is:
"What makes two requests the same business operation?"
That question should be answered before choosing the idempotency strategy.

5. The Third Hidden Contract: Ordering

Now suppose an application produces these events:
1
2
3
UserCreated
UserUpdated
UserDeleted
A developer might assume consumers always receive them in that order.
But distributed systems can make ordering more complicated.
Consider:
1
2
3
4
5
Producer
│
├── UserCreated
├── UserUpdated
└── UserDeleted
The consumer might observe:
1
2
3
UserCreated
UserDeleted
UserUpdated
If the application blindly applies every event, the final state could become incorrect.
This leads to an important design question:
Does this workflow actually require ordering?
Not every workflow does.
For independent events such as analytics records, strict ordering may provide little value.
For something like:
1
2
3
AccountCreated
AccountSuspended
AccountDeleted
ordering may be much more important.
The mistake is not choosing the "wrong" queue.
The mistake is requiring ordering without understanding why the business logic needs it.

6. Ordering Has a Cost

Suppose ten independent tasks belong to the same workload.
If all of them are forced through one ordered sequence:
1
Task 1 → Task 2 → Task 3 → Task 4
one slow task can delay everything behind it.
But if those tasks are genuinely independent:
1
2
3
4
5
Task 1 ──┐
Task 2 ──┤
Task 3 ──┼──→ Workers
Task 4 ──┤
Task 5 ──┘
they may be processed concurrently.
This is why ordering should be treated as a business requirement, not a default architectural preference.
Amazon SQS provides different delivery characteristics through Standard and FIFO queues. Standard queues favor high throughput and do not provide strict ordering, while FIFO queues provide ordering within message groups and support deduplication mechanisms.
The key is to start with the requirement:
1
2
3
4
5
6
7
Do I need ordering?
↓
Yes
↓
What exactly must be ordered?
↓
Can I divide independent work into separate groups?
That is a much better starting point than:
1
FIFO sounds safer → use FIFO everywhere

7. A Better Event Contract

A useful event should carry enough information for a consumer to make a safe decision.
For example:
1
2
3
4
5
6
7
8
9
10
11
{
"eventId": "evt-98231",
"eventType": "OrderUpdated",
"aggregateId": "order-482",
"version": 7,
"occurredAt": "2026-10-08T12:30:00Z",
"schemaVersion": 1,
"payload": {
"status": "SHIPPED"
}
}
Notice what this gives the consumer:
FieldPurpose
eventIdIdentifies this event
eventTypeDescribes what happened
aggregateIdIdentifies the business entity
versionHelps reason about state changes
occurredAtProvides event timing
schemaVersionAllows controlled evolution
payloadContains the relevant business data
Not every system needs all of these fields.
But explicitly thinking about them forces the team to answer important questions early.

8. AWS Architecture With Explicit Contracts

A practical architecture could look like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
┌───────────────┐
│ Client │
└───────┬───────┘
│
▼
┌───────────────┐
│ API Gateway │
└───────┬───────┘
│
▼
┌───────────────┐
│ Lambda │
└───────┬───────┘
│
┌────────┴────────┐
│ │
▼ ▼
┌─────────────┐ ┌──────────────┐
│ DynamoDB │ │ EventBridge │
└─────────────┘ └──────┬───────┘
│
┌────────┴────────┐
│ │
▼ ▼
┌──────────┐ ┌──────────┐
│ SQS │ │ SQS │
└────┬─────┘ └────┬─────┘
│ │
▼ ▼
┌──────────┐ ┌──────────┐
│ Lambda A │ │ Lambda B │
└──────────┘ └──────────┘
The services themselves are only half the design.
The other half is defining the contracts:

Time

How long can an operation remain valid?

Duplication

What happens if the same event is processed twice?

Ordering

Which events must be processed sequentially?

State

What happens if the event describes an older state than the database?

Schema

What happens when the event structure changes?
These questions are where many production problems actually begin.

9. Add a Version to Business State

One useful technique is attaching a version to state changes.
For example:
1
2
3
Order
Version 5
Status = PROCESSING
An incoming event might contain:
1
2
3
Order
Version 4
Status = CONFIRMED
The consumer can recognize that version 4 is older than the state it already knows about.
Instead of blindly overwriting the latest state, it can apply a rule such as:
1
2
3
4
5
6
7
Incoming version > stored version?
│
┌──┴──┐
Yes No
│ │
Apply Ignore/
handle
The exact behavior depends on the business requirement.
But the important part is that the application is no longer relying on timing alone.
It has made the state transition explicit.

10. Dead-Letter Queues Are Part of the Contract

Failures should not disappear silently.
Suppose a worker repeatedly receives an invalid event:
1
2
3
4
5
6
7
8
9
10
11
Invalid event
↓
Processing fails
↓
Retry
↓
Processing fails
↓
Retry
↓
Processing fails
Without a stopping mechanism, the same message can consume resources repeatedly.
Amazon SQS Dead-Letter Queues can isolate messages that repeatedly fail processing.
The architecture becomes:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
┌─────────────┐
│ Main Queue │
└──────┬──────┘
│
Worker
│
┌──────┴──────┐
│ │
Success Failure
│ │
▼ ▼
Delete Retry
│
Max attempts
│
▼
┌─────────┐
│ DLQ │
└─────────┘
But a DLQ is not the end of the design.
A production system should also answer:
1
2
3
4
5
6
7
8
9
Who monitors the DLQ?

Who investigates the message?

Can it be safely replayed?

What caused the failure?

Will replay create a duplicate side effect?
A DLQ without an operational process is simply a place where failures wait.

11. Observability Should Follow the Business Operation

Monitoring only infrastructure metrics is not enough.
You may know:
1
2
3
Lambda = Healthy
SQS = Healthy
DynamoDB = Healthy
and still have an application that is not working correctly.
Instead, track business-level signals such as:
1
2
3
4
5
6
7
Request accepted
↓
Event created
↓
Event processed
↓
Business operation completed
Useful metrics can include:
  • End-to-end processing time
  • Number of duplicate events detected
  • Number of stale events rejected
  • Age of the oldest queued message
  • DLQ message count
  • Failed workflow count
  • Percentage of operations requiring retries
This gives operators a view of what customers actually experience.

12. A Simple Design Checklist

Before deploying an event-driven AWS workflow, ask:

Time

  • Can events arrive late?
  • How long is an event considered valid?
  • Do we need the event's timestamp?

Duplication

  • Can the same operation be processed twice?
  • What is our idempotency key?
  • Is the side effect safe to repeat?

Ordering

  • Does ordering actually matter?
  • What specifically needs to be ordered?
  • Can independent work be processed concurrently?

State

  • Can an older event overwrite newer state?
  • Do we need versions or sequence numbers?

Failure

  • What happens after repeated processing failures?
  • Where do failed messages go?
  • Can failed work be replayed safely?

Observability

  • Can we trace one business operation across services?
  • Do we monitor business outcomes rather than only infrastructure health?
If these questions have clear answers, the architecture is much easier to reason about.

Conclusion

The most important part of an AWS architecture is not always the service selection.
Sometimes the real architecture is hidden inside the assumptions between those services.
A system may quietly assume:
1
2
3
4
5
Requests happen once.
Events arrive on time.
Messages arrive in order.
A successful response always reaches the caller.
Database state is always current.
Production eventually challenges those assumptions.
A stronger architecture makes them explicit.
Think of every distributed workflow as having three contracts:
1
2
3
4
5
6
7
8
9
10
11
DISTRIBUTED SYSTEM
│
┌────────────┼────────────┐
▼ ▼ ▼
TIME DUPLICATION ORDERING
│ │ │
"When?" "Again?" "Which first?"
│ │ │
└────────────┼────────────┘
▼
CORRECT STATE
AWS gives us powerful building blocks—Lambda, Amazon SQS, EventBridge, DynamoDB, CloudWatch, and many others.
But reliable systems come from understanding the behavior between those building blocks.
# architecture# serverless# event-driven# distributed-systems
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article