What changed when we stopped processing campaigns in order
Our notification system worked well when we first went to production, but larger campaigns soon brought complaints about slow processing. In Part 2, I share why we moved from FIFO to Standard queues, how that changed the way we tracked campaign completion, and the new challenges that surfaced when work had to be retried.
Series: Day 0 Is When You Push to Prod (2 articles)
- 2What changed when we stopped processing campaigns in order This article
What changed when we stopped processing campaigns in order
Day 0 Is When You Push to Prod — Part 2
When we first took our notification system to production at KonfHub, we used SQS FIFO queues. Campaigns were being processed, notifications were going out, and things initially looked fine. The design described in Part 1 was the one we were running for real organisers and attendees.
Then we started receiving complaints that campaigns were taking too long to process. For an organiser sending a reminder before a session, a notification arriving late could defeat the purpose of sending it. Getting the message out eventually was no longer enough.
We had put all the recipient work for each campaign into one FIFO message group. That meant Lambda couldn't process different batches from the same group at the same time. As campaigns grew, more work waited in that sequence, even when there was capacity to run more workers.
This was the kind of learning that gave this series its name. We had built something that worked, but production showed us an assumption we needed to revisit. Before replacing FIFO, we discussed whether changing how we grouped the work would be enough.
Could we keep FIFO and use more groups?
Our first question was whether we could remove the bottleneck while keeping the duplicate protection we had chosen FIFO for. Instead of putting every recipient from a campaign into one message group, we could spread them across several groups. Each group would still preserve its own order, but Lambda could process different groups at the same time. AWS describes how this works here .
For example, a campaign with 3,000 recipients could be divided into three groups of 1,000. These are example numbers, but they show the difference:
1
2
3
4
5
6
7
8
One group for the campaign:
Group A: 3,000 recipients → batches processed in sequence
Three groups for the same campaign:
Group A: 1,000 recipients → Lambda invocation
Group B: 1,000 recipients → Lambda invocation
Group C: 1,000 recipients → Lambda invocation
Can run at the same timeProvider limits and worker capacity would still affect the sending speed, but batches would no longer all wait in the same sequence.
We could also keep FIFO's protection against repeated submissions within its five-minute window. Using multiple groups does not, by itself, remove deduplication. The configuration matters: SQS can check deduplication IDs across the queue or within a message group. When the check is within a group, a retry needs to use the same group as well as the same deduplication ID. AWS documents those settings here .
There was also an application detail to account for: the same attendee could belong to several campaigns. Priya receiving an announcement from campaign A and a reminder from campaign B should receive both. Those are different notifications, even though the email address is the same.
If we used only her email address as the deduplication ID with a queue-wide check, the second notification could be suppressed within the five-minute window. An ID combining the campaign, recipient and notification type would distinguish those sends while still recognising a retry of the same notification. Multiple campaigns therefore don't make multiple FIFO groups impossible; they mean we must identify each notification correctly. SQS uses the deduplication ID supplied with the message .
We discussed multiple groups but did not implement that option. It could address the ordering bottleneck, but it would still leave us responsible for identifying duplicates correctly and handling failures after sending. That led to a broader question about whether we needed FIFO's ordering at all.
Did the recipients need to be processed in order?
Looking at the problem this way brought us back to what the campaign actually needed. If Priya and Arjun were both receiving an event reminder, there was no reason for Arjun's notification to wait until Priya's had finished processing. Sending to either attendee first was fine.
Splitting a campaign into groups would still mean choosing which independent recipients had to wait behind one another. We didn't need that order, and FIFO's deduplication couldn't resolve a worker failure after a provider had already accepted a notification.
We eventually moved to Standard queues. This allowed different batches from the same campaign to be processed concurrently without assigning them to separate ordered groups. Standard queues can deliver a message more than once and do not guarantee order, so our application had to account for both behaviours. These are part of the Standard queue delivery model .
The last message no longer meant the campaign was finished
Removing the ordering restriction addressed why batches were waiting, but organisers also needed a reliable answer to another question: had their campaign finished? That part of the application depended on our original design. We marked the final message added to a campaign with a
last_message flag and used it to help determine when the campaign had finished.Even with FIFO, that flag wasn't proof that every send succeeded. With Standard queues, the flagged message could finish before earlier work had even completed processing.
Imagine a campaign with three batches. The first batch is waiting for a slow provider response, the second has finished, and the third contains the last message we published:
1
2
3
4
5
Batch 1: still processing
Batch 2: finished
Batch 3: finished, including the message marked "last"
The campaign still has work left in Batch 1.If we marked the campaign complete when Batch 3 finished, an organiser could see a finished campaign while some notifications were still being processed. Splitting the campaign across multiple FIFO groups would have raised the same issue, because those groups could finish at different times too.
We started tracking how much work was finished
To give organisers a reliable status, we needed to account for every recipient, including failed sends. We kept the expected recipient count alongside the successfully processed and failed counts. Workers updated those counts as they processed the campaign.
For a campaign with 1,000 recipients, suppose 700 had been processed successfully and 20 had been recorded as failed. There were still 280 recipients to account for, regardless of which batch had finished most recently.
When all 1,000 were processed successfully, the campaign could be marked as processed. If the successful and failed counts together reached 1,000, with some failures, we marked it as partially processed. That distinction let us show that processing had ended without suggesting every send had succeeded. A notification waiting for another retry was still unfinished work. Only failures we would no longer retry should count towards ending the campaign.
The worker uses a DynamoDB update to add its processed count to the stored total. This avoids two workers both reading the same old count and overwriting each other's progress. It doesn't, however, prevent the same work from being counted twice if an update is repeated. The counts still depend on correctly handling retries.
We also needed to be clear about what “processed” meant. A provider accepting our send request does not mean the message has reached the recipient. Delivery updates arrive separately, and an email can still bounce after the provider accepts it. Campaign processing and delivery results answer different questions.
Recording progress brought us back to retries
A small successful test can make this approach look complete. Imagine sending a campaign to 10 test recipients: every provider call succeeds, every database update succeeds, and the processed count reaches 10. That confirms the straightforward path works, but it doesn't tell us what happens when a worker stops between those two steps.
With a larger set of recipients, there are more calls and more opportunities to encounter a timeout or a retry. Consider a worker that sends an email successfully and then stops before recording the result. The provider has accepted the email, while our campaign record still shows unfinished work. Retrying the send could produce a duplicate; repeating a progress update that already succeeded could count the same recipient twice.
Recording success before calling the provider has the opposite problem. If the worker stops before sending, our records could claim progress that never happened. Counting outcomes was useful, but those counts were only trustworthy if retries didn't lose or repeat the progress we recorded.
What we took away from this change
We started this phase with complaints about slow campaigns. Investigating them brought us back to the way we had grouped recipients in FIFO. Moving to Standard queues removed that ordering restriction, and tracking recipient counts gave us a way to determine campaign status without relying on the last message in the queue.
Those changes addressed the problems in front of us, but they also made the next question harder to ignore: after a failure, how could we tell which notifications still needed work? Some of that uncertainty had existed with FIFO too. Changing queues did not create every failure case, but it forced us to look more closely at assumptions that successful sends had allowed us to overlook.
That has been a recurring part of building this system. A feature can meet its initial requirements and still need changes once people depend on it in production. Each improvement needs to be checked against the rest of the application, because changing how work is processed can also change how we track it and recover when it fails.
In Part 3, I'll continue with that recovery problem: how we approached duplicate handling, what a retry should do when a send may already have succeeded, and why checking whether a recipient has a database record isn't enough on its own.
Series: Day 0 Is When You Push to Prod (2 articles)
- 2What changed when we stopped processing campaigns in order This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article