When we pick up a feature from Jira, the requirements tell us what it needs to do. We build it, test it, and deploy it. Once people start using it, we get to see how our decisions hold up with real traffic, overlapping requests and failures between services.

That is the idea behind Day 0 Is When You Push to Prod. In this series, I’ll share what we learned while building and running notifications at KonfHub, where I’m the Head of Engineering. Each post follows a problem we faced, the decision we made, and what we learned afterwards.

We’ll begin with sending campaigns and our first queue choice. Later posts will follow the questions that came up around campaign progress, retries, provider limits and email reputation.

Sending an email is easy to get started with. After setting up a provider, you can make an API call and check your inbox. SMS and WhatsApp have their own approval and template requirements, but once those are in place, getting a message through is fairly straightforward too.

At KonfHub, we needed organisers to be able to send those messages to their event attendees. But with thousands of organisers running thousands of events, several campaigns could be running at the same time.

Suppose one organiser sends an announcement to 20,000 recipients while another sends a reminder to 200. Both click send at 9 AM, and the reminder is for a session starting at 9:15. These are example numbers, but they show the kind of situation the system needs to handle. Each organiser has made one request, while our system now has 20,200 notifications to work through, alongside anything already running.

The organiser sending the reminder has no reason to know about the larger campaign. They expect their reminder to arrive while it is still useful. Both organisers also need to see progress and understand any failures, without repeatedly clicking “send” and wondering whether that will create duplicate notifications.

Before we could meet those expectations, we needed to answer a basic question: what should happen to notifications that we are not ready to send yet?

Preparing recipient work in our application did not mean the provider could accept all those notifications immediately. Providers had sending limits, so we needed to account for how quickly we could actually send the work we had prepared.

Suppose, the provider allows 100 recipients per second. Even with no other traffic or overhead, sending those 20,200 notifications would take more than three minutes. Starting more workers would not change that allowance. If they all tried to send as quickly as possible, we could hit throttling and create more work to retry.

We therefore needed to keep track of notifications that were waiting to be sent. Putting that work in a queue allowed campaign preparation to finish without having to wait for every provider call to complete.

The basic flow looked like this:

Campaign preparation added the recipient work to SQS. Lambda workers then took batches of that work and called the notification provider. If several campaigns arrived together, the notifications that workers had not picked up yet remained in the queue.

A queue alone does not keep sending within a provider’s limits. If too many workers make calls at once, the provider can still throttle them. We needed to control that separately, and I’ll cover how we approached it later in the series. For now, the queue gave us a place to hold work until it could be processed.

We then had to choose which kind of SQS queue to use.

SQS offers Standard and FIFO queues. With Standard queues, messages can be processed in a different order from the one in which they were added. FIFO queues preserve order within a message group. They also support deduplication, which helps prevent repeated submissions from adding duplicate messages to the queue.

That second feature was useful to us. If our application had to retry adding a notification to the queue, we did not want the retry to add another copy that could result in a second message to the attendee.

FIFO uses a deduplication ID to recognise a repeated submission. If we retry with the same ID within its five-minute deduplication window, SQS does not add another copy. AWS explains the deduplication window here .

For example, imagine adding Priya’s event reminder to the queue at 9:00. SQS accepts it, but the response never reaches our application. At 9:01, the application tries again. If that retry uses the same deduplication ID, SQS recognises the repeated submission instead of adding a second copy. Generating a fresh ID for every attempt would defeat the point: we need to identify the notification we are retrying, not just the attempt to submit it.

This prevents a particular kind of duplicate, but it does not guarantee that an attendee receives exactly one notification. For example, a worker might successfully call the provider and then fail before saving the result. When that work is retried, the application still needs to decide whether to send again. We’ll come back to that problem later in the series.

FIFO also requires messages to belong to a message group. For our initial design, we used one group per campaign: all the recipient work for campaign A carried the same MessageGroupId, while campaign B used a different one.

These two IDs do different jobs. The deduplication ID lets SQS recognise that we are submitting the same notification again. The message group ID tells SQS which messages must keep their order relative to one another. Giving all of campaign A’s messages the same group ID meant that the queue would preserve order across that campaign’s work.

We had chosen FIFO because its duplicate handling was useful. But using it this way also meant that workers could not freely pick up different batches from the same campaign at the same time. That behaviour came from the ordering within our message group, not from the deduplication check itself. As campaigns grew, the effect became more noticeable.

As campaigns grew, we started observing that larger campaigns were taking too long to process. Having more worker capacity available was not enough to make a single campaign finish faster.

To see why, picture a campaign whose work is processed in five batches. One Lambda invocation starts processing the first batch. Even if there is capacity to run four more invocations, Lambda cannot use them to process the other four batches from that same group at the same time. The later batches have to wait so that the group’s order is preserved. Work from other message groups can run alongside it. AWS documents this behaviour here .

Adding more recipients meant there were more batches to process, but those batches still belonged to the same group and had to wait their turn. A larger campaign therefore took longer, even when we had enough Lambda capacity to do more work at once.

FIFO was behaving as expected. The problem was that our choice of one group per campaign limited how much of that campaign we could process at once.

That left us with a question we should have asked more carefully at the start: did one attendee’s notification need to finish processing before another attendee’s could move ahead?

For our campaigns, the recipients were independent. In Part 2, I’ll cover how we approached that bottleneck, whether splitting a campaign across multiple FIFO groups could help, and why we eventually moved to Standard queues.