The S3 Notification Curse: One Script, An Entire Pipeline Disrupted
A simple automation script, an unexpected configuration replacement, and an entire pipeline disrupted. Discover what went wrong with S3 event notifications and how to safely update them using Python and Boto3.
Series: AWS OCTOBER MYSTERIES -2026 (5 articles)
- …
- 5The S3 Notification Curse: One Script, An Entire Pipeline Disrupted This article
Introduction
Automating infrastructure changes is meant to make deployments easier and more reliable. But one small script can have a much bigger impact than expected. In this case, a script designed to automate S3 event notification configuration ended up disrupting an entire pipeline, affecting multiple event-driven integrations that were already in place.
What was hiding behind this configuration change? And what went wrong with the script?
In this article, we will investigate what happened, uncover what went wrong with the script, and explore how to safely update S3 event notifications without disrupting existing workloads.
The Incident: What Happened?
The issue appeared shortly after a script was executed to add a new S3 event notification. The goal was straightforward: configure the bucket to send notifications to a new Lambda function whenever an object matching a specific prefix and suffix was uploaded.
The new notification appeared to work as expected. However, shortly after the script ran, existing S3 event notifications stopped being delivered to their configured destinations.
A few clues helped narrow down the investigation:
- No application code, IAM policies, or downstream configurations were intentionally modified.
- The issue started immediately after the automation script was executed.
The evidence pointed to the configuration update itself. The question was no longer why the new notification worked, but why adding it had affected the existing ones.
The Root Cause: An Update That Cursed the Entire Configuration
To understand what went wrong, let's look at the script that was executed. The intention was to add a new Lambda notification to an existing S3 bucket. Since the notification configuration needed to be automated, a Python script using Boto3 was introduced.
The script looked something like this:
Python
import boto3s3 = boto3.client("s3")bucket_name = "my-data-bucket"new_notification = {"LambdaFunctionConfigurations": [{"Id": "new-lambda-notification","LambdaFunctionArn": "arn:aws:lambda:region:account:function:new-function","Events": ["s3:ObjectCreated:*"],"Filter": {"Key": {"FilterRules": [{"Name": "prefix", "Value": "incoming/"},{"Name": "suffix", "Value": ".csv"}]}}}]}s3.put_bucket_notification_configuration(Bucket=bucket_name,NotificationConfiguration=new_notification)
At first glance, the script looks reasonable. It defines the new Lambda notification and submits it to S3. The problem lies in what the API call actually does.
> Before running the script:
The bucket already had several notification configurations in place. For example:
JSON
{"LambdaFunctionConfigurations": [{"Id": "existing-lambda-notification","LambdaFunctionArn": "arn:aws:lambda:region:account:function:existing-function","Events": ["s3:ObjectCreated:*"]}],"QueueConfigurations": [{"Id": "existing-sqs-notification","QueueArn": "arn:aws:sqs:region:account:existing-queue","Events": ["s3:ObjectCreated:*"]}],"TopicConfigurations": [{"Id": "existing-sns-notification","TopicArn": "arn:aws:sns:region:account:existing-topic","Events": ["s3:ObjectCreated:*"]}]}
These configurations represented existing integrations that needed to remain in place.
> After running the script:
The script submitted a notification configuration containing only the new Lambda notification. As a result, the bucket's configuration was replaced with the one provided:
JSON
{"LambdaFunctionConfigurations": [{"Id": "new-lambda-notification","LambdaFunctionArn": "arn:aws:lambda:region:account:function:new-function","Events": ["s3:ObjectCreated:*"]}]}
The previous Lambda, SQS, and SNS notification entries were no longer part of the bucket configuration.
What went wrong?
The issue was a misunderstanding of the PutBucketNotificationConfiguration API behavior.
Despite its name, this API does not append a new notification to the existing configuration. It replaces the bucket's entire notification configuration with the configuration provided in the request.
This means that any existing notification entries omitted from the request are removed from the configuration. The destination resources themselves are not deleted, but S3 no longer has the corresponding notification registrations.
The script was not adding a notification. It was replacing the entire configuration.
That explains the incident: the new notification worked, while the existing integrations stopped receiving events. The problem was not with S3 or the destination services, but with the assumption made when writing the automation script.
The Solution: Retrieve, Modify, and Update the Complete Configuration
The fix is not simply to change the API call, but to change how the notification configuration is managed. Since PutBucketNotificationConfiguration replaces the entire configuration, the script must retrieve the current state, preserve the existing entries, and then submit the complete updated configuration.
A safer approach can be broken down into three steps.
Step 1: Retrieve the existing configuration
Before making any changes, use GetBucketNotificationConfiguration to retrieve the current notification setup. This ensures the script starts with the bucket's existing configuration rather than building a new one from scratch.
Python
response = s3.get_bucket_notification_configuration(Bucket=bucket_name)notification_config = {key: valuefor key, value in response.items()if key != "ResponseMetadata"}
The response includes the existing Lambda, SQS, and SNS notification configurations, as well as the EventBridge configuration if enabled. We exclude ResponseMetadata because it is response metadata, not part of the configuration expected by the update API.
Step 2: Add the new notification without losing existing entries
Once the current configuration has been retrieved, add the new notification to the appropriate configuration list. In this example, we check whether a notification with the same ID already exists. If it does, we update it; otherwise, we append it.
Python
new_notification = {"Id": "new-lambda-notification","LambdaFunctionArn": "arn:aws:lambda:region:account:function:new-function","Events": ["s3:ObjectCreated:*"],"Filter": {"Key": {"FilterRules": [{"Name": "prefix", "Value": "incoming/"},{"Name": "suffix", "Value": ".csv"}]}}}lambda_configs = notification_config.setdefault("LambdaFunctionConfigurations", [])existing_index = next((index for index, config in enumerate(lambda_configs)if config.get("Id") == new_notification["Id"]),None)if existing_index is not None:lambda_configs[existing_index] = new_notificationelse:lambda_configs.append(new_notification)
This approach preserves the existing notification entries and makes the script safe to run repeatedly without creating duplicate entries with the same ID. The other configuration sections, including SQS, SNS, and EventBridge, remain untouched.
Step 3: Submit the complete configuration
Finally, submit the updated configuration using PutBucketNotificationConfiguration.
Plain text
s3.put_bucket_notification_configuration(Bucket=bucket_name,NotificationConfiguration=notification_config)print("S3 notification configuration updated successfully!")
At this point, the request contains both the existing notification configurations and the new entry. After the update, retrieve the configuration again and verify that all expected entries are still present. A successful API response is a good first check, but testing with an actual object upload provides additional confidence that the notifications are working as intended.
By following these three steps, the script updates the bucket's notification configuration without unintentionally removing existing integrations.
Trade-offs and Considerations
Retrieving the existing configuration before updating it is a safer approach, but there are a few important considerations when automating S3 event notifications.
- Concurrent updates: The read-modify-write approach is not atomic. If two scripts or teams update the same bucket configuration at the same time, one update may overwrite the other's changes. To avoid this, centralize configuration ownership or serialize updates through a controlled deployment process.
- Destination permissions: S3 validates that it can publish notifications to the configured destinations. Make sure the appropriate resource-based permissions are in place for Lambda, SQS, or SNS before applying the configuration.
- Notification filters and duplicate events: Review prefix and suffix filters carefully, especially when multiple notifications are configured on the same bucket. Overlapping filters can lead to unexpected notification behavior. Also, S3 event notifications provide at-least-once delivery, so consumers should be designed to handle duplicate events safely.
Finally, configuration verification should not stop at checking that the API call succeeded. Validate the resulting configuration and perform an end-to-end test to confirm that the expected notifications are delivered.
Conclusion
What started as a simple automation task turned into a reminder of how important it is to understand the behavior of the AWS APIs we rely on.
The key lesson from this incident is that PutBucketNotificationConfiguration replaces the entire S3 notification configuration rather than appending a new entry. When working with shared resources, overlooking this detail can have a much wider impact than expected.
Before automating infrastructure changes, always understand whether an API creates, updates, or replaces an existing configuration. Retrieve the current state, preserve what is already there, apply the change, and verify the result.
A small change should never come at the cost of breaking what is already working. Mystery solved!
Series: AWS OCTOBER MYSTERIES -2026 (5 articles)
- …
- 5The S3 Notification Curse: One Script, An Entire Pipeline Disrupted This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article