Introduction
Everything was working as expected. Files landed in an Amazon S3 bucket, and the processing workflow downloaded and processed them without a problem. Until one day, a new file arrived, and everything changed.
The workflow attempted to download the file, but Amazon S3 returned an AccessDenied error, bringing the pipeline to a halt. While every other file remained accessible, this one was different. What made it so special?
In this article, we'll follow the clues, investigate the configurations behind the scenes, and uncover the S3 Object Ownership detail that can turn an ordinary cross-account upload into an unexpected access problem.
The incident: A File Out of Place
The team began investigating the failed pipeline, but nothing had changed: the IAM permissions, bucket policy, and workflow code remained exactly the same. The new file was present in the S3 bucket, yet the workflow couldn't download it.
The team reviewed the bucket configuration and found that S3 Object Ownership was set to Bucket owner preferred, meaning ACLs were enabled.
There was only one other clue: the uploader's AWS account could read the file without any issues, while the bucket owner's workflow received AccessDenied.
If the bucket policy allowed uploads and the workflow permissions hadn't changed, what made this particular file different? Could its ownership hold the answer?
The Root Cause: When the Bucket Owner Doesn't Own the Object
Before diving into the root cause, we first need to understand how S3 Object Ownership works and how it interacts with bucket policies and object ACLs. These configurations determine who owns an uploaded object and who can access it, particularly when files are transferred between AWS accounts.
Understanding S3 Object Ownership:
Amazon S3 provides three Object Ownership settings, each defining how ownership is assigned and whether ACLs are enabled.
- Bucket owner enforced: ACLs are disabled, and the bucket owner owns all objects in the bucket, regardless of which account uploads them. Access is managed through policies.
- Bucket owner preferred: ACLs are enabled, and the bucket owner takes ownership of cross-account uploads when the uploader includes the bucket-owner-full-control canned ACL. Otherwise, the uploader retains ownership.
- Object writer: ACLs are enabled, and the uploading account owns the objects it uploads.
Bucket Policies vs. Object ACLs:
Although both mechanisms control access, they serve different purposes.
- Bucket policies define which principals can perform actions on bucket resources, such as uploading or downloading objects. They can allow cross-account uploads or enforce specific conditions on incoming requests.
- Object ACLs, when enabled, grant access to individual objects. With Bucket owner preferred, the bucket-owner-full-control canned ACL is particularly important because it allows the bucket owner to take ownership of a newly uploaded object.
The key distinction is that permission to upload an object does not automatically mean ownership of that object transfers to the bucket owner.
Connecting the Clues: What Went Wrong?
In our scenario, the bucket used Bucket owner preferred, and its bucket policy allowed uploads from an external AWS account. However, the uploader did not include the bucket-owner-full-control canned ACL in the upload request.
As a result, the uploader retained ownership of the new object. While the uploading account could read the file, the bucket owner's workflow received an AccessDenied error when attempting to download it.
Nothing had changed in the workflow code, IAM role, or bucket policy. The difference was the ownership of the newly uploaded object and the permissions available to the bucket owner.
The root cause was therefore the interaction between the bucket's Object Ownership setting and the uploader's request, rather than a change to the workflow itself.
Trade-offs and Considerations:
While adding the bucket-owner-full-control canned ACL resolves the ownership issue under Bucket owner preferred, there are a few considerations to keep in mind.
- Prefer Bucket owner enforced when possible. Disabling ACLs simplifies access management and ensures the bucket owner owns all objects, including those uploaded by other accounts.
- Enforce upload requirements. If Bucket owner preferred is necessary, consider using a bucket policy to reject uploads that omit the required ACL.
- Account for existing objects. Changing the Object Ownership setting does not automatically transfer ownership of existing objects when using Bucket owner preferred. These may require separate remediation.
Ultimately, choose the ownership strategy that fits your cross-account requirements. For most modern architectures, Bucket owner enforced is the simpler option because it removes ACL-based ownership complexity. If ACLs are required for compatibility, Bucket owner preferred can still work, provided uploaders consistently follow the ownership requirements.
Conclusion
The investigation led us to a small but important detail: S3 Object Ownership. The bucket policy allowed the upload, but because the bucket used Bucket owner preferred and the uploader didn't include the bucket-owner-full-control canned ACL, the uploader retained ownership of the new object. The bucket owner's workflow then encountered AccessDenied because it lacked the necessary access to that object.
The key lesson? A successful upload doesn't guarantee that the bucket owner can access the object.
In cross-account workflows, understanding object ownership is just as important as configuring IAM and bucket policies. When possible, use Bucket owner enforced to simplify ownership and access management. If your architecture requires ACLs, make sure upload requests follow the expected ownership configuration and verify permissions independently.
One small configuration detail was enough to disrupt an otherwise healthy pipeline. Mystery solved!
Loading article