
Our AWS Emails Kept Failing a Security Check — Here's the DNS Mystery Behind It
A domain's email security check kept randomly failing for weeks. The cause wasn't AWS — it was hiding in how the domain's DNS was set up. Here's how we found it, explained from the ground up.
Before we start: three quick definitions
If you're newer to AWS or networking, here's what you need before this
story makes sense. Already know this? Skip ahead.
- DNS is the internet's phone book. It translates a human-readable
domain name (like
example.com) into the technical records that say where to find things — mail servers, websites, verification records, anything.
- A nameserver is a computer that answers DNS questions for a domain. Most domains have more than one — usually four — so that if one goes down, the others still answer. Which nameserver responds to a given query is essentially random from the requester's side.
- DKIM is how an email service proves "I really am allowed to send email as this domain." It works by publishing a special record in DNS that only the domain owner could have added. SES is Amazon's email-sending service, and DKIM verification is the exact check it was failing.
With that out of the way, here's what happened.
The problem: a security check that kept flip-flopping
We use Amazon SES to send email from one of our domains. Part of
setup is DKIM verification — SES checks DNS to confirm we actually
control the domain before letting us send mail "as" it.
That verification kept flipping between "Verified" and "Failed," with
no changes on our end — no DNS edits, no SES config changes. It would
sometimes fix itself for a few hours, then break again. That kind of
unexplained flakiness is easy to shrug off, until it starts affecting
whether your emails actually arrive.
Step one: rule out the obvious
First move: check whether we'd changed something without realizing it.
DNS records, recent edits, TTLs — nothing had changed. The records that
existed were correct. That ruled out "we broke something" and left us
with "something is inconsistent, and we don't know where."
Step two: ask AWS
AWS Support checked their side and found no service issues affecting
SES in our region during the failure windows. Their working theory
pointed at DNS resolution — something about how our nameservers were
responding — not a problem with AWS itself.
Step three: the trick that actually found it
Here's the part worth remembering if you ever debug something like
this: querying DNS normally hides exactly this kind of problem.
A normal DNS lookup asks one of a domain's nameservers — whichever
answers first — and caches the result. It doesn't check all of them.
If three of four nameservers have the right answer and one doesn't, a
normal lookup usually never shows you that, because it only needed an
answer, and it got one.
So we queried each nameserver individually — one at a time, asking
each directly: "what do you have for this record?"
Two answered correctly. The other two returned nothing — not an error,
just silence, as if they'd never heard of it.
The root cause: four receptionists for one building
Picture a domain's nameservers as four receptionists covering the same
front desk on different days. If a courier asks any of them "where's
mailbox 12," they should all say "third floor, left." But imagine
mailbox 12 was added after only two of the four were briefed — the
other two have no idea it exists. Not broken, just never told.
That's what happened. Our internal DNS hosted a subzone covering DKIM
records for a few services, including SES. At some point our domain's
public delegation was extended to two more, externally-hosted
nameservers — but that DKIM subzone was never copied over to them (this
copying is called a zone transfer). Two of our four public
nameservers were structurally incapable of ever answering for those
records.
Since lookups can land on any of the four, and SES periodically
re-checks, verification succeeded whenever the query landed on a
correctly-configured nameserver and failed the rest of the time.
Nothing was "changing" — the gap had been there the whole time, just
invisible until something checked often enough to notice.
The fix
Zone-transfer the DKIM subzone to every nameserver in the delegation,
not just the original two. Then verify properly: query each
nameserver individually again and confirm all four now agree before
calling it fixed.
What to take away
If you're a student learning cloud/networking: this is a great
real example of why "it works when I check DNS" isn't the same as "it
actually works." Multiple nameservers for redundancy is a feature —
but it also means a config gap can hide for a long time if you only
ever check one of them.
If you're already working with this professionally:
- "No incident on our end" from a provider is a strong signal to check consistency across your own infrastructure, not just whether a record exists somewhere.
- Query authoritative nameservers directly when debugging DNS flakiness — cached lookups actively hide this exact kind of per-server inconsistency.
- Any DNS change has to be checked against every nameserver in the delegation, not just the "main" ones.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article