Weekend Showcase Challenge: Nightshift
An agent that files one fictional postmortem a day for an outage that never happened. A week of unattended runs produced eleven documents and every one of them was a SEV2, because the instruction telling it to vary named the system and the failure and never mentioned severity.
Weekend Showcase Challenge: Nightshift
Nightshift shipped a week ago for the Creative Agent Challenge: an agent that
files one fictional postmortem a day, built so that nobody ever has to open it.
For the seven days since, it got exactly the amount of attention it was designed
to need, which is none. It filed at 21:00 UTC anyway, every day, without being
asked.
So when I came back to it for this showcase I was not looking at a demo run. I
was looking at eleven documents produced without supervision, and reading eleven
of something is a different act from reading one.
Every single one of them was a SEV2.
That is what this weekend actually gave me. Not the feature I ran out of time
for last week, but a defect that does not exist in a single output and is
unmistakable in a column of eleven. The rest of this is what Nightshift is, what
I changed because of that column, and what the summer taught me.
Vision and What It Does
Nightshift writes a blameless postmortem every day for an outage that never
happened.
The system invents the service, the failure, the timeline, and the engineer's
quiet conclusion at the end. A cache that held too long. A token pool that
drained in the exact order it was designed to drain. A queue that stayed silent
because silence was the configured behaviour. None of it is real. All of it
reads like a document somebody filed at four in the morning.
The problem it addresses is small and specific. Postmortems are the most useful
writing in engineering and the least practised. You only get to read one when
something has already gone wrong at your own company, and by then you are too
close to it to learn calmly. Nightshift produces a steady supply of failures
that cost nothing, from systems nobody has to defend.
It is deliberately something you never open. The agent files the incident,
regenerates the page, and goes back to sleep. If you want it, it is there. If
you do not, nothing happened, which is exactly the point of the fiction.
When you do open it, you get a single column of incidents, newest first, with
no navigation and nothing to click. Each one is a document rather than a
summary: a severity and a duration in the header, a timeline of four to six
moments with timestamps, a paragraph explaining the actual mechanism, and one
closing sentence an engineer would underline. Each entry also carries the
timestamp of the unattended run that produced it. Reading one takes about a
minute, which is the length of the habit I was trying to make possible.
How I Built It
Deciding what "always-on" should mean
An agent that runs continuously and an agent that runs once a day are the same
thing from the reader's side and very different things on the bill. I picked
scheduled execution because the original challenge prompt had already answered
the question: the best tool is the one you never have to open. Something you
never open does not need to be awake.
That decision removed a server, a container, and a websocket from the design
before I wrote any code.
Writing the prompt before writing the infrastructure
I ran the reasoning prompt through the AWS CLI first, against Bedrock directly,
before creating a single resource. If the model could not invent a failure that
sounded like it came from a real incident channel, there was no product.
It could. The instruction that made the difference was negative rather than
positive: no heroics, no villain, the system did exactly what it was told.
Without that line the output drifted into thriller. With it, the tone settled
into the flat regret that real postmortems have.
Nova wraps its JSON in markdown fences no matter how firmly the system prompt
forbids it. Forcing the shape through Converse toolUse is the more correct
answer and I have left it undone on purpose: fourteen invocations in, CloudWatch
reports zero errors, so the fence-stripping has never once been the thing that
broke. Rewriting it this weekend would have been work I could point at instead
of work the archive was asking for.
Giving the agent a memory without giving it a database
The agent reads its own archive from S3 before generating, and the six most
recent titles go into the prompt as things to avoid. That is the entire memory
system. No DynamoDB table, no vector store, no embedding call. A JSON file it
overwrites once a day.
The same Lambda run regenerates the whole HTML page from that file, so the site
has no build step and no framework. Rendering an archive of sixty entries as a
string is cheaper than deploying anything that could render it properly.
The finishing touch: eleven SEV2s in a row
Here is the instruction that had been running unattended all week:
Pick a different kind of system than those, and a different failure mechanism.
Read it again and the hole is obvious, but I did not see it for seven days,
because the output looked varied. The systems were different every time. The
mechanisms were different every time. The archive drifted exactly as designed
along both axes I had named — and stayed frozen on the one I had not. Severity
was in the output schema, nobody had said anything about it, and the model
quietly picked the same answer eleven times.
An underspecified instruction does not fail loudly. It gets filled in, silently
and consistently, and the consistency is what makes it invisible.
The fix I shipped this weekend is not a better sentence. Asking the model to
"also vary the severity" would have been the same class of instruction that
caused the problem. Instead the Lambda now computes the answer before it asks:
function leastUsedSeverity(archive, window = 9) { const recent = archive.slice(0, window); const counts = { SEV1: 0, SEV2: 0, SEV3: 0 }; for (const e of recent) if (counts[e.severity] !== undefined) counts[e.severity] += 1; const min = Math.min(...Object.values(counts)); const candidates = Object.keys(counts).filter((s) => counts[s] === min); return candidates[Math.floor(Math.random() * candidates.length)];}The severity is decided in JavaScript from the archive itself, and the prompt
receives it as a fact rather than a goal: this incident must be exactly SEV1,
and scale the blast radius, the duration, and the tone to match.
Three runs after deploying, the archive had a two-hour-fifteen-minute SEV1 and
two SEV3s of nine and twelve minutes. The duration moved with the severity
without being asked, which is the part I liked: once the stakes were pinned,
the rest of the document reorganised itself around them. The live page now
carries the severity mix in its header, so the defect and the fix are both
visible on the artefact rather than only in this article.
AWS Services Used and Architecture
Service | Role |
|---|---|
Amazon EventBridge Scheduler | Wakes the agent once a day at 21:00 UTC |
AWS Lambda | Node.js 22, the whole agent, no framework |
Amazon Bedrock | Amazon Nova 2 Lite, via the Converse API |
Amazon S3 | Archive, memory, and static website hosting |
Amazon CloudWatch Logs | Execution record for each unattended run |
EventBridge Scheduler (cron, 21:00 UTC daily) | vAWS Lambda ──read──> S3: archive.json (the agent's memory) | ├── compute severity from the archive | ├──invoke──> Amazon Bedrock, Nova 2 Lite | └──write──> S3: archive.json + index.html | v S3 static website (public, no CloudFront)There is no API, no auth, and no client-side JavaScript. The page is a string
the Lambda writes. CloudFront was left out on purpose: it adds a cache to
invalidate and a distribution to wait on, and this site changes once a day.
The whole thing sits inside the Free Tier apart from Bedrock, and one short
generation a day puts the model cost below the rounding error on a monthly
bill.
What I Learned Across the Summer
I built two things this summer: StackCompass, a decision tool that returns
three structurally different options and the axis the choice actually turns on,
and Nightshift. They were built a day apart and I thought they had nothing in
common. The week since has convinced me they were the same lesson twice.
Proving autonomy is a build requirement, not a documentation step. I nearly
shipped Nightshift with a daily schedule and a screenshot of the code.
Temporarily setting the schedule to three minutes cost nothing and turned a
claim into a live artefact. This weekend the same principle paid again from a
different direction. Because the schedule had been running untouched, I had
eleven filed incidents to read as a set rather than one demo run to judge on
its own — and reading them as a set is what exposed the severity defect.
Designing for unattended operation did not only prove the agent worked; it
generated the diagnostic that told me it did not work as intended. An agent you
have to demonstrate is a demo. An agent with a week of timestamps behind it is
a system.
Vague instructions do not fail. They get filled in. This is the one I did
not expect to find in a weekend project. The severity bug was not a bug in the
model — the model did something reasonable with an instruction that never
mentioned severity. Every rule I write for an agent that leaves a decision
unstated is a decision I have silently delegated, and I will not find out what
was chosen until I look at a week of output side by side. One run tells you
nothing. Eleven in a column tell you everything.
Where a decision can be computed, compute it. Both fixes that mattered this
summer moved a judgement out of the prompt. StackCompass scores its trade-off
table on axes that actually differ rather than asking the model which option is
best. Nightshift now picks the severity in JavaScript and hands it over as a
constraint. Prompts are good at voice, texture, and invention. They are a poor
place to put a rule you could have calculated, because a rule in a prompt is a
suggestion with good manners.
Deliberate omission is a feature. No CloudFront, no database, no build step,
no framework, no auth. Every one of those was a considered no, and the payoff
arrived a week later rather than on the day. With no build pipeline to run, no
schema to migrate and no cache to invalidate, the distance between finding the
severity defect and having the corrected agent live was one deploy and three
verification runs. The smallest thing that satisfies the requirement is also
the thing that leaves you room to be wrong later, and I was.
Link to App
Every entry carries the timestamp of the unattended run that produced it, and
the header shows the current severity mix, so the defect described above and
the fix for it are both visible on the artefact itself.
A Builder Who Inspired Me
@Lewis Sawe, for The Museum That Grows:
https://builder.aws.com/content/3IDtrEijOvKe0sHVeQemYGTS64N/weekend-creative-agent-challenge-the-museum-that-grows
We published a day apart and, without knowing it, built the same machine. A
scheduler, a Lambda, one Bedrock call, and a single JSON file in S3 that the
agent reads back to itself before it writes. No database on either side.
What made me go back and look at my own archive was noticing that we point that
mechanism in opposite directions. The Museum feeds its whole collection back in
to hold a world together: artifact forty is interesting because it still
belongs beside artifacts one through thirty-nine. Nightshift feeds a handful of
recent titles back in to push away from them, because an archive of near
identical incidents is worth nothing.
Reading a design built for convergence is what made me ask what my own archive
was converging on while I was not looking. It was converging on SEV2. The
memory mechanism is the same in both projects; what differs is how precisely
you have said which axis it applies to, and I had only said two of the three.
#application
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article