BedRook | S01E01. The Rules: One Prompt, 65 Amazon Bedrock Models, Three Tries per Move

I'm Adrien, a Solutions Architect at AWS, and this is the episode I would have wanted first if I were you: the rules. Not a summary of them. The actual words every model receives, the exact three chances it gets, the clock, and the six opponents. If you have an Amazon Bedrock account, you can run the same game tonight.

Everything below is frozen. Change one character and a new season opens, because a benchmark you can quietly re-tune is a benchmark nobody should trust, starting with me.

Two messages, through the Converse API , the same call for all 65 models.

The system prompt is one sentence:

The user prompt, every move, is this and nothing else:

Two chess terms, then we move on. FEN  is the whole position written on one line: where every piece stands, whose turn it is. PGN  is the list of moves played so far. So the model gets the board, the history, its colour and the answer format.

It does not get the list of legal moves. With the list, the task becomes multiple choice, and I'd be measuring whether a model can pick from a menu. I want to know whether it can hold the board in its head.

No personas. No examples. No "you are a grandmaster". No per-model tuning, ever. If a model needs hand-holding to produce a legal move, that is the result, not a bug to fix.

Move 12. The model answers with three paragraphs about pawn structure and no move. I don't score that. I tell it:

It answers MOVE: Nf7, and there is no knight that can reach f7. I tell it that too:

Three failures in a row, whatever the mix, and the game is forfeited. Not lost. Forfeited. A loss is a chess result: the model played and the other side won. A forfeit means the model stopped being a legal participant. The two never share a column on the board: forfeits are kept out of the strength rating and published as their own number.

Each move gets 300 seconds. Season 0 had 60, and that number turned out to measure my patience rather than the model's chess: every one of claude-opus-5's forfeits at the large budget was a first token arriving after 61 seconds. Zero tokens received, game over. My bug, told in full in S01E07. At 300 a model can finish its thought, and how long it thought is recorded on every ply.

Each model is measured under two output budgets, 1 024 and 16 384 tokens per answer, as two separate rows on the board. Why two, and what it changes, is the whole of S01E02.

Sampling is temperature 0 wherever the model accepts it. The newest reasoning models refuse the parameter and run their own schedule; for those, the refusal is recorded, and the results reproduce in aggregate rather than game by game.

Six fixed players, the rungs of the staircase from S01E00 . Each one was rated inside this pool over 3 000 calibration games rather than trusted from its label, because a "1500" engine and a "1500" human are not the same 1500:

Every model plays 40 opening pairs against each rung, each opening once as White and once as Black, from a frozen suite of 60 engine-verified balanced positions. 6 × 40 × 2 = 480 games per budget, 960 per model. Ratings come from Glicko-2 , the same family of maths as Elo with a confidence interval attached, and the six rungs hold the scale still.

Because they are the only argument. A table from an undisclosed protocol is a claim. The same table with the protocol beside it is something you can argue with, and I'd rather you argued with the rules than took the numbers on faith.

Previous episode: the Pilot 

I'm Adrien Lebret. BedRook is a personal project: it runs on Amazon Bedrock, I work at AWS, and the opinions here are mine alone. For now I'm publishing the articles; if the series finds its readers, the code and the games follow in a public GitHub repo. If something wasn't clear, if there's a model you want to see on the board, or a duel you'd pay to watch, say so in the comments. That's how the next episodes get written.