BedRook | S01E00. Amazon Bedrock Model Benchmark: LLMs Play Chess
Sixty-five models on Amazon Bedrock, one chess board, and a question the usual benchmarks do not ask: how long does a model stay inside the rules when nobody is holding its hand?
Series: BedRook - Which Amazon Bedrock models can finish a game of chess? (3 articles)
- 1BedRook | S01E00. Amazon Bedrock Model Benchmark: LLMs Play Chess This article
I'm Adrien , a Solutions Architect at AWS. I've been one for a little over four years, and for the last three I've mostly been helping customers run models on Amazon Bedrock . I've also been losing chess games online for a few years. For a long time the two had nothing to do with each other.
Then the customers started asking a question the benchmark tables couldn't answer. Not which model is smartest. Closer to: how far can I let this thing run before it does something that isn't an answer? They were building agents, and an agent makes a hundred decisions in a row, each one depending on all the previous ones. A benchmark scores one answer.
I wanted a simple way to show the difference. I ended up building a chess league for language models. Sixty-five of them so far, every one available on Bedrock, playing the same opponents under the same rules, with every move recorded. This is episode zero: what it is, why a chess board, who the models play against, and what this series will and will not say.
Why a chess board?
I'm not asking whether a model is good at chess. I'm asking how long it stays a legal participant, what breaks first when it doesn't, and what that costs. A chess board is the cheapest honest way I know to ask that:
- The rules are fixed and public. A move is legal or it isn't. Nobody argues with a rubric.
- The judge isn't a model. A chess engine scores the game. It has no opinion about who's playing.
- Every step is recorded. Each move, each retry, each timeout. Anyone can recompute every number in this series.
- It's long. A hundred decisions or more, each depending on all the previous ones. That's the shape of an agent loop, without the ambiguity.
- Everyone knows it. A lost king means the same thing in every language.
Who do the models play against?
Not each other, at first. If two models can't get past move ten without an illegal move, the game tells you nothing about either. So every model plays the same six opponents, who never forfeit and never change. Think of it as a staircase:
| Rung | Opponent | In plain words |
|---|---|---|
| 1 | random-mover | Picks any legal move. Loses to anyone who can plan two moves ahead. |
| 2 | blend-035 | A casual human player 35% of the time, random the rest. A distracted beginner. |
| 3 | maia-1100 | A neural network trained to move like a human rated 1100. A decent casual player. |
| 4 | blend-090 | That same human 90% of the time, random the rest. A casual player having an off day. |
| 5 | stockfish-1500 | A chess engine, throttled to play at about 1500. Beats most people who play for fun. |
| 6 | maia-1900 | Moves like a human rated 1900. Tournament level. |
Maia is a chess engine trained to predict what a human of a given rating would actually play, mistakes included; Stockfish is the engine everyone uses to find the best move, here held back on purpose. The two "blends" are there because the gap between random and a real human turned out to be too wide to measure with nothing in between.
A model climbs as far as it can. The highest rung it beats is its level, and that's the first thing you'll see on the board, before any Elo. Because every model faces the same six opponents, levels are comparable across releases: when a new model lands on Bedrock, we know within a day whether the staircase just got taller.
What's different here?
- One prompt for everyone. No tuning, no personas, no examples. The full text is in S01E01.
- A forfeit is not a loss. Three illegal moves and the game stops. A model that stopped producing legal moves has stopped playing, and that's measured separately from a model that played and lost.
- The production settings are part of the measurement. Output budget, clock, tokens per game. Season 1's main finding (S01E02) is about a setting, not a model.
- Raw data, kept. Every game sits in an immutable bucket, move by move. For now this series is the window onto it; if the project finds its readers, the harness and the games go to a public GitHub repo.
- Seasons, not silent edits. Change a rule and you open a new season. Old numbers stay valid for their season, forever.
What this series will not do?
It won't rank vendors in a headline. It won't quote a vendor's benchmark figure as our finding, or ours as a correction of theirs. It won't name a model in a title to make it look bad; the ways a model leaves the board get one episode, by mechanism, with the data attached (S01E05). And the unflattering results stay in the data every time, because a board where everyone looks good is a board nobody believes.
The series I have in mind
- S01E00. A benchmark that plays (this one) The pilot. Who is writing this and why, what a chess board can measure that a table of scores cannot, who the models play against, and what the series will and will not claim.
- S01E01. Same rules. 65 minds. (coming soon) The rules of Season 1, verbatim: the prompt, the retries, the forfeit, the clock, the two budgets, the opponents. Nothing is tuned per model, and this episode is the proof.
- S01E02. One setting changed everything (coming soon) Season 1 measured every model twice, under two frozen output budgets. The difference between the two columns is the season's finding, and it is a production fact, not a chess fact.
- S01E03. OpenAI takes the crown (coming soon) The first arrival episode, and the template for every one that follows: what a new model did in its first 960 games on BedRook, in numbers you can replay.
- S01E04. The move was illegal (coming soon) Format episode for every duel to come: two models, colour-reversed pairs, the replay of the decisive games, and the two or three moments where it turned. Waiting on the duel scheduler (ADR-0007 D1).
- S01E05. Three tries. One verdict. (coming soon) The four mechanisms by which a model leaves the board, with the models grouped by mechanism rather than ranked, and why several of them were never built for this.
- S01E06. Can you replay the win? (coming soon) The A/B that decided Season 1's prompt, as a worked example of the only prompt test that means anything: same volume, same opponents, one variable, and a number you can reproduce.
- S01E07. What the board taught us (coming soon) Season 1's closing episode: the probe gate that was never wired, the timeout that killed seven games, the alert topic nobody was subscribed to, and the read model that almost overwrote a champion.
Next episode: the exact prompt every model receives, so you can run it yourself.
Thanks for reading this far.
I'm Adrien Lebret. BedRook is a personal project: it runs on Amazon Bedrock, I work at AWS, and the opinions here are mine alone. For now I'm publishing the articles; if the series finds its readers, the code and the games follow in a public GitHub repo. If something wasn't clear, if there's a model you want to see on the board, or a duel you'd pay to watch, say so in the comments. That's how the next episodes get written.
Chess gave me the rules. Bedrock gave me the players. The rook gave the project its name: BedRook is Bedrock plus rook, and Rooky, the mascot, is the rook that keeps explaining one thing at a time.
Series: BedRook - Which Amazon Bedrock models can finish a game of chess? (3 articles)
- 1BedRook | S01E00. Amazon Bedrock Model Benchmark: LLMs Play Chess This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article