BedRook | S01E02. maxTokens Decides Whether Your LLM Answers at All: 1,024 vs 16,384 on Amazon Bedrock

Previously on BedRook: Sixty-five Amazon Bedrock models entered a chess league against six fixed opponents (S01E00 ). They all received the same prompt, the same three retries, and the same clock (S01E01 ). This is the episode where a single API parameter broke the whole picture.

I’m Adrien, a Solutions Architect at AWS. This is the episode where the benchmark taught me something I now check on every customer project, and it is not about chess.

Season 0 gave every model 1 024 output tokens per move. It seemed generous. A chess move is four characters.

Then claude-opus-5 arrived, the newest model of its family, and forfeited every single game. Every one. Its predecessor, opus-4-8, played fine under the same rules and rated 663. The new model looked worse than the old one, by a lot, and I did not believe it.

Move 6. Opus-5 is writing. It writes 1 024 tokens, to the token. No MOVE: line anywhere. I ask again. 1 024 tokens. I ask a third time. 1 024 tokens. Forfeit.

It was not writing garbage. It was thinking. The model spent the entire budget reasoning about the position and was cut off before it could write the move. On every attempt, stopReason  said max_tokens. The harness filed it as UNPARSEABLE, which was honest and useless at the same time: the answer was not unreadable, it was unfinished.

I could have raised the budget and called it fixed. That would have hidden the finding. Instead Season 1 froze two budgets, 1 024 and 16 384 tokens, and ran every model through both: 480 games each, same prompt, same opponents, same clock. Two rows per model on the board, never averaged, because the gap between them is the measurement.

Share of games that reached a result on the board (checkmate, draw, or the chess equivalent of resignation), Season 1, 480 games per cell:

Three groups, and they barely overlap.

The adaptive thinkers (the GPT-5.6 and GPT-6 family, opus-5, sonnet-5, kimi-k3, deepseek-r1, gpt-oss) spend a short budget reasoning and never reach the answer. Give them room and they finish nearly every game. Sonnet-5 and r1 sit halfway: for them, 16 384 is sometimes not enough either.

The opus 4.x generation does not care. It answers in a few hundred tokens under either budget; finish rate and rating are the same in both columns.

Everything else finishes under 10% of its games either way, and the budget is not the reason. S01E05 is about what is.

Nobody deploys a chess bot. Everybody deploys a token limit.

The pattern is general: a maxTokens that is comfortable for one model family silently disables another. And the symptom is not a worse answer. It is no answer. The model does not degrade; it goes quiet. Your dashboard shows normal latency and a 200 status. Only stopReason tells the truth, and if you are not logging it, you are not seeing it.

Two things I now do on every project:

They do not say 16 384 is the right number. It is where these models play; sonnet-5 and r1 hint that some would want more, and I measured 32 768 as an experiment: it buys about 2% more plies, at a quota cost every single call pays. They say nothing about quality either: a finished game is a game inside the rules, not a good one. Strength is the next episode.

Next episode: the first arrival on the board, and the template for every model that lands on Bedrock from now on.

I'm Adrien Lebret. BedRook is a personal project: it runs on Amazon Bedrock, I work at AWS, and the opinions here are mine alone. For now I'm publishing the articles; if the series finds its readers, the code and the games follow in a public GitHub repo. If something wasn't clear, if there's a model you want to see on the board, or a duel you'd pay to watch, say so in the comments. That's how the next episodes get written.