Beyond the price per token: choosing the right OpenAI model on Amazon Bedrock for your workload
This post compares OpenAI models on Amazon Bedrock (GPT-5.6 Luna, Terra, and Sol) against GPT-5.4 mini and nano using production-focused benchmarks that measure cost per successful outcome, agent efficiency, and professional-quality outputs—not just token pricing. It also provides an open-source, reproducible benchmarking framework and a workload-based decision guide reflecting the latest July 2026 Amazon Bedrock pricing.
Co-written by: Sudeesh Sasidharan (OpenAI, AI Deployment Engineer), Sharadha Kandasubramanian (Sr WW Specialist - GenAI)
Organizations building generative AI applications usually compare models the same way: dollars per million tokens. It's the number on every pricing page, so it becomes the number in every spreadsheet. But production workloads don't buy tokens — they buy outcomes: a resolved support ticket, a completed research brief, a correct financial summary. Between the pricing page and the outcome sit multipliers the sticker price ignores: how often the model is right, how many tokens it needs to get there, and — for agentic workloads — how many turns it takes, because every turn re-sends the growing conversation.
In this post, we share results from an open-source benchmarking harness that measures those multipliers across OpenAI models on Amazon Bedrock (gpt-5.6-luna, gpt-5.6-terra, and gpt-5.6-sol) and two widely used cost-efficient models on OpenAI's API (gpt-5.4-mini and gpt-5.4-nano). We chose the latter two as the cost-optimized baselines many teams start from — not as like-for-like generational peers — because "we run mini or nano today; is a newer model on Amazon Bedrock worth it?" is the question we hear most. We focus on three questions:
- What does a correct answer cost? — not what does a token cost.
- What do agent trajectories cost? — where turn count can dominate the bill.
- Can it produce work a professional would accept? — measured on real occupational deliverables, not quiz questions.
Everything here is reproducible: the harness, openai-on-aws/benchmarks-openai , runs one identical code path (the OpenAI Responses API) against both platforms, and we encourage you to run it on your own tasks before making any decision.
Solution overview
The harness evaluates all five models through the same Responses API client, switching the backend and model ID while holding the evaluation logic constant. The results still reflect differences in the models, provider infrastructure, and model-specific configuration. In particular, the Amazon Bedrock models ran with reasoning disabled, while the OpenAI API baselines ran at their defaults. This is a comparison of practical deployment configurations, not a controlled estimate of intrinsic model capability.
We measure three things: single-call accuracy and cost on benchmarks that still separate frontier models (AIME competition mathematics, GPQA Diamond graduate-level science, and MMLU-Pro); multi-turn agent trajectories on live web-research tasks; and rubric-graded professional deliverables. Grading combines deterministic checks with an LLM judge (gpt-5.5, never one of the evaluated models) using frozen prompts whose hashes are recorded in every result file. Each run writes a timestamped result JSON, and every number and chart in this post is generated from those files at build time.
How to read the results: sample sizes range from 48 to 198 items. Treat small gaps as directional unless accompanied by uncertainty estimates, and reproduce the evaluation on your own workload before selecting a model.
Measuring the cost of a correct answer
For each benchmark we divide a model's total spend — across right and wrong attempts alike — by its number of correct answers. This estimates the observed cost of a correct answer in our sample. The following graph shows accuracy and cost per correct answer on AIME for all five models; the pattern repeats on GPQA Diamond and MMLU-Pro (full tables in the repo).

Two things stand out:
- Capability tiers are visible in this sample. Sol solves 75% of AIME problems to mini's 37%, and leads GPQA Diamond (68% vs 43%) and MMLU-Pro (82% vs 59%). If attempts were independent and accuracy remained constant, 37% accuracy would imply roughly 2.7 attempts per success on average. Real retries are correlated, so teams should measure their actual retry strategy rather than assume that estimate transfers directly.
- Token efficiency decided the invoice even before prices moved — and then prices moved. At its original list price (roughly 1.5× mini's), luna was already 25% cheaper per correct AIME answer in this configuration, because with reasoning disabled it used fewer billed tokens than mini at its defaults. Following the July 30, 2026 price reduction for GPT-5.6 Luna and Terra on Amazon Bedrock (luna −80%, terra −20%), our recorded cost per correct AIME answer is $0.0021 for luna versus $0.0139 for mini. Across these samples, luna has the lowest observed cost per correct answer, including against nano, even though nano's nominal token price is slightly lower under the price assumptions recorded in the result files.Pricing note: the result files currently use $0.22/$1.32 per 1M input/output tokens for luna and $2.20/$13.20 for terra. Confirm the applicable Bedrock inference tier and region against the live pricing page before publication; the July 30 announcement describes first-party price parity, while pricing-page updates may lag.For customers, the more useful model-selection question is not only "which is cheapest per token?" but "which is cheapest per outcome, at a quality level my business accepts?"
Measuring agent trajectory cost
Single-call pricing misses a defining feature of our agentic workload. The harness uses client-managed history with
store: false , so every turn re-sends the system prompt, prior tool results, and conversation context. Per-turn context grows roughly linearly; cumulative billed input can therefore grow approximately quadratically with turn count. Each turn also adds a round trip of latency. A model that finishes in 5 turns instead of 8 can save more than the 37% reduction in turns alone suggests.To measure this on a real workload, we ran a 50-question stratified sample of DeepSearchQA — multi-step web-research questions — through a live agent loop with real
web_search and fetch_page tools. Answers were scored using a deterministic pre-pass followed by a frozen gpt-5.5 autorater; a question passes at F1 ≥ 0.7. The following graph shows how turn count drives input-token volume (left) and what each passing answer ends up costing (right).
The left panel is the mechanism; the right panel is the consequence. Mini took the most turns per question of any model (7.6, mostly re-search loops), and every extra turn re-sent a context containing accumulated search results — 2.3× terra's input-token volume by the end (114k vs 50k tokens per question). In this sample, terra's higher token price was offset by fewer turns and higher quality: it cost $0.31 per passing answer versus mini's $0.40, with mean F1 of 0.50 versus 0.39. The three GPT-5.6 configurations recorded higher mean F1 than both baselines, while luna combined fewer turns than mini with $0.05 per passing answer versus mini's $0.40. Nano's nominal token price was lower, but at an 18% pass rate its observed cost per passing answer was $0.07 versus luna's $0.05. With 50 questions, treat close gaps as directional.
The customer takeaway: turn efficiency is a pricing variable, and it is invisible on the pricing page. If your agents chain tool calls — research, multi-hop lookups, iterative retrieval — benchmark trajectory cost alongside call cost.
Evaluating professional deliverables with GDPval
Benchmarks grade answers. Much of what customers ship is documents — compliance briefs, financial plans, care protocols — where "correct" is a rubric, not a string match. So we ran a 48-task slice of GDPval: real occupational deliverables created by professionals averaging 14 years of experience, each graded against its human-authored rubric. A deliverable passes at ≥70% of weighted rubric points. The following graph shows pass rates and cost per passing deliverable.

All three gpt-5.6 configurations achieved higher observed rubric scores than mini and nano with reasoning disabled. The largest observed differences appeared in law, nursing, and financial-advice tasks, where rubrics demand specific caveats, structure, and completeness; the per-category samples are small, so this pattern should be treated as exploratory. Luna scored higher than mini on 31 of 48 deliverables, lower on 9, and tied on 8; it passed 27 deliverables to mini's 20.
This workload is a single call — no turn effects — so before the repricing, mini and nano were cheaper per passing deliverable and gpt-5.6 bought pass rate alone. After repricing, luna has the lowest observed cost in this sample: $0.010 per passing deliverable versus mini's $0.030 and nano's $0.012, at a 56% pass rate to mini's 42% and nano's 35%. Terra and sol still carry a premium per pass while recording higher rubric performance. Whether that premium is justified depends on the cost of review and rework in the target workflow.
GDPval caveat: outputs were capped at 8,192 tokens, truncating 6 luna, 9 terra, 7 sol, 0 mini, and 1 nano deliverables. These are real outcomes under the tested cap. A higher cap may improve quality, but it may also increase cost; test both together.
Putting it together: a decision framework
If you run gpt-5.4-mini or nano today, the migration question comes down to workload shape:
Your workload looks like… | Start with | Why |
|---|---|---|
High-volume, simple tasks; failures are cheap | gpt-5.6-luna on Amazon Bedrock | Luna recorded the lowest cost per successful outcome across these samples; nano retains a slightly lower nominal token price under the recorded price assumptions |
Interactive apps; accuracy matters; latency SLOs | Benchmark gpt-5.6-luna on Amazon Bedrock first | Strong observed cost per success; latency results are a point-in-time regional snapshot and should be re-measured against your SLO |
Agents that chain tool calls (research, multi-hop) | Benchmark gpt-5.6-luna and -terra on Amazon Bedrock | Both recorded higher DeepSearchQA F1 with fewer turns than mini; luna cost about 8× less per passing answer in this 50-question sample |
Quality-gated document production | gpt-5.6-luna on Amazon Bedrock | Highest pass rate among the cost-efficient models on GDPval (27/48 vs mini's 20/48) at the lowest cost per passing deliverable ($0.010 vs $0.030) |
Accuracy is a hard gate on genuinely difficult work | Benchmark gpt-5.6-sol on Amazon Bedrock | A different observed capability and price tier (AIME 75%, GDPval 31/48); consider it when lower-cost configurations do not clear your quality gate |
We also measured same-model latency on both platforms with the same streaming harness. In our July 2026 runs (us-west-2, single-region, point-in-time — shared services vary with load, so treat these as a snapshot and re-measure), Bedrock's median time-to-first-token averaged 21% lower for luna and 5% lower for terra across the 12 matched configurations. Luna's throughput averaged 43% higher on Bedrock at ≥500-token outputs (terra +4%). The observed worst-case TTFT-to-median ratios were 2.1–2.5× on Bedrock versus 4.6–6.6× on the OpenAI API. These maxima indicate tail variability but are not estimates of p99 latency. Sol behaves differently: it is a deep-reasoning model with inherently long and variable time-to-first-token, and its Bedrock runs used us-east-1. Per-configuration detail for all three models is documented in the repository's performance section , and
performance/run_all.sh reproduces the comparison on your own account.Run it on your workload
These results reflect our sample sizes and configuration choices: 50–198 questions per suite (DeepSearchQA 50, AIME 60, MMLU-Pro 140, GPQA Diamond 198), 48 deliverables, and reasoning disabled on the Bedrock models — a deliberate cost floor; enabling it raises both quality and spend. Your tasks are not our tasks. The methodology is the durable part:
git clone https://github.com/openai-on-aws/benchmarks-openaipip install -r requirements.txtexport OPENAI_API_KEY=sk-... # your own key; never storedexport AWS_REGION=us-west-2 # plus your standard AWS credentials
python quality/quick_evals.py # accuracy + cost per correct answerpython quality/deepsearchqa/run_deepsearchqa.py # live-web agent trajectoriespython quality/gdpval_eval.py # professional deliverables, rubric-judgedperformance/run_all.sh # latency, both platforms
Swap in 50–100 of your own tasks with known-good outputs and the same scripts produce cost-per-success numbers in your domain, on your account.
Conclusion
The per-token price is one input to a decision, not the decision. In the configurations we tested, normalizing cost by successful outcomes changed the model ranking once accuracy, token efficiency, and trajectory length were included. Pricing pages also move, as the July 2026 GPT-5.6 reductions showed. Define the outcome and quality threshold your application needs, measure the full cost of successes and failures, and re-run the evaluation when prices, models, or workload patterns change.
To get started with OpenAI models on Amazon Bedrock, see the Amazon Bedrock documentation . To reproduce these benchmarks on your own tasks, clone openai-on-aws/benchmarks-openai and follow the steps in the previous section.
Methodology notes: all runs July 2026; every number and chart traces to a timestamped result JSON in the repository (charts generated by
docs/img/build_blog_charts.py ). Bedrock models ran with reasoning: {effort: none} ; models on OpenAI's API ran at their defaults. Grading used deterministic checks where possible plus an LLM judge (gpt-5.5, never a candidate model) with frozen prompts hashed into result files. GDPval scores use a rubric-anchored LLM judge and are not comparable to the paper's human pairwise win rates. An 8,192-token output cap truncated more gpt-5.6 deliverables than gpt-5.4 deliverables (luna 6, terra 9, sol 7, mini 0, nano 1 of 48); these results describe performance under that cap. DeepSearchQA n=50 (sol 49 after one transient API error) and GDPval n=48 per model; treat small gaps as directional. DeepSearchQA absolute F1 is sample-dependent: a harder 30-question extension lowered every model's score relative to the initial 20-question run, while preserving the observed ordering. Cost calculations use the price assumptions recorded in each result file.Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article