AWS Publishes an OpenAI-on-Bedrock Benchmark—and Luna Wins Its Cost-per-Outcome Samples

AWS published the benchmark on September 11

Amazon Web Services published the report on September 11, 2026 with authors from AWS and OpenAI. It evaluates GPT-5.6 Luna, Terra and Sol through Amazon Bedrock, then compares them with GPT-5.4 mini and nano through OpenAI’s API. The public repository provides the harness and timestamped result files, so readers can inspect more than a marketing chart.

This is benchmark news, not a new model launch. The report does not introduce a new model ID, API or region. The Bedrock IDs used by the harness are openai.gpt-5.6-luna, openai.gpt-5.6-terra and openai.gpt-5.6-sol on the bedrock-mantle endpoint; the direct OpenAI baselines are gpt-5.4-mini and gpt-5.4-nano. AWS says its common client uses the OpenAI Responses API shape for both paths.

The useful metric is cost per accepted outcome

The report’s central argument is practical: applications pay for successful work, including the failed attempts and repeated context needed to get there. Dividing total spend by accepted outcomes can reverse a ranking based only on price per million tokens. For agents, turn count matters because client-managed history is sent again as the trajectory grows.

That framing is sound, but “success” is not universal. AWS uses exact-answer checks for some benchmark questions, an F1 threshold for DeepSearchQA and weighted rubrics for GDPval. A support resolution, compliance memo or coding change needs a locally defined acceptance gate. Reuse the denominator; do not inherit the report’s definition of a passing task without checking it against your work.

Sol led the sampled accuracy tier

AWS reports that GPT-5.6 Sol answered 75% of 60 sampled AIME problems correctly, compared with 36.7% for GPT-5.4 mini. The pinned JSON records 45 correct Sol answers and 22 correct mini answers. AWS also reports Sol at about 68% on 198 GPQA Diamond questions versus about 43% for mini; the checked artifacts record 134 correct Sol answers with one API error and 86 correct mini answers.

Those figures show a large difference in these samples, not a universal model ranking. The models were not configured identically: AWS says reasoning was disabled for the Bedrock GPT-5.6 runs while the OpenAI API baselines used their defaults. Provider infrastructure and model generation also differ. The report correctly calls this a comparison of practical deployment configurations rather than intrinsic capability.

Luna recorded the lowest AIME cost per correct answer

The checked Luna AIME artifact contains 60 attempts, 23 correct answers, 38.3% accuracy and a repriced cost per success of $0.002092. The mini artifact contains 22 correct answers, 36.7% accuracy and $0.013929 per success. AWS therefore rounds the comparison to $0.0021 for Luna and $0.0139 for mini.

That result combines observed token use with a July 31 repricing transformation in the repository. The Luna artifact says costs were multiplied by 0.2 to reflect the July 30 list-price change while token counts stayed unchanged. This is transparent and auditable, but it is still a reconstructed bill under recorded assumptions—not a charge AccessAllGPT observed in an AWS account.

Agent turns changed the DeepSearchQA bill

For its 50-question DeepSearchQA slice, the Luna file records 6.18 mean turns, about 76,165 mean input tokens and 17 passes at the stated F1 threshold. Mini records 7.62 turns, about 114,072 input tokens and 11 passes. Dividing each arm’s recorded total cost by passes gives about $0.05 for Luna and $0.40 for mini.

The mechanism matters more than the ratio. The harness manages history with store: false, so prior prompts and tool output return on later turns. More search loops make the context—and the bill—grow. Teams evaluating research or retrieval agents should therefore report successful-task cost beside turns, input tokens, tool calls and wall time. A token-rate spreadsheet cannot reveal this trajectory effect.

GDPval adds a professional-work rubric—with a judge caveat

AWS ran a 48-task text-only GDPval slice. In the checked files, Luna passed 27 tasks at a 70% weighted-rubric threshold and mini passed 20. Repriced cost per pass was $0.010097 for Luna and $0.030068 for mini. The report also discloses truncation at an 8,192-token output cap, including six Luna outputs and no mini outputs in this slice.

The grading is not the official GDPval human pairwise protocol. A GPT-5.5 judge scored the candidates against rubrics. The repository explicitly says the values are only for within-run comparison and are not comparable with the paper’s win rates. Because the judge and every candidate are from the OpenAI family, possible family bias belongs in the result record even though no candidate graded itself.

Pricing is recorded, regional and mutable

The report’s result files use $0.22 per million input tokens and $1.32 per million output tokens for Luna, and $2.20 input and $13.20 output for Terra. AWS’s July 30 announcement confirms an 80% Luna reduction and a 20% Terra reduction effective that day, with Sol unchanged. It says the changes applied automatically to on-demand inference.

AccessAllGPT found Luna, Terra and Sol in the live Bedrock pricing page’s OpenAI model selector, but its initial server-rendered table showed a different default model rather than the selected GPT-5.6 rates. We therefore treat the exact numbers above as the benchmark’s recorded pricing assumptions, not an independently confirmed quote for every tier and region. Check the selected model, Standard/Priority/Flex/Batch tier, region and current billing page when calculating a migration.

Availability differs by endpoint and region

AWS’s July announcement lists Luna and Terra in US East (N. Virginia), US East (Ohio) and US West (Oregon) through the Responses API on bedrock-mantle. The repository notes that Sol was served from us-east-1 for the recorded run and tells users to list models for their own region. It also documents inference-profile IDs such as us.openai.gpt-5.6-luna for the separate bedrock-runtime path.

Do not turn a checked-in model name into an entitlement claim. Before a trial, list the models available to the target account and region, confirm the endpoint and exact ID, then make a minimal response call. Record service tier, quota, IAM path and cross-region behavior. A successful benchmark from another account does not establish your access or data-location posture.

Latency results are snapshots, not service levels

AWS says its July runs found median time to first token on Bedrock averaging 21% lower for Luna and 5% lower for Terra across 12 matched configurations, with Luna throughput averaging 43% higher at outputs of at least 500 tokens. It also labels the tests single-region, point-in-time measurements and warns that shared services vary with load.

The reported worst-case-to-median ratios are not p99 estimates, and Sol’s deep-reasoning behavior was measured in a different region. Treat the latency section as a reason to run the harness, not a latency promise. Repeat at representative concurrency, prompt sizes, output lengths, regions and hours; publish p50, p95, error and retry rates together.

The evidence is reproducible but not independent

The strongest part of the release is provenance. The repository records raw result JSON, timestamps, model and backend identifiers, token fields, judge settings and prompt hashes. AccessAllGPT retrieved commit d0e60ba1050be7cf880a4ca7b206c1442088d3aa and matched selected claims against its checked-in artifacts. The code and result trail make disagreement testable.

The ownership boundary remains important. AWS publishes and hosts the comparison, an author is identified as OpenAI staff, and all evaluated models come from OpenAI. The report compares newer GPT-5.6 configurations on Bedrock with older cost-focused GPT-5.4 baselines on OpenAI’s API—not the same model across providers for the quality ranking. It is high-value vendor evidence, not an independent certification of Bedrock or an all-provider leaderboard.

What buyers should do next

Use Luna as the first challenger when a high-volume or multi-turn workload currently runs on mini or nano and failures are measurable. Add Terra when the lower-cost model misses the quality gate. Add Sol only where accuracy is a hard requirement and the lower tiers fail. Keep the incumbent in the matrix so a migration must beat an observed baseline rather than a product description.

Run 50 to 100 frozen tasks with known-good outcomes, then calculate total spend divided by accepted results. Lock model ID, endpoint, reasoning effort, output cap, retry policy and grader before comparing. Review failed cases manually, rerun uncertainty-sensitive gaps, and reprice the saved token records when rates move. Adopt only if quality, tail latency, availability, governance and cost per accepted outcome pass together.

Disclosures

AccessAllGPT did not receive AWS or OpenAI credits, credentials, benchmark access, briefing, test data or compensation for this article. AWS and OpenAI did not sponsor, review or endorse it. AccessAllGPT Research is operated by NeuralArc, is independent, and is not affiliated with Amazon Web Services, Amazon, OpenAI, Google or organizations cited. Publication-wide relationships are listed on the disclosures page.