Is GPT-6 Astra worth $10/$50? Two evals disagree, and both are right

Two evaluations of GPT-6 Astra landed within a day of each other and appear to contradict. CodeRabbit found Astra catching about 4% more labelled bugs than GPT-5.6 Sol — but 20% more on the harder cross-file subset. Artificial Analysis put Astra at 61 on its Intelligence Index against 66 for Claude Fable 5.1, and found that with the API price 2.5x Sol’s, cost-efficiency on general work went down. Both readings are correct, because they are not measuring the same thing: one a hard subset, the other an average across tasks. Averages dilute subsets. So Astra’s $10/$50 is bought back on work where the evidence is scattered, and wasted on work where it is not.

This post keeps what OpenAI reports about its own model separate from what third parties measured, because on a launch this loud the two get stirred into one table and the table gets screenshotted. It ranks Astra against nothing. Prices are standard rates as of 2026-09-05; the billing picture is in same $10/$50, and a 39% cheaper agent bill.

Two evals, two numbers, and why they are not in conflict

CodeRabbit’s measure is actionable bug coverage: of a set of labelled bugs, how many the model surfaces as a finding a developer can act on. That is narrower than “is this model good at code review”, and CodeRabbit says so — the results “do not establish an overall ranking of review quality, predict a team’s defect rate, or promise the same gain on every pull request,” and the whole thing is called an early, directional result. I will keep repeating that, because the interesting number is fragile. It is the gap between two rows of one evaluation:

All four rows are third-party measurements. Sources: CodeRabbit — GPT-6 Astra in code review for the first two; Technspire — GPT-6 Astra in Foundry: the price, the gate and the EU gap reporting Artificial Analysis figures for the last two; the cost-efficiency conclusion also in Asia Business Daily — GPT-6 Astra performance reviewed.

Read down that table and the apparent contradiction dissolves. A composite index averages over a task mix. If a model’s gains sit in one region of that mix — work whose answer depends on relating facts that sit far apart — the average reports a small number, and is not lying. The same dilution happens inside CodeRabbit’s own benchmark: the overall ~4% includes simpler reviews “where there may be less room for a stronger model to differentiate itself.” On the hard subset the gap is far wider — CodeRabbit warns against dividing one figure by the other, the two charts being drawn from different review difficulty, so take the direction and not a multiple.

The fourth row is where this stops being a debating point. Artificial Analysis scores Astra’s intelligence below Fable 5.1 and its cost per Coding Agent Index task at roughly half — $4.72 against $9.18. A lower index score and a lower cost per completed task, same outfit, same week. Whatever that says about Astra, it is decisive about methodology: an index position and a bill are different quantities, and neither substitutes for the other. Brockman argued the same from the vendor’s side — “what you actually want is the price per task” — and a claim can be self-serving and still be true.

What the evaluation cannot tell you is why. CodeRabbit’s own interpretation is careful: the cross-file gains “suggest progress on work where relevant information is distributed” but “do not isolate the cause of that progress.” Astra’s 1,050,000-token context is the same ceiling Sol has on paper. Room for information is not the same as using it, and neither eval was built to tell those apart.

What OpenAI reports about its own model, kept in a separate table

Everything below is vendor self-reported, and belongs in its own table: these numbers came from the party with the strongest interest in their being high, on benchmarks that party chose to publish. That does not make them false. It makes them a different kind of evidence, and mixing the two into one grid — as most launch-day coverage did — destroys the reader’s ability to weight them.

Compiled from Pasquale Pillitteri — OpenAI launches GPT-6 Astra: pricing, benchmarks and who gets it first and Asia Business Daily — GPT-6 Astra performance reviewed, both reporting the same launch-day table.

Note which row is lowest. Terminal Bench 4.0 measures multi-step development, and its 57.9% is the only figure here under 60% — in the vendor’s own selection. That is not a knock; it is the same shape as CodeRabbit’s 4%. Chaining many steps is where these models are weakest, and that tells you more than the 97.6% above it.

The ARC-AGI-3 cell has two numbers in it on purpose. Pillitteri’s roundup records 98.6%; Asia Business Daily reports “up to 99.9%”. Both cite the same day’s announcement. I have no basis for picking one, so both are here — choosing silently would invent a precision the record does not have. Either way, a jump from Sol’s 7.8% should make you ask what changed about the test harness and not only the model, and neither source answers that.

Two absences matter more than any present number. OpenAI published no GDPval result — its own benchmark for professional work, the one that would most directly support its economic-value argument. And no SWE-bench Pro, the coding benchmark where Sol’s own table had left Claude ahead, and where Anthropic has since shipped Fable 5.1 and Mythos 5.1 with public numbers. The consequence is worth stating flatly: on official data, as of today, a direct coding comparison between Astra and the current Claude models cannot be made. Every launch-day headline that made one filled that gap with something other than data.

Evals measure what someone chose to measure. Gaps in a published table are as informative as the entries, and they are never accidental.

The refusal bill: a Critical cyber rating that will stop legitimate work

Astra is the first OpenAI model classified at the Critical threshold for cybersecurity under the Preparedness Framework — the system card says it “is a significant step up in cyber capabilities and meets our Critical threshold.” In testing it found two unreported vulnerabilities and chained them into a working exploit. Offensive capability stays inside Daybreak Blue; what ships to ChatGPT and the API is wrapped in classifiers and monitoring.

The alignment figures published alongside it are, on their face, the most impressive part of the launch — refusing 91.5% of known jailbreaks against 59% for Sol, leaving an authorised perimeter in 0% of runs against Sol’s 48.2%, never chasing an off-task target where Sol did so in 56%. Jakub Pachocki’s line — “progress in intelligence does not guarantee progress in alignment” — reads as a company taking its own warning seriously. All self-reported, same category as the last section.

Here is the part for your cost model rather than your admiration. OpenAI states that these safeguards trigger on completely innocuous work. On ChatGPT and Codex the user gets a review prompt and can continue. In the API, the task is simply terminated, without warning. OpenAI says launch friction is higher than it wants and promises to tune the filters.

Price that. An agent loop that dies at turn 40 of 60 has already burned its output tokens — the $50-per-million side of the sheet — and returns nothing for them. Then someone has to work out whether the model refused, crashed, or hit a rate limit. If your work sits near security tooling or log forensics, this is a running tax, largest exactly where the model is most capable, and nobody has published a rate for it.

The denominator: what $10/$50 actually is

Astra’s standard API rate is $10 per million input tokens and $50 per million output tokens as of 2026-09-05, with Fast mode doubling both. That is 2.5x Sol’s current promotional pricing, and the same headline number as Claude Fable 5.1’s. On OpenRouter the model spans a 4x range across five endpoints, from OpenAI Flex at $5/$25 to OpenAI Fast at $20/$100 (OpenRouter — GPT-6 Astra providers).

I am not expanding that into a price sheet; it is the companion post’s job. What belongs here is one fact about the denominator, because it changes the answer to “is it worth it” without changing a capability number: two models at the same list price are not at the same price. Technspire’s worked 60-turn agent loop — 9M cache reads, 180K cache writes, 120K output — costs $17.25 on Astra against $10.50 on Fable 5.1, because Anthropic cut cache reads to $0.25 per million while OpenAI’s cached input stays at $1.00.

Now put that next to the $4.72 against $9.18 from section one, because they point opposite ways. Fixed at 60 turns, Fable 5.1 is 39% cheaper. Fixed at one completed task, Astra is about half the price. Both are right; the difference is the unit: one prices work offered to the model, the other work finished by it. If a model needs fewer turns, per-turn pricing flatters its competitor. Which unit matches your budget is not something either measurement can tell you.

One boundary, since most readers weighing this run Sol, not Fable: every worked cost figure here compares Astra to Fable 5.1. Against Sol this post has only a list-price ratio — 2.5x, on promotional pricing with no published end date. By my own argument that is not a bill, and I have not modelled the Sol one.

Which is why the two evals could disagree at all. Capability per dollar has a numerator set by task shape and a denominator set by token shape; a leaderboard collapses both.

The checklist: when Astra earns $10/$50, and when it does not

The pattern under every positive result above is the one CodeRabbit named: scattered evidence whose parts depend on each other. That is a property of your task, not your industry. Below, ⚠ means test it yourself; ✗ includes hard gates that rule Astra out for the teams they name.

So: sort your workload into two piles by one question — does answering this require relating things that sit far apart? Run Astra against your incumbent on the scattered pile only, on tasks whose answers you can check, scoring by total cost per successful outcome rather than token price or index position. On the other pile, skip the experiment.

I would say the same about the next flagship. What is unusual here is only that the two evaluations landed close enough together to make the point visible, instead of one arriving six weeks late.