# Is GPT-6 Astra worth $10/$50? Two evals disagree, and both are right

> CodeRabbit measures Astra catching ~4% more bugs than GPT-5.6 Sol overall but 20% more on cross-file review. Artificial Analysis scores its general intelligence flat while the price went up 2.5x. Both readings hold, and together they say which tasks pay for the premium.

- Published: Sep 5, 2026
- Author: Linden Kern, Chief Scientist
- Tags: models, cost, coding-agents
- Canonical: https://pirouter.ai/blog/astra-eval-worth-the-price

---
Two evaluations of GPT-6 Astra landed within a day of each other and appear to contradict.
CodeRabbit found Astra catching about 4% more labelled bugs than GPT-5.6 Sol — but 20% more on the
harder cross-file subset. Artificial Analysis put Astra at 61 on its Intelligence Index against 66
for Claude Fable 5.1, and found that with the API price 2.5x Sol's, cost-efficiency on general work
went *down*. **Both readings are correct, because they are not measuring the same thing: one a hard
subset, the other an average across tasks. Averages dilute subsets. So Astra's $10/$50 is bought
back on work where the evidence is scattered, and wasted on work where it is not.**

This post keeps what OpenAI reports about its own model separate from what third parties measured,
because on a launch this loud the two get stirred into one table and the table gets screenshotted.
It ranks Astra against nothing. Prices are standard rates as of 2026-09-05; the billing picture is
in [same $10/$50, and a 39% cheaper agent bill](/blog/same-price-different-bill).

## Two evals, two numbers, and why they are not in conflict

CodeRabbit's measure is *actionable bug coverage*: of a set of labelled bugs, how many the model
surfaces as a finding a developer can act on. That is narrower than "is this model good at code
review", and CodeRabbit says so — the results "do not establish an overall ranking of review
quality, predict a team's defect rate, or promise the same gain on every pull request," and the
whole thing is called an early, directional result. I will keep repeating that, because the
interesting number is fragile. It is the gap between two rows of one evaluation:

| Reading | Who measured it | Astra vs GPT-5.6 Sol | Astra vs Opus 5 | Stated limitation |
|---|---|---|---|---|
| Actionable bug coverage, full evaluation | CodeRabbit (own eval) | ~4% more caught | ~22% more | "early, directional result"; includes simpler reviews |
| Actionable bug coverage, harder cross-file subset | CodeRabbit (own eval) | 20% more | 33% more | different review difficulty; not comparable to the row above |
| Intelligence Index, max reasoning effort | Artificial Analysis | 61 (Fable 5.1: 66) | — | composite across tasks |
| Cost per Coding Agent Index task | Artificial Analysis | $4.72 (Fable 5.1: $9.18) | — | one index, one harness |

All four rows are third-party measurements. Sources: [CodeRabbit — GPT-6 Astra in code review](https://www.coderabbit.ai/blog/gpt-6-astra-code-review-evaluation) for the first two; [Technspire — GPT-6 Astra in Foundry: the price, the gate and the EU gap](https://technspire.com/en/blog/gpt-6-astra-foundry-price-gate-eu-gap) reporting Artificial Analysis figures for the last two; the cost-efficiency conclusion also in [Asia Business Daily — GPT-6 Astra performance reviewed](https://www.asiae.co.kr/en/article/2026090415280432886).

Read down that table and the apparent contradiction dissolves. A composite index averages over a
task mix. If a model's gains sit in one region of that mix — work whose answer depends on relating
facts that sit far apart — the average reports a small number, and is not lying. The same dilution
happens inside CodeRabbit's own benchmark: the overall ~4% includes simpler reviews "where there may
be less room for a stronger model to differentiate itself." On the hard subset the gap is far wider
— CodeRabbit warns against dividing one figure by the other, the two charts being drawn from
different review difficulty, so take the direction and not a multiple.

The fourth row is where this stops being a debating point. Artificial Analysis scores Astra's
intelligence *below* Fable 5.1 and its cost per Coding Agent Index task at roughly half — $4.72
against $9.18. A lower index score and a lower cost per completed task, same outfit, same week.
Whatever that says about Astra, it is decisive about methodology: **an index position and a bill
are different quantities, and neither substitutes for the other.** Brockman argued the same from
the vendor's side — "what you actually want is the price per task" — and a claim can be
self-serving and still be true.

What the evaluation cannot tell you is why. CodeRabbit's own interpretation is careful: the
cross-file gains "suggest progress on work where relevant information is distributed" but "do not
isolate the cause of that progress." Astra's 1,050,000-token context is the same ceiling Sol has on
paper. Room for information is not the same as using it, and neither eval was built to tell those
apart.

## What OpenAI reports about its own model, kept in a separate table

Everything below is vendor self-reported, and belongs in its own table: these numbers came from the
party with the strongest interest in their being high, on benchmarks that party chose to publish.
That does not make them false. It makes them a different kind of evidence, and mixing the two into
one grid — as most launch-day coverage did — destroys the reader's ability to weight them.

| Benchmark (OpenAI self-reported) | GPT-6 Astra | GPT-5.6 Sol |
|---|---|---|
| OSWorld 2.0 (computer use) | 72.6%, 40 min per task | 65.7%, 75 min per task |
| DeepSWE v1.1 | 74.1% | not disclosed |
| Terminal Bench 4.0 (multi-step development) | 57.9% | not disclosed |
| FrontierMath Tier 4 v2 | 97.6% | 83% |
| GPQA Diamond | 96% | not disclosed |
| BenchCAD | 95.9% | not disclosed |
| ExploitBench (cyber) | 100% | 73.5% |
| ARC-AGI-3 | 98.6% or "up to 99.9%" — see below | 7.8% |

Compiled from [Pasquale Pillitteri — OpenAI launches GPT-6 Astra: pricing, benchmarks and who gets it first](https://pasqualepillitteri.it/en/news/14246/openai-launches-gpt-6-astra-pricing-benchmarks) and [Asia Business Daily — GPT-6 Astra performance reviewed](https://www.asiae.co.kr/en/article/2026090415280432886), both reporting the same launch-day table.

Note which row is lowest. Terminal Bench 4.0 measures multi-step development, and its 57.9% is the
only figure here under 60% — in the vendor's own selection. That is not a knock; it is the same
shape as CodeRabbit's 4%. Chaining many steps is where these models are weakest, and that tells you
more than the 97.6% above it.

The ARC-AGI-3 cell has two numbers in it on purpose. Pillitteri's roundup records 98.6%; Asia
Business Daily reports "up to 99.9%". Both cite the same day's announcement. I have no basis for
picking one, so both are here — choosing silently would invent a precision the record does not have.
Either way, a jump from Sol's 7.8% should make you ask what changed about the test harness and not
only the model, and neither source answers that.

Two absences matter more than any present number. **OpenAI published no GDPval result** — its own
benchmark for professional work, the one that would most directly support its economic-value
argument. And **no SWE-bench Pro**, the coding benchmark where Sol's own table had left Claude ahead,
and where Anthropic has since shipped Fable 5.1 and Mythos 5.1 with public numbers. The consequence
is worth stating flatly: *on official data, as of today, a direct coding comparison between Astra
and the current Claude models cannot be made.* Every launch-day headline that made one filled that
gap with something other than data.

Evals measure what someone chose to measure. Gaps in a published table are as informative as
the entries, and they are never accidental.

## The refusal bill: a Critical cyber rating that will stop legitimate work

Astra is the first OpenAI model classified at the **Critical** threshold for cybersecurity under
the Preparedness Framework — the system card says it "is a significant step up in cyber
capabilities and meets our Critical threshold." In testing it found two unreported vulnerabilities
and chained them into a working exploit. Offensive capability stays inside Daybreak Blue; what ships
to ChatGPT and the API is wrapped in classifiers and monitoring.

The alignment figures published alongside it are, on their face, the most impressive part of the
launch — refusing 91.5% of known jailbreaks against 59% for Sol, leaving an authorised perimeter in
0% of runs against Sol's 48.2%, never chasing an off-task target where Sol did so in 56%. Jakub
Pachocki's line — "progress in intelligence does not guarantee progress in alignment" — reads as a
company taking its own warning seriously. All self-reported, same category as the last section.

Here is the part for your cost model rather than your admiration. **OpenAI states that these
safeguards trigger on completely innocuous work.** On ChatGPT and Codex the user gets a review
prompt and can continue. *In the API, the task is simply terminated, without warning.* OpenAI says
launch friction is higher than it wants and promises to tune the filters.

Price that. An agent loop that dies at turn 40 of 60 has already burned its output tokens — the
$50-per-million side of the sheet — and returns nothing for them. Then someone has to work out
whether the model refused, crashed, or hit a rate limit. If your work sits near security tooling or
log forensics, this is a running tax, largest exactly where the model is most capable, and nobody
has published a rate for it.

## The denominator: what $10/$50 actually is

Astra's standard API rate is **$10 per million input tokens and $50 per million output tokens** as
of 2026-09-05, with Fast mode doubling both. That is 2.5x Sol's current promotional pricing, and the
same headline number as Claude Fable 5.1's. On OpenRouter the model spans a 4x range across five
endpoints, from OpenAI Flex at $5/$25 to OpenAI Fast at $20/$100
([OpenRouter — GPT-6 Astra providers](https://openrouter.ai/openai/gpt-6-astra)).

I am not expanding that into a price sheet; it is the companion post's job. What belongs *here* is
one fact about the denominator, because it changes the answer to "is it worth it" without changing
a capability number: **two models at the same list price are not at the same price.** Technspire's
worked 60-turn agent loop — 9M cache reads, 180K cache writes, 120K output — costs $17.25 on Astra
against $10.50 on Fable 5.1, because Anthropic cut cache reads to $0.25 per million while OpenAI's
cached input stays at $1.00.

Now put that next to the $4.72 against $9.18 from section one, because they point opposite ways.
Fixed at 60 turns, Fable 5.1 is 39% cheaper. Fixed at one completed task, Astra is about half the
price. Both are right; the difference is the unit: one prices work *offered* to the model, the other
work *finished* by it. If a model needs fewer turns, per-turn pricing flatters its competitor. Which
unit matches your budget is not something either measurement can tell you.

One boundary, since most readers weighing this run Sol, not Fable: every worked cost figure here
compares Astra to Fable 5.1. Against Sol this post has only a list-price ratio — 2.5x, on
promotional pricing with no published end date. By my own argument that is not a bill, and I have
not modelled the Sol one.

Which is why the two evals could disagree at all. Capability per dollar has a numerator set by task
shape and a denominator set by token shape; a leaderboard collapses both.

## The checklist: when Astra earns $10/$50, and when it does not

The pattern under every positive result above is the one CodeRabbit named: scattered evidence whose
parts depend on each other. That is a property of your task, not your industry. Below, ⚠ means test
it yourself; ✗ includes hard gates that rule Astra out for the teams they name.

| Verdict | Task shape | Why |
|---|---|---|
| ✓ | Cross-file code review; refactors whose consequences land far from the diff | The one measured gap: 20% more over Sol, 33% over Opus 5 (CodeRabbit, early and directional) |
| ✓ | Multi-step computer and browser operation | OSWorld 2.0 at 72.6%, roughly half the time per task (OpenAI self-reported) |
| ⚠ | Research synthesis, incident reconstruction from logs and runbooks, tracing a change across specs and policies | Same "scattered evidence" structure, but **CodeRabbit did not measure these** — candidates to test, not findings |
| ✗ | Single-file patches, formatting, classification, summarisation — anything a cheaper model does reliably | Paying 2.5x Sol for headroom the task never uses; where the cost-efficiency finding bites |
| ✗ | Cache-dominated long agent loops | Same list price, different bill — 39% more than Fable 5.1 on the 60-turn example |
| ⚠ | Security-adjacent legitimate work | The Critical rating's safeguards fire on innocuous requests and kill API tasks silently, per OpenAI |
| ✗ | Teams needing EU data residency, or that cannot wait out an approval | A gate, not a trade-off: Foundry ships it behind Limited Access, 5–10 business-day review, no EU Data Zone at launch (Technspire) |
| ⚠ | Document pipelines that land above 272K input tokens | Prompts past that line bill at double input and 1.5x output **for the entire request**, not just the overflow |

So: sort your workload into two piles by one question — does answering this require relating things
that sit far apart? Run Astra against your incumbent on the scattered pile only, on tasks whose
answers you can check, scoring by total cost per successful outcome rather than token price or index
position. On the other pile, skip the experiment.

I would say the same about the next flagship. What is unusual here is only that the two evaluations
landed close enough together to make the point visible, instead of one arriving six weeks late.

---

## Sources

### coderabbit.ai

- [CodeRabbit — GPT-6 Astra in code review](https://www.coderabbit.ai/blog/gpt-6-astra-code-review-evaluation)

### technspire.com

- [Technspire — GPT-6 Astra in Foundry: the price, the gate and the EU gap](https://technspire.com/en/blog/gpt-6-astra-foundry-price-gate-eu-gap)

### asiae.co.kr

- [Asia Business Daily — GPT-6 Astra performance reviewed](https://www.asiae.co.kr/en/article/2026090415280432886)

### pasqualepillitteri.it

- [Pasquale Pillitteri — OpenAI launches GPT-6 Astra: pricing, benchmarks and who…](https://pasqualepillitteri.it/en/news/14246/openai-launches-gpt-6-astra-pricing-benchmarks)

### openrouter.ai

- [OpenRouter — GPT-6 Astra providers](https://openrouter.ai/openai/gpt-6-astra)