Claude Code vs mini-SWE-agent: similar scores in one study, different bills

Claude Code and mini-SWE-agent were statistically equivalent within a five-percentage-point margin on the two Qwen models in one study’s large task pool. Separate experiments on DeepSeek and GLM found clearer differences in the bill, with reported costs varying by about threefold across harnesses—the software that supplies tools, manages context and decides when to stop—on the same model. These findings do not establish that every harness performs the same—or that the cheapest run delivers the same work.

The paper, What Does a Harness Buy? Tokens, Mostly, is useful precisely because it measures more than a leaderboard score. It holds the model fixed, changes the surrounding software and repeats some configurations to see how much a result moves without any engineering change. The practical question is not whether a larger harness looks more capable. It is whether its extra machinery produces a benefit distinguishable from another run of the same setup.

The large comparison covered two models, not the whole ladder

The arXiv submission record dates the paper’s first version to October 3. The Decoder covered the harness study on October 24; that later discussion is not a new experiment or a new paper version.

The researchers compared Claude Code, mini-SWE-agent and OpenCode. mini-SWE-agent supplies a minimal Bash-based interface without the context-management machinery of the other two.

Five models were tested across all three harnesses: Qwen3.6-35B-A3B, Qwen3.8-27B, DeepSeek-V4-Flash release 0731, GLM-5.3-Flash and HY4-Preview. Claude Opus 5 ran only inside Claude Code. Its score is therefore not evidence of what Opus would do in the other harnesses.

The task sets also need separating. From SWE-bench Verified’s 500 tasks, the researchers removed eight whose verification needed a public web service. They used the 45 tasks in the two hardest annotated difficulty bands as one set, and the remaining 447 as another. The large pool covered only the two Qwen models. It was not a 447-task comparison of all five models.

These were specific software versions, not timeless product identities. The scored configurations used Claude Code 2.1.259–2.1.273, mini-SWE-agent 2.4.6 and OpenCode 1.18.30–1.18.31. Network access was restricted for the scored agent runs; early web-enabled runs were discarded after agents retrieved upstream fixes. OpenCode’s search dependency was preinstalled for the main reported configuration. The result describes a prepared benchmark environment, not an unmodified install in an everyday connected repository.

Equivalent within five points is not identical

On the large pool, the reported Claude Code-minus-mini-SWE-agent differences were −1.8 percentage points for Qwen3.6 and −1.4 for Qwen3.8. Paired comparisons used the tasks with usable results on both sides, so their denominators were smaller than the nominal 447. Infrastructure failures were excluded; ordinary failures caused by exhausting context, steps or time remained failures.

The authors did more than fail to find a significant difference. They tested whether the difference fit inside a margin of five percentage points. Claude Code and mini-SWE-agent met that criterion on both Qwen models.

That is evidence of equivalence at the chosen margin, not proof of identical performance. A team that considers a one-point difference important has a stricter decision than this result resolves. A five-point bound also does not mean five percent less work: percentage points describe the difference between success rates, not a relative discount on one of them.

The hard set offers a different lesson. Forty-five difficult tasks sounds demanding, but difficulty does not create statistical precision. The paper reports that, at the observed level of disagreement between runs, this sample would detect a roughly 13-point gap only about half the time. Smaller differences could easily remain unresolved.

Repeated configurations also changed which individual tasks passed. Some wins disappeared on the next run, and the apparent ordering of harnesses was unstable. Task-level flips can go in both directions, so the share of changed outcomes is not the same as a net change in the overall score.

This is why “no significant difference on 45 tasks” cannot do the work of an equivalence test. One result says the experiment could not separate the estimates clearly enough. The other says the uncertainty fits inside a stated tolerance. Choosing a tool on the strength of the first while assuming the second is a much stronger claim than the data support.

OpenCode’s losses are the counterexample to “harnesses do not matter”

The paper does find differences that clear the noise. OpenCode trails on the large task pool, including gaps of roughly eight to nine percentage points on Qwen3.8. Those two Qwen3.8 comparisons survive the authors’ multiple-comparison correction. A broad claim that harnesses never affect results would erase one of the study’s clearest findings.

The researchers identify output-limit handling as one associated failure mode. Some OpenCode runs ended when a model response hit its output cap, while Claude Code could ask the model to continue. Removing affected tasks from a comparison shrank part of the gap, but that is not a controlled demonstration that adding one recovery step fixes the product. A small transplant experiment did not establish a statistically clear improvement.

There are other confounds. Although every trial received a ceiling of 300 model calls, wall-clock budgets were not equal on the locally served Qwen configurations. Claude Code could run for 36,000 seconds, while OpenCode’s own soft timeout bound it at 5,760 seconds. The paper reports affected hard-set failures and treats the asymmetry as uncontrolled, rather than presenting it as a clean harness advantage.

The minimal harness had failure modes too. Without context management, long trials could exhaust the model’s window. Several such tasks also failed under the other tools, so the existence of an overflow does not prove that automatic compaction would have rescued them.

The useful reading is not “complexity is wasted.” It is that extra machinery needs a task-specific explanation: which loss does it prevent, under which limit, and does that change survive a fair comparison? A smaller system can omit overhead; it can also omit a recovery mechanism that matters on your workload.

The bill starts with the prompt, then grows with the steps

The paper’s monetary analysis primarily uses DeepSeek-V4-Flash and GLM-5.3-Flash. Costs are reconstructed from each trial’s cache-hit input, cache-miss input and output using the vendor’s model price sheet. The locally served Qwen configurations have token counts, not comparable hosted-service bills. These are research cost calculations, not this publication’s paid invoices or a current subscription comparison.

For GLM-5.3-Flash, the paper reports the following cost per trial:

The authors explain the difference through two quantities: how much input a harness carries into each step, and how many steps the model takes. In the analyzed traces, the system prompt, tool schemas and message wrapper contributed about 16,600 tokens for Claude Code, 7,000 for OpenCode and 800 for mini-SWE-agent before the task’s growing history was counted. Those are measurements for the tested configurations, not permanent specifications of the products.

A small prefix does not guarantee the smallest bill. On DeepSeek, mini-SWE-agent took more steps, accumulating more history to read again. Its lightness per step did not translate into a clear cost advantage over Claude Code. The model’s interaction with the harness changed the accounting.

Conversely, a short run can look cheap because it stops early. The study associates OpenCode’s lower costs with both lighter per-step input and earlier stopping. If some of those shorter runs lose tasks, saving tokens is not the same as delivering work efficiently.

There is also a tempting comparison the study does not license: take the similar Qwen success rates from the large pool and divide them by the GLM costs above to announce a universal cost-per-success winner. They are different models and experiment groups. The success rate, task set, stopping conditions and cost need to belong to the same comparison before they can support that decision.

Choose a harness by the failures you can observe

The authors’ data and analysis repository describes a way to regenerate aggregate analysis. That is different from rerunning the agents. The available materials did not include every original trajectory and environment component when checked, and no inference or statistical recomputation was performed for this article.

The scope is limited in other ways too. Most evidence comes from one code-repair benchmark. A small Terminal-Bench check does not cover general-purpose agents, long connected workflows or every model. The closed frontier reference did not cross harnesses. Nor does a code-repair pass rate measure review ergonomics, permission controls or whether a developer can understand a proposed change.

For a team considering a switch, the strongest idea to borrow is the repeat, not the winner. Hold the model and task conditions steady, run the same configuration again, and compare the proposed improvement with the movement that occurs without a change. Keep the success-rate uncertainty beside the price and record why each run stopped.

Then inspect the bill’s structure: the first request’s size, history growth, number of steps and cache treatment. A lower average is more useful when it survives successful completion than when it comes from an early cutoff. A heavier harness is easier to justify when it demonstrably prevents a failure your work encounters.

The study makes the burden of proof clearer. A polished tool surface is not evidence of extra solved tasks, and a minimal implementation is not evidence that nothing valuable was removed. Choose the machinery whose benefit you can observe on the work you need done, at a cost measured on those same runs.