# Weight-Space Programming: What the Model-Weights Experiments Actually Show

> A model identified tiny game policies from their weights; a separate Qwen edit used Python and feedback. Here is what each experiment establishes—and what neither proves.

- Published: Nov 4, 2026
- Author: Linden Kern, Chief Scientist
- Tags: models, opinion
- Canonical: https://pirouter.ai/blog/weight-space-programming-evidence

---
An LLM can sometimes infer what a small neural network does from a text dump of its weights. In a separate experiment, researchers used an LLM with Python and limited model feedback to edit a much larger language model and sharply reduce a failure mode. Those are interesting results from the [November 2 weight-space programming research post](https://rikiyat.github.io/blog/weight-space-programming/), but **they are not the same result**—and neither demonstrates general-purpose, autonomous improvement of language models without a training process.

The distinction became easier to miss after [Simon Willison ran one of the small-network tasks locally](https://simonwillison.net/2026/Nov/3/qwen-can-interpret-model-weights/) on November 3. His Qwen 3.8 27B identified three game policies and missed a fourth. The researchers' different target, Qwen3.5-0.8B, was the model whose weights their system *edited*. The similar names conceal different roles, model sizes and levels of access. Reading the toy network is where the strongest no-tools claim lives; editing the language model is where tools, practice examples and feedback enter the room.

## What can a model learn from 17,253 game-policy weights?

Imagine receiving a list of numbers rather than a playable game or an image. It describes a policy network that takes a grid as input and emits five possible actions. Could you determine which game that network was trained to play?

In the research team's first experiment, four policies had the same dimensions: a multilayer perceptron (MLP), a small feed-forward network, with 500 inputs, two hidden layers of 32 units and five outputs, or 17,253 weights per network. The policies had been trained with proximal policy optimization (PPO) on the team's own 10×10 grid versions of Breakout, Pong, Pac-Man and Freeway. Every network saw five channels—agent, moving objects, reward targets, obstacles and previous frame—and chose among no-op (do nothing), up, down, left and right. The researchers gave their interpreting model, GPT-6 Astra, the weights as text plus an explanation of the grid and actions, but not the game names or a tool for running the policies. According to the [experiment 1 setup and results](https://rikiyat.github.io/blog/weight-space-programming/), it named all four games correctly.

That is more specific than “it read a neural network.” The input has a known spatial layout. A weight from the agent channel near the bottom row can point to the part of a board where a Breakout paddle usually sits; patterns in the obstacle or reward channels can suggest a maze. The researchers describe concentration in first-layer weights and distinctions at the action outputs. **The model had no execution tool in this task, but it was not handed an unlabelled mystery architecture.** It knew what each channel and output action represented.

![Schematic of experiment 1: a known 10 by 10 grid and five labeled channels lead to a text-only weight inspection and game identification; the research team found four of four and a separate local reader found three of four.](/blog/images/weight-space-programming-evidence-fig-1.png "Known input labels and layout are part of this toy-policy task; the two four-game results are not general success rates. Original editorial schematic. Source: [research post](https://rikiyat.github.io/blog/weight-space-programming/) and [Willison's local run](https://simonwillison.net/2026/Nov/3/qwen-can-interpret-model-weights/).")

Willison tested the same four game-policy prompts using a *different reader*: Qwen 3.8 27B running on his laptop, with a sufficiently long context window. His [firsthand account and transcripts](https://simonwillison.net/2026/Nov/3/qwen-can-interpret-model-weights/) report Pac-Man identified in 46 minutes 12 seconds, Pong in 1 hour 4 minutes 54 seconds and Breakout in 52 minutes 25 seconds. It called Freeway a Pac-Man-like grid game instead. He notes that the default 8,192-token context setting failed; the long-context runs mattered to the result. His account is a useful independent *conditional demonstration* that the toy-task behavior is not exclusive to the research team's chosen reader. It is not a reproduction of their Qwen weight edit.

Three correct identifications in four local runs are not a success-rate estimate for arbitrary networks. The four policies share a deliberately constrained representation and a familiar family of games; a guess about a recognizable board is not a complete characterization of how the policy behaves. Even the local model's Pac-Man answer described an “8×8” maze, whereas the original setup describes 10×10 grids. Getting the game name right does not make every detail of the explanation correct.

## Is reading those weights beyond human analysis?

Sean Goedecke offers a less mystical interpretation. Because each input maps to a known grid cell and named channel, a researcher can reshape first-layer weights into a spatial view. His [counterpoint and illustrative heatmap method](https://www.seangoedecke.com/llms-are-not-superhuman-at-reading-and-writing-neural-network-weights/) use the magnitude of the 32 first-layer connections for each grid-and-channel position. A bottom-row player pattern or a maze-shaped obstacle pattern then looks less like an inexplicable insight into millions of opaque parameters and more like visual analysis of a small, structured policy.

That does not erase the no-tools result. In the research task, the interpreting model had to work from a long text representation without running even a short heatmap script; Willison's local reader spent substantial time reasoning through the numbers. Goedecke acknowledges that the mental calculation is impressive. His argument is narrower: the *method of extracting the clue* is accessible to a human with computation, so this experiment does not establish superhuman research ability. He did not publish a controlled human-versus-model trial. Both observations can stand: the model performed a constrained but nontrivial analysis, and the analysis has a conventional explanation.

Nor does the entire research post collapse into “look at the first layer.” In [experiment 3](https://rikiyat.github.io/blog/weight-space-programming/), the team removed input and output layers from three small character-level transformers and let Astra use Python to analyze only the middle blocks, without running those blocks on text. They report that it identified whether the networks were trained on Python, English or Japanese written in romaji, and located copying attention heads. This is a different setup with tools for analysis but no forward passes; it is not explained by spotting a paddle on an input grid.

The same post supplies a useful counterexample. In [experiment 7](https://rikiyat.github.io/blog/weight-space-programming/), Astra could not distinguish two fine-tuned Gemma 3 270M models' sycophantic versus factual behavior from weights alone in four attempts. Only after the team allowed access to attention patterns on model-written texts did it identify the sycophantic model in their tested pairs. The extra access changes the evidentiary claim. A headline that says models can diagnose hidden behavior “just from weights” cannot use that later success without mentioning the added measurements.

## The Qwen3.5-0.8B edit used a different kind of access

The larger editing result is the one most likely to be retold as “an AI improved another AI without training.” Here the target is **Qwen3.5-0.8B**, a roughly 0.8-billion-weight reasoning model—not Willison's Qwen 3.8 27B reader. The research team gave Astra a Python session over the target's weights, practice problems from the GSM8K and MATH training sets, and budgets of 10, 25 or 50 forward passes—runs of the target model used to inspect its responses. One pass could answer a batch of problems or read a batch of texts. The test sets used for evaluation were held out from Astra, and the authors prohibited gradient descent and training loops. Those restrictions matter; so do the tools and feedback that were allowed ([experiment 8 setup](https://rikiyat.github.io/blog/weight-space-programming/)).

The authors observed a specific defect: the base target often looped in its reasoning until it reached a 32,768-token limit. They report 85 of the first 96 practice answers reached that limit. Astra investigated a direction associated with looping in the MLPs of layers 15 and 19 and subtracted it from 64 output columns in each layer. The reported edit covered 131,072 weights, or about 0.016% of the model. It is an intervention in weights rather than a gradient update, but it was conducted inside an instrumented investigation, not from one unaided glance at a raw parameter dump.

![Two experiments have different access: text-only reading of a 17,253-weight toy game network versus Python and up to 50 forward passes to edit Qwen3.5-0.8B; Qwen 3.8 27B was a separate local reader.](/blog/images/weight-space-programming-evidence-fig-2.png "Reading a toy policy without tools and editing Qwen3.5-0.8B with Python and bounded feedback are separate demonstrations. Original editorial diagram. Source: [research post](https://rikiyat.github.io/blog/weight-space-programming/) and [Willison's local run](https://simonwillison.net/2026/Nov/3/qwen-can-interpret-model-weights/).")

With the 50-pass budget and the authors' stated sampling settings, the research post reports gains on two math test sets: **MATH-500 rising from 7.9% to 48.1%**, and GSM8K from 13.8% to 67.1%. On IFEval, an instruction-following evaluation, the reported score rose from 21.0% to 62.3%. The share of MATH-500 answers hitting the token limit fell from 93.0% to 33.1%. These are the *researchers' evaluation numbers*, not a test by this publication or a replication by Willison. They show a large measured change in this configuration; they do not establish that the method is cheap, reliable across models, or a replacement for training. The source also specifies multiple samples per problem and particular decoding settings, so a number detached from that setup is easy to misread.

## Did the edit teach math, or let the model finish?

The distribution across feedback budgets is more informative than the largest percentage alone. At 10 and 25 passes, the authors report MATH-500 scores of 12.3% and 14.1%, while 87.2% and 86.4% of answers still hit the limit. At 50 passes, the score reaches 48.1% and the limit-hit share drops to 33.1%. This is consistent with an edit that suppresses a costly looping behavior. It is not, by itself, proof that the edit created new mathematical reasoning ability.

The researchers make much the same distinction in their own [additional low-looping model experiment](https://rikiyat.github.io/blog/weight-space-programming/). Gemma 3 270M rarely reached its token limit in their setup—8.5% of answers—and Astra's edits did not establish a comparable math gain. Its MATH-500 base was 6.8%; reported edited scores ranged from 7.0% to 7.7% at the tested budgets. The authors describe these small changes as unestablished improvements and contrast removing an obstruction with adding an ability a model does not already have.

Goedecke argues that the Qwen procedure resembles a familiar weight-steering or abliteration playbook: compare looping and non-looping behavior, find an associated direction, edit contributing weights, then check the outcome. That is a substantive challenge to calling the process a radically new route to model improvement. It is also an interpretation of the researchers' workflow, **not** an independent rerun of their benchmark or proof that a particular human would have produced the same edit on the same budget. The stronger reading of the result is useful enough without either extreme: a general-purpose model participated in a targeted, tool-assisted weight-level repair; the study has not shown that this approach reliably manufactures missing skills.

## What should an AI developer take from weight-space programming now?

For a small policy with labeled spatial inputs, raw weights may reveal interpretable structure even without executing the network. For a larger language model, a model-driven investigation may use code and limited feedback to isolate and change a particular failure mode without running a gradient-based training loop. **The access conditions are part of the result, not a footnote.** The first finding should not inherit the large model's scale; the second should not inherit the first's “no tools” property.

If a team wants to evaluate the broader claim, the next useful comparison is not another striking single score. Test unseen architectures and failures, include a human-researcher or conventional steering baseline with comparable access, report tool calls and compute, repeat with different random seeds, and measure regressions on tasks unrelated to the target. Those are proposed evaluation criteria, not experiments we performed. The authors say they are building a standardized benchmark; until such evidence is available, the practical conclusion is narrower than “models now improve themselves.” It is that **weight-level debugging deserves careful study**, especially when a discrete failure can be isolated—and that the line between reading numbers, running analysis and learning from feedback must remain visible.

---

## Sources

### rikiyat.github.io

- [November 2 weight-space programming research post](https://rikiyat.github.io/blog/weight-space-programming/)

### simonwillison.net

- [Simon Willison ran one of the small-network tasks locally](https://simonwillison.net/2026/Nov/3/qwen-can-interpret-model-weights/)

### seangoedecke.com

- [counterpoint and illustrative heatmap method](https://www.seangoedecke.com/llms-are-not-superhuman-at-reading-and-writing-neural-network-weights/)