MIT CSAIL proposes Weight-Space Programming: GPT-6 Astra modifies 0.016% of Qwen3.5-0.8B's weights…

Changing just 0.016% of the weights of Qwen3.5-0.8B with GPT-6 Astra fixes an infinite loop, boosting math ability by up to ~6x and instruction following by ~3x

Researchers from MIT CSAIL have discovered a new capability, "Weight-Space Programming": without gradients or writing training loops, a general frontier model (GPT-6 Astra) can directly read or modify the weight numbers of a neural network given a natural-language goal. Early results cover "recognizing behavior," "fixing faults," "writing small policy networks from scratch," and, after changing about 0.016% of the weights of Qwen3.5-0.8B, raising MATH-500 from 7.9% to 48.1%, with instruction following rising as well. The authors argue that small symbolic optimizers like Adam / Muon inject rough human priors, and that the next step may be LLMs deliberately reasoning about the weights themselves.

https://t.co/oaL4IN85s8

How is this different from current practice? People have long had LLMs modify prompts, harnesses, training code, and hyperparameters; the rationale is often that "text is hard to climb with gradients." Weights have always been left to differentiable optimization or specially trained "weight-writing" networks. This paper argues that reasoning-driven optimization has already reached into the weights of differentiable networks, sometimes even with no environment feedback at all.

Can read: guessing behavior from weights alone · Four PPO policies with the same structure (17,253 weights, 10×10 grid, same action set): Astra guessed from the raw weight text that they were Breakout / Pong / Pac-Man / Freeway respectively; it relied on things like which row the first-layer weights fall in and how the output layer splits actions. · Three small diffusion models (circle/square/triangle): without running a forward pass, it recognized the shapes from the weights; what it drew were outlines, suggesting it is more like reading geometric structure than secretly computing a generation. · Three 7.2M character-level language models, with the input and output layers removed and only the middle four Transformer blocks kept: it could still distinguish Python / romanized Japanese / English, and accurately flagged all copying (induction) heads without mislabeling other heads as this type, and in its reasoning it even arrived on its own at a weight-composition analysis similar to that of Elhage et al.

Can modify, can write · Kinetix uphill car: PPO was stuck in the "charge straight up the slope" local optimum; in a single reply Astra changed about 3.5% of the weights, raising the success rate from 0 to 100% across 1024 episodes, adding a bypass that reads position and velocity, reversing to build up momentum and then handing off to the original policy. · Three MinAtar games: with no data, no gradients, and no environment rollouts, it wrote the weights in one shot; the scores beat PPO trained for 100 million steps, and on Breakout it was an order of magnitude higher than PPO trained for 500 million steps. The idea is to encode a "program" into the layers: for example, first compute where the ball will land, then drive the paddle. · A 9×9 Go network with about 1.48 million parameters: the weights don't fit in context, so it used a Python sandbox to read and write tensors; with 0 forward passes it beat a smaller PPO, and with 50 forward passes (up to 128 games of practice per batch) it beat a same-scale PPO trained for about 20 million games. The approach resembles hand-writing small programs: liberties, dangerous chains, capturing/saving stones/forbidden eye-filling, then revising the corresponding parts based on practice failures.

Scaling up to language models · Sycophancy detection: two Gemma 3 270M models fine-tuned with the same setup (embeddings and output layers frozen); from the weights alone it could not tell which was sycophantic; after opening up "attention patterns on self-written text," it picked out the sycophantic one all three times, and noted that middle-layer heads focus more on the user's hinted answer. · Fixing the thinking infinite loop in Qwen3.5-0.8B: the model card itself warns that this setting easily loops. Astra worked on the weights in Python, using GSM8K/MATH practice problems for limited forward passes (10/25/50), located a certain direction written by the MLPs of layers 15 and 19, and subtracted it from each of their 64 output columns—changing 131,072 weights in total (0.016%). After 50 forward passes: MATH-500 7.9→48.1, GSM8K 13.8→67.1, IFEval 21.0→62.3, and the proportion hitting the token limit 93%→33%. · Control: on Gemma 3 270M, which doesn't loop much, the failures were ordinary miscalculations and illegal algebra steps, and no single obstacle could be found to twist off; small edits barely moved MATH-500. The authors point out that weight editing is more like removing an obstacle blocking existing ability, and is not yet equivalent to injecting abilities the model doesn't have out of thin air.

What they envision Auditing hidden capabilities or backdoors without running evaluations; one-shot initialization to lower training costs; steering/forgetting capabilities with little or even zero data while damaging the rest of the model less; recursive self-improvement shifting from "writing code that launches training" to "directly modifying weights"; grafting circuits across architectures rather than only averaging isomorphic ones. At the same time they acknowledge that context and harnesses are still clumsy, and that they are building benchmarks and harnesses to turn the "black magic" into reproducible engineering.