MIT researchers propose weight-space programming: GPT-6 Astra modifies about 0.016% of Qwen3.5-0.8B's weights…
MIT CSAIL researchers propose "weight-space programming," in which GPT-6 Astra directly reads and modifies the neural network weights of Qwen3.5-0.8B. By adjusting only about 0.016% (131,072) of the parameters, it raised the model's score on MATH-500 from 7.9% to 48.1% and on GSM8K from 13.8% to 67.1%, all without using gradient descent or a traditional training loop. Astra first located the repeated directions written by the model in the MLPs of layers 15 and 19, weakening the signals that caused reasoning loops, which reduced the proportion of math test responses left incomplete after hitting the 32,768-token limit from 93% to 33.1%, and raised the IFEval score from 21.0% to 62.3%. In the controlled experiments, because Gemma 3 270M originally exhibited few reasoning loops, its MATH-500 score rose only from 6.8% to 7.7%; based on this, the research team believes the method can remove obstacles to existing capabilities but may not necessarily create capabilities the model did not originally possess.
Why read this
AI score 60
The MIT CSAIL team had GPT-6 Astra directly rewrite a small number of Qwen3.5-0.8B's weights, bypassing gradient descent, and provided controlled experiments to illustrate the boundaries of this approach.
Topics
Body · AI translation
MIT Researchers Propose "Weight-Space Programming": GPT-6 Astra Modifies Only About 0.016% of Qwen3.5-0.8B's Weights, Raising Math Score to 6 Times the Original
2026-11-03 10:00 IT Home - Wen Zhou
IT Home, November 3: Researchers at the Massachusetts Institute of Technology Computer Science and Artificial Intelligence Laboratory (MIT CSAIL) have proposed a new method called "Weight-Space Programming."
They had OpenAI's GPT-6 Astra directly analyze and modify the neural network weights of Alibaba Cloud's popular Tongyi Qianwen model Qwen3.5-0.8B, adjusting only about 0.016% of the parameters to raise the model's score on the MATH-500 math benchmark from 7.9% to 48.1%, about 6.1 times the original. The entire process reportedly did not use gradient descent or a traditional training loop.
The research was completed jointly by Rikiya Takehi, Jayden Teoh, Alex Zhang, and Omar Khattab. Takehi published the research on his blog on November 2 local time, calling it "Weight-Space Programming." The core idea is to have a large language model directly read, understand, and modify the weights of another neural network, rather than adjusting model parameters through a conventional training process.
In the specific experiment, GPT-6 Astra did not simply adjust a small number of parameters at random; it first located the reasoning-loop problem in Qwen3.5-0.8B.
The researchers found that the model tended to repeatedly generate the same content when answering math questions, causing a large number of responses to exhaust the 32,768-token length limit. Of the initial 96 test responses, 85 hit this limit.
The research team provided GPT-6 Astra with a Python environment that could read and modify model weights, and allowed it to perform a limited number of forward inference passes using practice problems from the GSM8K and MATH training sets. Astra then located a repetition direction written by the MLPs (multilayer perceptrons) in layers 15 and 19, and by modifying some of the weights in these two layers, weakened the signal that caused the reasoning loop.
Ultimately, Astra modified 131,072 weights, accounting for 0.016% of Qwen3.5-0.8B's approximately 800 million parameters. In an experiment allowing 50 rounds of forward inference, the model's score on MATH-500 rose from 7.9% to 48.1%, and on GSM8K from 13.8% to 67.1%.
At the same time, the modified model's instruction-following ability also improved. On the IFEval test, which measures model instruction-following performance, the modified model's score rose from 21.0% to 62.3%, about 3 times the original. Meanwhile, the proportion of math test responses that failed to complete an answer because generation reached the length limit dropped from 93% to 33.1%.
It should be noted that this experiment was not entirely without computation. GPT-6 Astra used a Python environment to analyze and modify weights, and also performed up to 50 rounds of forward inference; it simply did not use gradient descent, backpropagation, or a traditional training loop to update parameters.
The researchers also conducted a control experiment using Gemma 3 270M. Because this model originally rarely exhibited reasoning loops, Astra could not significantly improve its math ability through similar local weight modifications; its MATH-500 score rose only from 6.8% to 7.7% at most. Based on this, the research team pointed out that weight-space programming can remove obstacles encountered when a model exercises capabilities it already has, but it cannot necessarily create capabilities the model did not originally possess out of thin air.
The research also includes other experiments, such as having GPT-6 Astra identify games the model was trained on solely by reading neural network weights, determine the training language of small language models, and directly write neural networks capable of running games. The researchers believe this method may provide new ideas for neural network interpretability research and model optimization.
Reference:
"Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights"
Advertising statement: External redirect links contained in the article (including but not limited to hyperlinks, QR codes, passwords, and other forms) are used to convey more information and save selection time; the results are for reference only. All articles on IT Home that contain external links include this statement.