MIT 团队提出 Weight-Space Programming，让 LLM 直接读写神经网络权重

Researchers at the Massachusetts Institute of Technology (MIT) have shown that large language models (LLMs) can read, interpret, and modify neural network weights directly, without relying on traditional training methods like gradient descent.

In a research blog post published on November 2, the researchers introduced a technique called Weight-Space Programming, which lets general-purpose LLMs examine a neural network’s weights, identify problems in its behaviour, and modify them based on natural-language instructions.

The research was conducted by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab, all affiliated with MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL).

The researchers used OpenAI’s GPT-6 Astra to demonstrate the approach across eight experiments, including identifying what a neural network was trained to do, writing networks from scratch and improving the mathematical reasoning capabilities of an existing language model.

“We find that frontier LLMs are surprisingly good at natively reading and writing neural network weights, well past what we'd expect from humans or conventional algorithms,” the researchers wrote.

One key experiment involved improving Qwen3.5-0.8B, a small language model, without retraining it or using gradient information. Astra identified a problem in the model’s weights that caused it to repeat reasoning steps instead of producing answers.

By modifying just 0.016% of the model’s weights, Astra increased its score on the MATH-500 benchmark from 7.9% to 48.1%. Its score on GSM8K, another mathematical reasoning benchmark, increased from 13.8% to 67.1%.

The researchers also found that the changes improved the model’s instruction-following performance. Its IFEval score increased from 21% to 62.3%.

In the first set of experiments, the researchers tested whether Astra could identify a network’s behaviour by examining its weights without running the network.

They trained four neural networks using Proximal Policy Optimisation (PPO) to play Breakout, Pong, Pac-Man and Freeway. All four networks had the same architecture, input channels and possible actions.

Astra received the 17,253 weights of each network as text, without being told which game it had been trained on. It correctly identified all four games.

In another experiment, the researchers tested three diffusion models trained to generate circles, squares and triangles. Astra identified the shape each model generated by examining its weights, without running the models or performing forward passes.

The researchers then tested whether an LLM could create neural networks without training them.

They gave Astra the rules and source code for three MinAtar games: Breakout, Seaquest and Asterix. The model had no access to training data, loss functions, gradients or tools. It had to generate the network weights directly from the game descriptions.

The resulting networks outperformed PPO-trained networks that had undergone 100 million training steps in all three games.

In Breakout, Astra’s network scored 517.4, compared with 15.8 for a PPO network trained for 100 million steps and 25.3 for one trained for 500 million steps.

In Asterix, Astra scored 53.2 against 55.2 for the 500-million-step PPO network. In Seaquest, it scored 53.1, compared with 88.9 for the same PPO benchmark.

The researchers also tested Astra on the board game Go using a network with nearly 1.5 million weights.

With no forward passes, Astra generated a network that won more than half its games against two smaller PPO networks. When allowed 50 forward passes to examine its performance, the generated network beat an equally sized PPO network trained on 20 million games.

For this experiment, Astra used a Python sandbox to write and analyse the weights. It could not use the internet, train the network or use tree search.

The researchers believe the technique could change how neural networks are optimised and audited.

The researchers also point to recursive self-improvement, where an LLM could inspect and modify its own weights or those of another model.

They are developing a benchmark and testing framework to evaluate how well LLMs can read and write neural network weights.

“We believe this marks the beginning of a dramatic shift in fields like mechanistic interpretability and neural network optimisation,” the researchers wrote.

The team has also released the experiment logs, including the prompts, model responses, code and evaluation results, on GitHub. The edited Qwen3.5-0.8B model is available on Hugging Face.