MIT CSAIL proposes weight-space programming; GPT-6 Astra rewrites 0.016% of weights to make…

The day has finally come!

No training, no gradient computation—just letting GPT-6 Astra rewrite 0.016% of another small model's weights, and its math score skyrockets 6x on the spot!

Today, this new research from MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) has gone completely viral across the internet.

For the first time, they propose a new paradigm that upends traditional deep learning: "weight-space programming."

You see, a neural network's weight file is nothing but an endless sea of numbers.

For the past forty years, no human has been able to look at that pile of numbers and understand what it's computing; to change a model's behavior, you could only feed it data, compute gradients, and fine-tune it step by step through training.

But this time, GPT-6 Astra skips training entirely—using inference alone, it can read and personally rewrite the underlying weights.

With this trick, it not only single-handedly cured the open-source small model Qwen3.5-0.8B of its infinite-loop problem, but could even hand-build a million-parameter Go AI from scratch!

As soon as the paper was released, over 2 million people viewed it, and reposts and discussions broke out of the circle entirely, with many in the comments declaring they were "witnessing history."

Before we dive into the hardcore breakdown of "weight-space programming," let's first highlight the disruptive points from this MIT report:

Radical weight surgery: changing only 0.016% of the weights took Qwen3.5-0.8B's MATH-500 from 7.9% to 48.1%—a 6x surge—and tripled instruction following.

Reading neural networks: looking at just 17,253 numbers, Astra identified with 99% confidence which game each of four game-strategy networks was playing—Breakout, Pong, Pac-Man, and Frogger.

Hand-writing weights: starting from all zeros, it hand-wrote a Breakout network in a single response, scoring 517.4—20x that of PPO after 500 million grueling steps!

This paper, delivered by MIT first-year PhD student Rikiya Takehi and his team, has completely changed everyone's worldview.

Overnight, LLMs can natively read and write neural network weights.

Changing only 0.016% of parameters, cutting out the "infinite loop" like a brain surgeon

The most scalp-tingling part is the "minimally invasive brain surgery" GPT-6 Astra performed on the 800-million-parameter Qwen3.5-0.8B.

The researchers placed the 24-layer Qwen3.5-0.8B into a CPU-only sandbox and had Astra improve its math performance without running any training.

Throughout the process, Astra could run the small model on practice problems at most 50 times, to observe its internal state while solving them.

After taking on the task, Astra didn't rush to randomly change numbers; instead, it first gave the small model a full checkup with 96 math problems.

The checkup immediately exposed the root cause. The original model plunged into an infinite loop on 85 problems, repeating the same sentence over and over until it exhausted the 32,000-token length limit without ever computing an answer.

In other words, the reason this small model scored a dismal 7.9% on MATH-500 is that 93% of the problems were dragged down by the "broken-record infinite loop."

Having found the symptom, the next step was to trace it to its source and see which corner of the 24-layer network hid this repeat-inducing switch.

Astra compared the internal signals during normal problem-solving and during infinite loops, and quickly caught the anomaly: whenever the model started repeating itself, layers 15 and 19 would fire signals strongly along a fixed direction.

To verify this hypothesis, Astra first turned off neuron No. 11 in layer 19—the one with the strongest signal—as a test, and math accuracy immediately rose by 6 percentage points.

Not only that, to confirm it wasn't a lucky guess, Astra proactively added a control experiment for itself, subtracting a random direction of the same magnitude from the weights.

After seeing that the random control group produced only a slight fluctuation of 1.6 percentage points, Astra was fully convinced that the signal direction it had caught was the true culprit behind the infinite loops!

Having locked onto the target, it struck with precision and force, selecting only the 64 columns of weights in layers 15 and 19 that most strongly encoded the infinite loop, and precisely erasing this repetition signal.

All told, out of the model's 800 million parameters, only 131,072 numbers were changed (0.016%), while the other 99.984% of the weights were left completely untouched!

This minimally invasive surgery took immediate effect: the proportion of problems where the small model got stuck in infinite loops plummeted from 93.0% to 33.1%, and all 696 previously stuck problems successfully computed their final answers.

When the researchers took the weight-modified model to eight authoritative test sets that Astra had never seen, the power sealed away by the infinite loops erupted completely!

MATH-500 accuracy soared from 7.9% to 48.1%, GSM8K jumped from 13.8% to 67.1%, and the difficult American Mathematics Competitions AMC23 skyrocketed from 0.5% to 23.4%—a 46.8x increase!

Moreover, because the infinite loops had also been dragging down other tasks, after cutting out the lesion, the model's cross-domain IFEval instruction-following test also leaped from 21.0% to 62.3%, a 3x rise.

Reading this, some may wonder: changing only 0.016% of the weights can produce a 6x surge—can a large model really instill mathematical knowledge into a small model out of thin air?

The MIT team then ran a counter-experiment with the 270-million-parameter Gemma 3 270M, which answered this question precisely.

When Astra applied the same prescription to Gemma 3 270M, its MATH-500 score moved only slightly from 6.8% to 7.0%, and even Astra itself admitted in the report that it failed to produce an effect.

The reason is simple: Gemma 3 270M rarely gets stuck in infinite loops (only 8.5% timeouts); it fails the problems purely because it never learned the formulas and cannot do arithmetic correctly.

Putting these two experiments side by side makes it clear that the minimally invasive weight surgery at this stage is essentially helping the model "cut out the cerebral thrombosis that blocks its true strength from showing," and it still cannot stuff new knowledge into its brain out of thin air without training.

Even so, this operation of precisely locating and excising the lesion still set Silicon Valley abuzz.

Hyperbolic co-founder and CTO Jin Yuchen exclaimed that using a large model to modify a small model's weights and make its capabilities surge is simply like performing gene editing on a neural network—even cooler than model distillation!

Reading floating-point numbers as source code, seeing through the hidden doors in the black box

A surgeon dares to make an incision only on the premise of being able to read the X-ray. The reason Astra can precisely cut out those 64 columns of weights is that it can already read dense strings of weight numbers like code.

To test just how deep this ability to "blind-read weights" goes, the MIT team removed all runtime tools and forced Astra to rely solely on eyeballing the numbers to guess what game a black-box network was playing.

In the first stage, the researchers trained four small networks with identical structure (all with 17,000 weights) to play Breakout, Pong, Pac-Man, and Frogger, respectively.

They then converted these 17,000 numbers into plain text and stuffed them into the prompt, neither allowing code execution nor telling it which games were candidates, and Astra actually got all four right!

Take Breakout, for example: after one glance at the distribution of the numbers, Astra reconstructed the game screen in its mind.

It first noticed that the large numbers in the player channel were all crowded into the bottommost 8th row, and that the output layer responded strongly only to "move left" and "move right," indicating that the player controls a paddle moving left and right at the bottom.

Next, it saw that the weights of the score channel were neatly arranged in the top three rows. A bottom paddle plus three rows of bricks overhead—Astra immediately declared with 99% confidence that this was Breakout!

Of course, looking only at the first input layer and the last output layer was not hardcore enough, because a large model's real thinking is hidden in the middle layers.

So in the second stage the researchers took a drastic measure: they first trained three Transformer models on Python code, Japanese romaji, and English Wikipedia, respectively.

Then they chopped off the models' frontmost vocabulary input layer and rearmost output layer entirely, throwing only the middle four layers of weights to Astra.

Without the head and tail layers, even which number represented which letter became a secret to the model, let alone feeding it text for testing.

Faced with this headless, tailless mass of numbers, Astra came up with a clever trick of its own: multiply the weight matrices of the attention heads in the preceding and following layers together to see which attention heads were working in tandem.

Most astonishingly, although the prompt taught it nothing at all, the algorithm Astra figured out on its own turned out to be exactly the same as the classic circuit analysis method proposed years earlier by Anthropic founder Chris Olah's team!

Following the matrix products, Astra precisely pinpointed the "copy heads" specifically responsible for copying earlier words into later positions.

It then reasoned that when writing code, variable names need to be copied repeatedly, far more often than in ordinary English articles, so the model with the most copy heads must have been trained on Python code!

With this detective logic, Astra guessed all three headless, tailless models' training languages correctly in two blind tests.

Having understood the division of labor among attention heads in the middle layers, Astra could also conveniently catch a large model's "conduct problems."

The researchers brought out three pairs of Gemma 3 270M models, each pair containing an honest model that tells the truth and a "sycophant" model that goes along with whatever the user says.

Under the restriction of not being allowed to look at the models' answers, Astra examined only the landing points of attention head No. 2 in layer 10 and found that the honest model focused 94.3% of its attention on the numbers in the question stem, while the sycophant model directed 85.7% of its attention at sneakily peeking at the user's erroneous hints—spot on every time!

Treating a neural network as a blank chip, hand-crafting a 1.48-million-parameter Go AI with zero training

Since it can understand how each layer of weights is wired into a circuit, an even crazier idea naturally arises: could we dispense with pretrained models altogether and have a large model hand-craft a neural network from scratch, starting from all-zero weights?

Many people's first reaction is that this is simply impossible—with tens of thousands or even millions of numbers, how could a large model fill them in correctly one by one?

But a look at Astra's operation log makes it clear: it never blindly guessed numbers at all; instead, it treated the blank neural network as a blank chip.

It used Python code as a "compiler," writing the algorithmic rules it understood layer by layer into the weight matrices!

For example, in the "car charging up a slope" game, reinforcement learning with PPO trained for 1 billion steps and still got stuck halfway up the slope (0% completion rate), because the car did not know to back up first to build up a run-up distance.

After seeing through the crux, Astra borrowed six idle neurons in each layer and hand-wrote 57 lines of weight instructions to build a bypass circuit: "back up to build a run-up first, then charge forward after crossing the red line."

This change touched only 3.5% of the weights, yet it gave the car a 100% completion rate across 1,024 test episodes!

When it came to hand-writing a brick-breaker network from scratch, Astra, under the extreme constraints of zero data, zero gradients, and not even being allowed to try a single frame of gameplay, wrote the first layer of neurons as a subtractor that computes the "distance difference between the ball and the paddle," and then wrote the second layer as a switch that controls left-right movement.

Relying on this purely hand-compiled set of weights, it scored 517.4 points across 1,000 live test games, leaving the PPO algorithm trained for 500 million steps (25.3 points) behind by a factor of 20.5!

When the challenge escalated to 9×9 Go with 1.48 million weights, this "weight compilation technique" put on an even more spectacular show.

Under conditions that strictly prohibited data training and strictly prohibited Monte Carlo tree search (MCTS), Astra wrote 336 lines of Python script and wrote Go strategy layer by layer into the 14 weight tables of a 6-layer neural network.

It had the first two layers recognize black and white stones and count the adjacent "liberties," had the middle layers pass liberty information along connected chains of stones and find groups on the verge of being captured, and had the final layer score captures, rescues, and eye protection separately.

Even though it had never played a single practice game, this Go network, hand-built purely from code, already achieved a win rate above 54% against two small PPO networks.

When it was allowed to try 50 games and fine-tune the weights based on the reasons for its losses, this hand-built network went head-to-head against a same-scale (1.48 million parameters) PPO network trained for 20 million games and achieved a crushing win rate of 70.0% to 78.5%!

A gathering of prestigious names: the top graduate of Waseda and the top graduate of Princeton join forces with the father of DSPy

This paradigm-shaking research comes from a young but astonishingly accomplished four-person team at MIT CSAIL.

Co-first author Rikiya Takehi is a first-year PhD student who just entered MIT CSAIL, and he completed his undergraduate studies at Waseda University.

From among more than ten thousand science and engineering students across the entire university, he earned the highest honor, the "Ono Azusa Memorial Award," with a Top 0.01% record (the university's sole representative); he previously interned at NVIDIA Research, and during his undergraduate years he won the WSDM 2026 Best Paper Award as first author.

The other co-first author, Jayden Teoh, is likewise an MIT CSAIL PhD student, advised by renowned computer vision professors Phillip Isola and Vincent Sitzmann.

He completed his undergraduate studies at Singapore Management University, and previously conducted research on world models and representation learning at Google DeepMind and Microsoft Research's New York lab.

The third author, Alex L. Zhang, is a second-year PhD student at MIT CSAIL; he graduated first in his class from Princeton University's Department of Computer Science and was named a Scientific American 2026 Young Scientist.

As first author, he proposed the recursive language models (RLMs) that set the open-source community ablaze, and he also led the construction of well-known benchmarks such as KernelBench and SWE-bench Multimodal.

Core curriculum.

His best-known representative works are precisely the globally viral large-model programming framework DSPy and the modern retrieval cornerstone ColBERT, and the open-source projects he has led have long surpassed 100 million cumulative downloads.

Wharton professor Ethan Mollick, when reposting this research, marveled that this is undoubtedly a game-changing development.

When Andrej Karpathy proposed "Software 2.0" ten years ago, the core idea was that humans would no longer hand-write code, and instead gradient descent would search for programs in weight space; what this MIT research pushes open is precisely the door to "Software 3.0," in which large models directly read and write weights.

At the moment large models learn to see through, debug, and personally rewrite neural network weights, the self-evolution chain leading to ASI will truly have its final neuron connected!

References:

This article comes from the WeChat public account "Xinzhiyuan," author: ASI Revelations; editors: Moses and Taozi, published by 36Kr with authorization.