Cammarata's interpretability research sparks debate in the AI community: Are neural networks easier to understand than we think?

This article completely overturned the understanding in the field of artificial intelligence, sparked widespread speculation, and even Sam Altman publicly expressed his appreciation.

Researcher Adam Majmudar wrote on X: This is the most important result of the year, comparable in importance to "Attention Is All You Need." It moves us from the stage of "whether the alignment problem can be solved" to the stage of "we might be able to solve the alignment problem next year."

Others said: "It feels as if my understanding of neural networks has been advanced by five years."

The article's author, Cammarata, said that the timing of his publication coincided with a moment when people were looking for hope.

What researchers and policymakers in the field of AI safety worry about most is that powerful models may bring enormous risks. At present, they urgently need a "lifeline" to support themselves. This is one of the reasons the article has caused such a big response.

The judgment put forward in the article is quite bold: large neural networks may be easier to understand than people previously thought, and many internal structures regarded as difficult to disentangle may have all along followed some recognizable pattern.

However, Jacob Hilton of the Berkeley nonprofit research lab Alignment Research Center does not think Cammarata's blog is as important as everyone says.

Hilton pointed out in a tweet on X that researchers understand less than 0.01% of the internal workings of top-tier LLMs.

"I have always admired Cammarata's optimism, but unfortunately, this presents the public with an unrealistic picture of the current state of the field."

Black box, perhaps we have been seeing it wrong

Almost all large models use neural networks to some extent.

A neural network is a machine learning model designed to be inspired by how the human brain works, composed of many small units called neurons that are connected to one another in layers.

Cammarata has long worked on neural network interpretability research and in his early years participated in establishing an influential theoretical framework.

In early vision models, the same neuron would respond to two types of images: cat faces and the front of cars. What exactly was it recognizing? Cats? Cars? Or was it cramming two unrelated concepts together?

Cammarata also believed back then that there were many such entangled concepts inside models.

Researchers therefore developed the "superposition" hypothesis. The explanation is this: the model has far too many concepts to express, but the number of available neurons is limited, so the network has no choice but to squeeze different concepts into the same representation space. One neuron thus carries several seemingly unrelated meanings.

To untangle these intertwined concepts, researchers also developed tools such as sparse autoencoders, attempting to break neural activity down into clearer units.

The superposition of the nearly plane wave coming from afar (the diagonal line) and the waves produced by the duck's wake only approximately satisfies a linear relationship in the water.

Cammarata now believes that he may have overthought it back then.

That neuron may have been recognizing a V-shaped structure. Behind the outline of a cat face, the sharp angle formed by the front of a car, the tongue shape of a flame, and the branching of water flow hides the same visual pattern. Objects that seem completely unrelated actually share the same geometric feature.

Another case once taken as evidence of "superposition" may also have a simpler explanation.

Researchers once found that features for recognizing cars seemed to be hidden in a group of dog-related neurons. After re-examining it, Cammarata offered another explanation: dogs and cars may share a pattern related to shooting angle—they often face the camera in similar poses in images.

If these classic cases all have other explanations, is the inside of neural networks really that difficult to understand?

Cammarata reports that after improving the architecture and analysis methods, even without increasing model size or training data, about 99% to 100% of the neurons in the last layer of the early vision model InceptionV1 can be explained relatively clearly.

He also studied models spanning about 300 times the scale and believes that these models retain a considerable degree of readability at different scales.

Cammarata uses "recursive generalization" to describe another pattern he observed.

During learning, a neuron may start from some narrow pattern and gradually extend related branches, just as a tree branch grows side shoots.

Taking language models as an example, a neuron that initially recognized the letter combination "iy" may later respond to "diy" and the semantically related "do it yourself." If one looks only at the trigger list after the model has finished training, these responses seem very far apart; if one traces how they formed during learning, it may be possible to find a continuous path of association.

In Cammarata's view, many complex phenomena come from ordinary patterns continuously expanding, combining, and differentiating, rather than concepts being arbitrarily stuffed into the same neuron.

This line of thinking has already received preliminary verification. In Qwen3-32B, about 90% of neurons have a primary function in each of their positive and negative response directions, about 7% involve two related functions, and the portion that has not yet been explained is about 2% to 4%.

The more we understand, the safer we are

If it becomes possible to reliably trace the flow of information inside a model, safety research will have the opportunity to move beyond observation and into the analysis of mechanisms.

Researchers can try to identify specific neurons, computational circuits, and information pathways within a model, and then, through intervention experiments, observe whether changes in a particular internal mechanism alter the model's behavior.

If a group of neurons participates in a certain kind of reasoning, and a computational circuit is associated with a particular judgment, then researchers have the opportunity to test them in a more targeted way.

Compared with judging whether a model is safe based only on its final answers, this approach attempts to push safety evaluation into the internal processes that produce behavior.

This is precisely why Cammarata's research has attracted so much attention.

If the internal structure of neural networks is indeed more orderly than people previously believed, then the cost of understanding it may decline. Researchers can directly study the mechanisms a model has already formed, trace how these mechanisms work together, and gradually build a more complete internal map.

However, a model's internal mechanisms are not equivalent to the psychological concepts we are accustomed to using. A neuron associated with deception is not necessarily some independently existing "deceptive intent" inside the model; and a feature that can predict refusal to answer is not necessarily enough to show that the model understands the reasons for refusing.

Researchers need to verify explanations through multiple kinds of evidence, observing neural activity while also conducting interventions, counterfactual tests, and cross-context replication. Otherwise, so-called understanding may still amount to nothing more than attaching a plausible-sounding label to some statistical correlation.

In safety work, model evaluation, red-team testing, access control, runtime monitoring, and deployment restrictions on high-risk capabilities each play roles that cannot easily be replaced. Researchers' understanding of models always has limits, and real-world risks often arise from the compounding of multiple factors.

When it comes to the long-term safety of AI, three possible outcomes recently proposed by Sam Altman push the question to a larger institutional level.

One is to establish an international organization similar to the International Atomic Energy Agency, responsible for setting and enforcing safety standards; another is to integrate all AI laboratories into a single institution to jointly develop new technologies and ensure their safe application; and yet another is for some AI developer to accumulate enough economic power to cut off other developers' computing resources, leaving potentially dangerous developers unable to continue advancing their projects.

The development of nuclear technology has long shown that the risks of certain technologies may exceed the control of any single country, requiring commonly recognized standards, verification mechanisms, and channels for international cooperation. The experience of the International Atomic Energy Agency provides a reference for this.

However, there are clear differences between artificial intelligence and nuclear technology. Models can be copied through software and deployed remotely, and the pace of technological iteration is also far faster than in many traditional industrial systems.

Even if an international institution is established, how to determine which models require strict review, how to verify whether developers comply with standards, and how to handle technological competition among different countries all remain thorny problems.

If all AI laboratories belonged to the same institution, researchers would not need to speed up development in order to compete for market share, and teams could share safety research results and avoid repeating mistakes. Concentrating resources could also subject high-risk experiments to unified review, reducing the room for a team to bypass safety procedures in order to get ahead.

But who would manage this institution, and who would oversee the managers?

When all key technologies, computing resources, and research capabilities are concentrated in a single organization, safety standards may indeed become easier to unify, but power may also become difficult to check.

Once a systemic error occurs in the institution's internal judgments, the entire technological system may be affected. If management's judgments about certain risks are biased, the corrective role that other independent laboratories could originally have played will also be weakened.

The third scenario does not require all developers to accept the same set of management rules, nor does it require waiting for a broadly authorized international institution to reach consensus. As long as one developer has enough capital, computing power, and infrastructure, it may be able to limit other developers' ability to continue training or deploying models by controlling GPUs, data centers, and related supply chains.

Altman said this is the situation he least wants to see.

Returning to Cammarata's research, progress in interpretability may be able to provide a new technical foundation for these governance issues. If researchers can more reliably identify the key mechanisms inside models, they will have the opportunity to improve safety evaluations, audit high-risk capabilities, and verify whether certain interventions are truly effective.

As technological capabilities continue to grow, what will matter is not only whether we can understand increasingly complex models, but also whether society can establish sufficiently reliable mechanisms to ensure that those who understand the technology have the ability to oversee it, and that those who control the technology are always subject to constraints.

This article comes from the WeChat public account "APPSO," author: the one who discovers tomorrow's products, published by 36Kr with authorization.