Anthropic 可解释性团队：注意力机制研究进展

We report some developing work on the Anthropic interpretability team, which might be of interest to researchers working actively in this space. We'd ask you to treat these results like those of a colleague sharing some thoughts or preliminary experiments for a few minutes at a lab meeting, rather than a mature paper.

Last year, we announced that studying attention superposition was one of our team’s priorities, motivated in part by a toy model demonstrating attention features in superposition. Since then, we’ve tried several ideas to study this problem in real language models, and we’d like to give an update on what we’ve tried and where we think the problem is now.

At a high level, we found significant evidence of attention superposition, and additionally of cross-layer attention representations. We looked for and did not find evidence of preferred OV dimensions for attention featuresOne might have expected to find that features either prefer to have one dimension, to encode a skip-trigram, or else a high dimension, to copy large chunks of information. Instead we see a continuous spectrum.. We further found evidence that the QK condition (“where is information moving to/from?”) and the OV-condition (“what information is being moved?”) are coupled, and have found a practical method for studying these in the form of Multi-Token Transcoders.

We believe that the core remaining problem is understanding why models attend where they do, which is to say understanding attention pattern formation. We believe we have a promising path forward on this problem, in the form of QK diagonalization, which we report on below.

Factoring the Problem

We want to understand the nature of computation in attention. To get a handle on this problem, we’ve found it useful to decompose it into a number of smaller pieces, each of which would either inform further investigations or else provide a practical tool for analysis if solved. Those pieces are:

To what extent is there superposition across attention heads within a layer?; To what extent can we combine the QK-condition (“where is information moving to/from?”) and the OV-condition (“what information is being moved?”) into individual attention features? Are these naturally coupled?; To what extent are attention features multidimensional in the OV circuit (e.g. \mathrm{Rank}(W_{\rm OV}) > 1)?; To what extent are attention features multidimensional in the QK circuit (e.g. \mathrm{Rank}(W_{\rm QK}) > 1)?; To what extent are attention features spread across layers?; How are attention patterns formed / how can we best understand the nonlinear action of the QK circuit?

We began our study with an ambitious approach that, if successful, would have answered most of these in one go. We then conducted several more targeted studies to solve individual pieces, and we now have preliminary results on several of these.

In the rest of this update we document our approaches to date, explain how each has updated us, and provide a roadmap of next steps.

Approaches

Attention Replacement Layer

Our first approach was an “Attention Replacement Layer”. The basic idea was to form a wide (many-head) attention layer and train it to mimic the behavior of an attention layer in the base model of interest (i.e. given the same inputs, train on MSE versus the outputs of the base layer). The wide model would be regularized to be simple, in the sense of the QK and OV circuits being low-rank. In some variants we also regularized to make the attention pattern sparse. With this approach, an attention feature is a single head in the replacement layer.

We made some progress with this approach, including some signs of life (interpretable heads in the replacement layer). At the same time, we hit significant ML difficulties that we were unable to resolve. Our current view is that it is likely possible to get this approach to work, but not easy.

We also made some conceptual progress pursuing this approach. We studied a wide array of different regularization schemes, and found that a regularization penalty of the form \left[L_{\rm nuc}(W_{\rm QK}) L_{\rm nuc}(W_{\rm OV})\right]^\beta, 1/2 1

We may also return to study attention replacement layers, likely through a lens that looks like a combination of QK diagonalization and MTC’s.