Anthropic 研究：Claude 3.5 Haiku 如何用几何特征流形完成换行计数任务

Intelligent systems need perception to understand, predict, and navigate their environment. These sensory capabilities reflect what's useful for survival in a specific environment: bats use echolocation, migratory birds sense magnetic fields, Arctic reindeer shift their UV vision seasonally. But when your world is made of text, what do you see? Language models encounter many text-based tasks that benefit from visual or spatial reasoning: parsing ASCII art, interpreting tables, or handling text wrapping constraints. Yet their only “sensory” input is a sequence of integers representing tokens. They must learn perceptual abilities from scratch, developing specialized mechanisms in the process.

In this work, we investigate the mechanisms that enable Claude 3.5 Haiku to perform a natural perceptual task which is common in pretraining corpora and involves tracking position in a document. We find learned representations of position that are in some ways quite similar to the biological neurons found in mammals who perform analogous tasks (“place cells” and “boundary cells” in mice), but in other ways unique to the constraints of the residual stream in language models. We study these representations and find dual interpretations: we can understand them as a family of discrete features or as a one-dimensional “feature manifold”/“multidimensional feature” .All features have a magnitude dimension; so a discrete feature is a one-dimensional ray, and a one-dimensional feature manifold is the set of all scalings of that manifold, contracting to the origin. See What is a Linear Representation? What is a Multidimensional Feature? In the first interpretation, position is determined by which features activate and how strongly; in the latter interpretation, it's determined by angular movement on the feature manifold. Similarly, computation has two dual interpretations, as discrete circuits or geometric transformations.

The task we study is linebreaking in fixed-width text. When training on source code, chat logs, email archives, scanned articles, or judicial rulings that have line width constraints, how does the model learn to predict when to break a line? Michaud et al. looked for “quanta” of model skills by clustering gradients . Their Figure 1 shows that predicting newlines in fixed-width text formed one of the top 400 clusters for the smallest model in the Pythia family, with 70m parameters. Human visual perception lets us do this almost completely subconsciously – when writing a birthday card, you can see when you are out of room on a line and need to begin the next – but language models see just a list of integers. In order to correctly predict the next token, in addition to selecting the next word, the model must somehow count the characters in the current line, subtract that from the line width constraint of the document, and compare the number of characters remaining to the length of the next word. As a concrete example, consider the below pair of prompts with an implicit 50-character line wrapping constraint.The wrapping constraint is implicit. Each newline gives a lower bound (the previous word did fit) and an upper bound (the next word did not). We do not nail down the extent to which the model performs optimal inference with respect to those constraints, rather focusing on how it approximately uses the length of each preceding line to determine whether to break the next. There are also many edge cases for handling tokenization and punctuation. A model could even attempt to infer whether the source document used a non-monospace font and then use the pixel count rather than the character count as a predictive signal! When the next word fits, the model says it; when it does not, the model breaks the line:

To orient ourselves to the stages of the computation, we first studied the model using discrete dictionary features. In this frame, we can understand computation as an “attribution graph” where a cascade of features excite or inhibit each other.We actually first tried to use patching and probing without looking at the graph as a kind of methodological test of the utility of features, but did not make much progress. In hindsight, we were training probes for quantities different than the ones the model represents cleanly, e.g., a fusion of the current token position and the line width.

The attribution graph shows how the model performs this task by combining features that represent different concepts it needs to track:

Features for the current position in the line (the character count) as well as features for the total line width (the constraint) are computed by accumulating features for individual token lengths; The model then combines these two representations — current position and line width — to estimate the distance from the end of the line, leading to “characters remaining” features; Finally, the model uses this estimate of characters remaining along with features for the planned next word to determine if the next word will fit on the line or not

The attribution graph provides a kind of execution trace of the algorithm, showing on this prompt which variables are computed and from what. After finding large feature families involved in representing these quantities across a diverse dataset, we suspected a simpler lens might be provided in terms of lower-dimensional feature manifolds interacting geometrically. We found geometric perspectives on the following questions:

How does the model represent different counts? The number of characters in a token, the number of characters in the current line, the overall line width constraint, and the number of characters remaining in the current line are each represented on 1-dimensional feature manifolds embedded with high curvature in low-dimensional subspaces of the residual stream. These manifolds have a dual interpretation in terms of discrete features, which tile the manifold in a canonical way, providing approximate local coordinates. Manifolds with similar geometry arise for a variety of ordinal concepts, and a ringing pattern we see in the embedded geometry in all these cases is optimal with respect to a simple physical model (§Representing Character Count).Ringing, in the manifold perspective, corresponds to interference in the feature superposition perspective.

How does the model detect the boundary? To detect an approaching line boundary, the model must compare two quantities: the current character count and the line width. We find attention heads whose QK matrix rotates one counting manifold to align it with the other at a specific offset, creating a large inner product when the difference of the counts falls within a target range. Multiple heads with different offsets work together to precisely estimate the characters remaining (§Sensing the Line Boundary).

How does the model know if the next word fits? The final decision — whether to predict a newline — requires combining the estimate of characters remaining with the length of the predicted next word. We discover that the model positions these counts on near-orthogonal subspaces, creating a geometric structure where the correct linebreak prediction is linearly separable (§Predicting the Newline).

How does the model construct these curved geometries? The curvature in the character count representation manifold is produced by many attention heads working together, each contributing a piece of the overall curvature. This distributed algorithm is necessary because individual components cannot generate sufficient output variance to create the full representation (§A Distributed Character Counting Algorithm).

We validate these interpretations through targeted interventions, ablations, and “visual illusions” — character sequences that hijack specific attention mechanisms to disrupt spatial perception (§Visual Illusions).

Zooming out, we take several broader lessons from this mechanistic case study:

When Models Manipulate Manifolds. For representing a scalar quantity (e.g., integer counts from 1 to N), it is inefficient to use N orthogonal dimensions, and not expressive enough to use just oneOrthogonal dimensions would also not be robust to estimation noise.. Instead models learn to represent these quantities on a feature manifold with intrinsic dimension 1 (the count) embedded in a subspace with extrinsic dimension 1 > }} ;| || `, @@.

We further analyzed the extent to which there was a relationship between ‘distraction’ of the important attention heads and the impact on the newline prediction. Indeed we found that many of the sequences with potent modulation of newline probability—and especially code-related character pairs—also exhibited substantial modulation of attention patterns.

While in the aluminum prompt the task is implicit, this illusion generalizes to settings where the comparison task is made explicit. These direct comparisons are perhaps more analogous to the Ponzo, Sander, and Müller-Lyer illusions, where the perception and comparison is more direct.

These effects are robust to multiple choice orderings. Moreover, if the length of the text following the @@ exceeds that of the alternative choice, the alternative choice is selected as being shorter.

While we are not claiming any direct analogy between illusions of human visual perception and this alteration of line character count estimates, the parallels are suggestive. In both cases we can see the broader phenomena of contextual cues, and the application of learned priors about those cues, modulating estimates of object properties of entities. In the human case, priors such as three-dimensional perspective can influence perception of object size, or color constancy can influence estimates of luminance (such as in the checker shadow illusion). Here, one possible interpretation of our results is that mis-application of a learned prior, including the role of cues such as @@ in git diffs, can also modulate estimates of properties such as line length.

Related Work

Objective. This work is at the intersection of LLM “biology” (making empirical observations about what is going on inside models; e.g. ) and low level reverse engineering of neural networks (attempting to fully characterize an algorithm or mechanism; e.g. ). Methodologically, our work makes heavy use of attribution graphs with QK attributions built on top of crosscoders .

Linebreaking. Michaud et al. identified linebreaking in fixed-width text as one of the top 400 “quanta” of model behavior in the smallest model (70m parameters) in the Pythia suite.

Position. Prior interpretability work on positional mechanism has largely focused on token position (e.g., ). These works have shown that there exist MLP neurons , SAE features , and learned position embeddings with periodic structure encoding absolute token position. Our work illustrates how a model might also want to construct non-token based position schemes that are more natural for many downstream prediction tasks.

Others have also studied, even going back to LSTMs, the existence of mechanisms in language models for controlling the length of output responses , as well as performed more theoretical analyses of the space of counting algorithms .

Geometry and Feature Manifolds. Beyond position, there has been extensive work in understanding the geometric representation of numbers, especially in toy models (e.g., ) and in the context of arithmetic in LLMs (e.g., ). Collectively, these works have shown that both real LLMs and toy transformers learn periodic representations , with numbers arranged in a helix to enable certain matrix multiplication based addition algorithms , and that these representations are provably optimal in certain settings . In our context, we similarly observe helical representations , numeric dilation , and distributed algorithms across components that collectively implement a correct computation .

Multidimensional features with clear geometric structure have been found in more natural contexts , like in the representation and computation of certain ordinal relationships (e.g., months of the year). In vision models, curve detector neurons and features have been especially well studied and closely resemble the kind of discretization we observe with the families of character counting features. Many other topics have received interpretability analysis of the underlying geometry, such as grammatical relations , multilingual representations , truth , binding , refusal , features , and hierarchy , though more conceptual research is needed .

Perhaps most relevant is recent work from Modell et al. , who provide a more formal notion of a feature manifold, and propose that cosine similarity encodes the intrinsic geometry of features. When testing their theory, they observe highly structured and interpretable data manifolds that have ripples and dilation, similar to our counting manifolds. These observations raise a methodological challenge in how to best capture data with different structure (see e.g. ), but also the exciting hypothesis that many naturally continuous variables (e.g., ) exist in more organized manifolds.

Biological Analogues. The geometric and algorithmic patterns we observe have suggestive parallels to perception in biological neural systems. Our character count features are analogous to place cells on a 1-D track and our boundary detecting features are analogous to boundary cells . These features exhibit dilation—representing increasingly large character counts activating over increasingly large ranges—mirroring the dilation of number representations in biological brains . Moreover, the organization of the features on a low dimensional manifold is an instance of a common motif in biological cognition (e.g., ). While the analogies are not perfect, we suspect that there is still fruitful conceptual overlap from increased collaboration between neuroscience and interpretability .

Discussion

In this paper, we studied the steps involved in a large model performing a naturalistic behavior. The linebreaking task, frequently encountered in training, requires the model to represent and compute a number of scalar quantities involving position in character count units that are not explicit in its input or outputTokens do not come annotated with character counts, and there are no vertical bars on the page showing the line width., then integrate those values with the outputs of complex semantic circuits (that predict the next proper word) to predict the next token. We found sparse features corresponding to each important step of the computation, and for those steps involving scalar quantities, we were able to find a geometric description that significantly simplified the interpretation of the algorithm used by the model. We now reflect on what we learned from that process:

Naturalistic Behavior and Sensory Processing. Deep mechanistic case studies benefit from choosing behaviors that the model performs consistently well, as these are more likely to have crisper mechanisms. This means prioritizing tasks that are natural in pretraining over tasks that seem natural to human investigators, and ideally, that are easily supervisable. As in biological neuroscience, perceptual tasks are often both natural and easy to supervise for interpretability (e.g., it is easy to modify the input in a programmatic way). Although we sometimes describe the early layers of language models as responsible for “detokenizing” the input , it is perhaps more evocative to think of this as perception. The beginning of the model is really responsible for seeing the input, and much of the early circuitry is in service of sensing or perceiving the text similar to how early layers in vision models implement low level perception .

The Utility of Geometry. Many of the representations and computations we studied had elegant geometric interpretations. For example, the counting manifolds are the result of an optimal tradeoff between capacity and resolution, with deep connections to space-filling curves and Fourier features. The boundary head twist was especially beautiful, and after discovering one such head, we were able to correctly predict that there would need to be additional heads to provide curvature in the output. The distributed character counting algorithm was more complex, but we were still able to clarify our view by studying linear actions on these manifolds. For other computations, like the final breaking decision, the linear separation was clearly a part of the story but there must be some additional complexity we were not able to see yet to handle multitoken outputs. For the more semantic operations, we purely relied on the feature view. Of course, describing any behavior in full is immensely complicated, and there is a long list of possible subtleties we did not study: how the model accounts for uncertainty in its counting, its mechanism for estimating the line width given multiple prior lines of text, how it adapts to documents with variable line width, how it handles multiple plausible output tokens of different lengths or multitoken words, or various special cases (e.g., a LaTeX \footnote{} or a markdown link). For the inspired, we share transcoder attribution graphs for a fixed-width line break prompt on Gemma 2 2B and Qwen 3 4B, using the new neuronpedia interactive interface.

Unsupervised Discovery. It likely would not have been possible to develop this clarity if it were not for the unsupervised sparse features. In fact, when we started this project, we attempted to just probe and patch our way to understanding, but this turned out poorly. Specifically, we did not understand what we were looking for (e.g. we didn’t know to distinguish line width vs. character count), where to look for it (e.g., we didn’t expect line width to only be represented on the newline), or how to look for it (we started by training 1-D linear regression probes). However, after identifying some relevant features but before spending substantial effort systematically characterizing their activity profiles, we were also confused by what they were representing. We saw dozens of features that were vaguely about newlines and linebroken text, but their differences were not obvious from flipping through the activating examples. Only after we tested these features on synthetic datasets did their role in the graph and the underlying computation become clear. We suspect better automatic labels enhanced with agentic workflows would accelerate this work, especially in less verifiable domains.

Feature-Manifold Duality. The discrete feature and geometric feature-manifold perspectives offer dual lenses on the same underlying object. For example, in this work the model's representation of character count can be completely described (modulo reconstruction error) by the activities of the features we identified, where the action of the boundary heads is described by virtual weights that expand out the feature interactions via attention head matrices. The same character count representation can be described by a 1-dimensional feature manifold – a curve in the residual stream parametrized by the character count variable – where linear action of the boundary heads is described by continuous “twisting” of the manifold. In general, geometric structures learned by the model will likely admit both global parametrizations and local discrete approximations.

The Complexity Tax. Despite this duality, the descriptions produced by the two perspectives differ in their simplicity. The discrete features shatter the model into many pieces, producing a complex understanding of the computation. This seems like a general lesson. It seems like discrete features and attribution graphs may provide a true description of model computation, which can be found in an automated way using dictionary learning. Getting any true, understandable description of the computation is a very non-trivial victory! However, if we stop there, and don't understand additional structure which is present, we pay a complexity tax, where we understand things in a needlessly complicated way. In the line breaking problem, constructing the manifold paid down this tax, but one could imagine other ways of reducing the interpretation burden.

A Call for Methodology. Armed with our feature understanding, we were able to directly search for the relevant geometric structures. This was an existence proof more than a general recipe, and we need methods that can automatically surface simpler structures to pay down the complexity tax. In our setting, this meant studying feature manifolds, and it would be nice to see unsupervised approaches to detecting them. In other cases we will need yet other tools to reduce the interpretation burden, like finding hierarchical representations or macroscopic structure in the global weights .

A Call for Biology. The model must perform other elegant computations. We can find these by starting with a specific task the model performs well, study this from multiple perspectives, develop methodology to answer the remaining questions, and relentlessly attempt to simplify our explanations. Because the investigation is grounded in specific examples of a behavior, it provides a fast feedback loop, can shed light on weaknesses of existing methods and inspire new ones, and can sharpen our conceptual language for understanding neural networks. We would be excited to see more deep case studies that adopt this approach.

Citation Information

For attribution in academic contexts, please cite this work as

BibTeX citation

Acknowledgments

We would like to thank the following people who reviewed an early version of the manuscript and provided helpful feedback that we used to improve the final version: Owen Lewis, Tom McGrath, Eric Michaud, Alexander Modell, Patrick Rubin-Delanchy, Nicholas Sofroniew, and Martin Wattenberg. We are also thankful to all the members of the interpretability team for their helpful discussion and feedback, especially Doug Finkbeiner for discussions of rippling and ringing, Jack Lindsey on framing, Tom Henighan for feedback on clarity, Brian Chen for improving the design of the figures and line edits of the text, and the team who built the attribution graph and QK attribution infrastructure .

Haiku Task Performance

Haiku is able to adapt to the line length for every value of k, predicting newlines at the correct positions with high probability by the third line. Of course, some error is to be expected even with a perfect estimate of line length, as the model may incorrectly predict the next semantic token. Below is the mean log-prob and accuracy for newline prediction of Haiku on 200 prose sequences that were synthetically wrapped to have lines of character length k, for k = 20, 40, \ldots, 140.

Feature Splitting and Universality

It is natural to ask if the character counting features are fundamental, or simply one discretization of the space among many. We found that dictionaries of different sizes learn features with very similar receptive fields, so this featurization – including the slowly dilating widths – is in some sense canonical. We hypothesize that this canonical structure emerges from boundary constraint: positions near zero (start of line) create a natural anchoring point for feature development.

The geometry of the decoder directions is also fairly consistent between the dictionaries, showing characteristic ringing.

However, we do see some evidence of feature splitting. For example, below are three character count features which activate on the same interval (~20–45 characters in the line), but differentially activate for lines of different widths: LCC2.a activates on all line widths, LCC2.b preferentially activates on long line widths, and LCC2.c preferentially activates when close to the line width boundary.

Recent work has raised the possibility that feature dictionaries could behave pathologically where there exist feature manifolds , because a dictionary could allocate an increasing number of features in a finer tiling of the space. However, our observation that cross-coders of varying size tile this feature manifold in a canonical way suggests that this behavior does not occur in this setting.

Line Width Features

Line width features tile the space similar to character count features.

Dynamical System Model

We simulate N = 100 points on the unit (n-1)-sphere in \mathbb{R}^n (n \in \{3,\ldots,8\}) with pairwise forces: \mathbf{F}_{ij} = \begin{cases} \frac{1 - (d_{ij} - 1)/2}{r_{ij}} \hat{\mathbf{r}}_{ij} & \text{when }d_{ij} \leq w \\ -\frac{\min(5, 1/r_{ij})}{r_{ij}} \hat{\mathbf{r}}_{ij} & \text{when }d_{ij} > w \end{cases}, where r_{ij} = \|\mathbf{x}_j - \mathbf{x}_i\|, \hat{\mathbf{r}}_{ij} = (\mathbf{x}_j - \mathbf{x}_i)/r_{ij}, w is the attractive zone width parameter, and d_{ij} = \min(|j-i|, |j-i+N|, |j-i-N|) is the index distance (for the circular topology; for the interval it is just d_{ij} = |j-i|). Evolution follows \dot{\mathbf{v}}_i = \sum_{j \neq i} \mathbf{F}_{ij} - 0.05\mathbf{v}_i and \dot{\mathbf{x}}_i = \mathbf{v}_i with sphere constraint \mathbf{x}_i \leftarrow \mathbf{x}_i/\|\mathbf{x}_i\| enforced after each timestep (\Delta t = 0.01, damping \alpha = 0.95).

We explore a deeper connection between the ringing observed in the character count feature manifold, and a connection to Fourier analysis in an analytical construction.

Suppose that we wish to have a discretized circle's worth of unit vectors, each similar to its neighbors but orthogonal to those further away. Then the cosine similarity matrix X of these will be the circulant matrix of a narrow-peaked function f (left, below). The columns of the square root of X are n vectors v_1,\ldots,v_n \in \mathbb{R}^n, where n is the number of discrete points on the circle, whose inner products reproduce the similarity matrix.The entire continuous circle embeds into the infinite-dimensional Hilbert space L^2\mathbb{S^1} via this construction. Now suppose we would like to find vectors in a lower-dimensional space whose similarity matrix approximates X, like the model does for character counts. The best k-dimensional approximation of this similarity, in an L^2 sense, is given by taking the eigendecomposition of X and truncating it to the top k eigenvectors; the square root of the result will provide n vectors in k-dimensions with the corresponding similarity pattern. If \pi_k is the projector onto the span of the top k eigenvectors, then the images \pi_k v_i, which live in a k-dimensional subspace, are precisely those vectors. We can see below that the resulting low-rank matrix has ringing (colab notebook). Finally, because the discrete Fourier transform diagonalizes circulant matrices, the Fourier coefficients of f are in fact the principal values of X; the low-rank approximation consists of truncating the small Fourier coefficients of f; and the resulting rows of X exhibit ringing.

One essential feature of the representation of line character counts is that the “boundary head” twists the representation, enabling each count to pair with a count slightly larger, indicating that the boundary is close. That is, there is a linear map QK which slides the character count curve along itself. Such an action is not admitted by generic high-curvature embeddings of the circle or the interval like the ones in the physical model we constructed. But it is present in both the manifold we observe in Haiku and, as we now show, in the Fourier construction. First, note permutating the coordinates of \mathbb{R}^n by taking e_i \mapsto e_{i+1} has the effect of mapping v_i \mapsto v_{i+1}. That is, the (linear) action of this permutation \rho on \mathbb{R}^n acts by a rotation of the embedded circle with respect to its intrinsic geometry. Because conjugation by \rho fixes the circulant matrix X, it therefore respects its eigendecomposition, and thus commutes with the projection \pi_k onto the vector space spanned by its top k eigenvectors. The restriction of \rho to that subspace, \overline{\rho}:=\pi_k \circ \rho \circ \pi_k, acts by rotation on the lower-dimensional vectors \overline{\rho}: \pi_k v_i \mapsto \pi_k v_{i+1}. Thus we have found n vectors in a k-dimensional space, whose similarity is as close as possible to X (and has ringing), with a linear action of \mathbb{R}^k that rotates the vectors along a rippled embedded circle.

We evaluate whether a Fourier decomposition of the character count curve is optimal, and find that it is quite close given that it does not account for dilation. Fourier components explain at most 10% less variance than an equivalent number of PCA components, which are optimal for capturing variance.

Finally, we note that as one moves through layers, the representation becomes more peaked. This sharpening of the receptive field is useful to the model to better estimate character counts, and corresponds to higher curvature in the embedding and, as predicted by the model above, more pronounced ringing. Below we show cross-sections (at character count 30, 60, 90, 120) of the cosine similarity matrix of probes trained after layers 0, 1, 2, and 3. With each subsequent layer, the graphs get more tightly peaked and secondary rings go higher.

Geometry of Twisting

Different heads access and manipulate the space in different ways. Below, we show the cosine similarity of both probe sets through QK for three heads: one which keeps them aligned, one which shifts character count to align better with later line widths, and one which does the opposite.

We can also look at this transformation by visualizing the Singular Value Decomposition of each set of probes in a joint basis after passing them through QK. Once more, the alignment, left offset, and right offset can be read directly from the components.

We can directly plot the first 3 components of the joint probe space after passing them through each QK. Doing so shows that one head keeps the representations aligned, while the others twist them either clockwise or counterclockwise.

Break Predictor Features

Boundary detector features (at about ~⅓ model depth) do not take into account the length of the next token.

Later in the model, there exist features which incorporate both the number of characters remaining and the length of the most likely next token. These features only activate when the most likely next token is longer than the number of characters remaining (i.e. below the red diagonal below), as is the case in our aluminum prompt.

We also found features for the converse: features which suppress the newline because the predicted next token is shorter than the number of characters remaining in the line.

Both break prediction and suppression features sometimes also have interpretable logit effects on the output of all tokens, not just the newline. For instance, the features below respectively excite and suppress the newline as their top effect, but also systematically suppress tokens with more characters. This is because if the model is wrong about the value of the next token (and whether it's a newline), the token must at least be short enough to fit on the line.

Representing Token Lengths

We find layer 0 features that activate as a function of the character count of individual tokens.

These features are overlapping (e.g. there are tokens for which the long word and medium word features are both active) and non-exhaustive (none of them fire on some common tokens, where we suspect the representation of character length is partially absorbed into features which just activate for that token).

The Mechanics of Head Specialization

Heads collaborate to generate the count manifold, but how does each head aggregate counts?

As a toy model, consider the following construction for character counting with a single attention head:

The head uses the previous newline token as a “sink” where it defaults all of its attention to (i.e., attention 1); Each token since the newline gets \alpha attention, such that after j tokens the newline has 1 - \alpha j attention. Note this limits the construction to only work on lines of up to 1 / \alpha tokens, but the model could use multiple heads at different offsets to count for longer sequences. The model could also dedicate attention to tokens proportionally to token length; The output of the head on the newline is 0, and the output of each non-newline token is a vector with the same direction but magnitude proportional to the character count of the token

This produces a ray with total length proportional to the character count of the line.

In practice, we observe that individual attention heads do indeed use the newline as an attention sink, but at different offsets. As an example, we visualize the attention patterns of 4 important Layer 0 heads on several prompts with different line widths (starting from the first newline in the sequence).

To characterize the mechanism more precisely, we compute the average attention as a function of the number of tokens since the previous newline and also as a function of the character length of individual tokens.

Similar to boundary detection, individual attention heads specialize in particular offsets to tile the space. Moreover, we observe that most of these attention heads have a bias towards attending to longer tokens.

In addition to QK, a head can change its output based on the OV circuit . We study this by analyzing the pairwise interaction of probes as mediated by the OV matrix. Specifically, for the averaged token embedding vectors for each token length E_t That is, for each token character length i, we compute the average embedding vector in W_E. We also prepend this with the newline embedding vector to make the plot below., our line length probes P_c, and the weight matrices for the attention output of each head W_{OV}, we compute P_c^T W_{OV} E_t.

The output of each head can be thought of as having two components: (1) a character offset from the newline driven by the attention pattern and (2) an adjustment based on the actual character length of the tokens. Note that the average character count of a token is approximately 4.5 (and the median is 4), so we can interpret these effects shifting from a mean response (i.e. the transition point is always around count 4).

To walk through a head in action, consider the perspective of L0H1, which attends to the newline for the first ~4 tokens and then spreads out attention over the previous ~4–8 tokens:

While attending to the newline, L0H1 writes to the 5–20 character count (5–20CC) directions and suppresses the 30–80CC directions. This makes sense as an approximation because attending to the newline implies that the line is currently at most 3 tokens long (newline has no width) and on average, any 3 token span is ~15 characters (and is unlikely to be >30); While not attending to the newline at all, L0H1 defaults to predicting CC40, since not attending to the newline implies that there are ~8 tokens in the line with ~5 characters each on average (including spaces). Then, there is an additional correction applied depending on how long the tokens are:

(1) If tokens being attended to are short ( 40CC; (2) If tokens being attended to are long (≥5 chars) then do the opposite

In cases with some attention on newlines and nonnewlines, linearly interpolate the above predictions

Other heads perform a similar operation, except with different offsets depending on their newline sink behavior. Layer 1 heads also perform a similar operation, though they also can leverage the character count estimate of the Layer 0 heads (see Layer 1 Head OVs).

Layer 1 Head OVs

Similar to the OVs of the Layer 0 attention heads, Layer 1 heads write to the character count features in accordance to how long the tokens they attend to are.

However, in addition to the token character length, Layer 1 heads also use the initial line length estimate constructed in Layer 0 to create a more refined estimate of the character count.

These repeated computations appear responsible for implementing the sharpening of representations.

Full Layer 0 Attention Results

Below, we show head sums for 3 different prompts with different line widths.

As before, we can look at their decomposition.

More Sensory and Counting Representations

While in this work we carefully studied the perception of line lengths and fixed width text, there are many tasks which language models must perform which benefit from a positional, visual, or spatial representation of the underlying text. In the course of our investigation, we came across several other feature families and representations for these behaviors and report several below.

What Follows an Early Linebreak?

In addition to line width for tracking the absolute character length of a full line of text, there also exist features that are sensitive to lines which have ended early (i.e., lines where the character count is substantially shorter than the line width k). While these features are less useful for linebreaking, they enable the model to better predict the token following a linebreak. Specifically, if a line ends c characters before the line limit k, the next word should be at least c characters, otherwise it would have been able to fit in the previous line.

It is worth emphasizing that the role of these features, like others in this work, is not obvious from a typical workflow of quickly looking at dataset examples. It might be tempting to ignore these as "newline" features, but careful analysis yields quite clear behavior.

Markdown Table Representations

In addition to prose, language models must parse other kinds of more structured data like tables. Accurate prediction of a table’s content requires careful integration of row and column information (e.g. is this a column of text or numbers?). To facilitate this, we use a synthetic dataset of 20 markdown tables to find feature families which activate on separator tokens, specialized to particular rows or columns. Visualizing feature activations on each of these 20 tables (arranged by location in the table) showed clear patterns.

On a synthetic dataset of larger tables, we also observe counting representations for the column and row index that resemble the character counting representations. Specifically, we see ringing in the pairwise probe cosine similarities and the characteristic “baseball seam” in the PCA basis.

Rejected Titles

A General Language Assistant as a Laboratory for A-line-ment; A-line-ment Science: The Geometry of Textual Perception; The Geometry of Textual Perception: How Models Stay Aligned; The Geometry of Counting: How Transformers Perceive and Manipulate Spatial Structure in Text; The Mechanistic Basis of Alignment; Reading between the lines: The Perception of Linebreaks; Linebreaking: More than you wanted to know; The Line Must Be Drawn Here! Character Counting in Neural Networks; Newline, Who Dis? Attention Heads and Their Distractible Nature; The End of the Line: How Language Models Count Characters; How I Learned to Stop Worrying and Love the Carriage Return; Breaking down the Linebreak Mechanism: The Geometry of Text Perception; We Found a GPS Inside a GPT