Authors
Andrew Lee
Kamesh Krishnamurthy
Jack Merullo
Dan Balsam
Ekdeep Singh Lubana
Published
November 20, 2026
Today, Thinking Machines released Inkling 2 Mini, a small open-weights model. The model combines multiple architectural innovations, including attention residuals 1 and looping its middle layers 2.
Thinking Machines provided Goodfire with early access to the weights of Inkling 2 Mini so we could investigate how these architectural innovations affect the model’s internal mechanisms. We studied a few key questions:
- How does information flow through the model?
- How do model components coordinate that information flow across layers and loops?
- How can we decode such information that is being processed throughout the network?
Our preliminary findings include insights into how layers coordinate with each other, and with future and past versions of themselves: counterintuitively, looped model layers can write to themselves, or read from “future” layers!
We believe these insights are just the beginning of what we can learn about these models. As their architectures continue to evolve, we believe our intuitions and understanding of their internal mechanisms will be able to evolve with them; we hope that interpreting Inkling 2 Mini will contribute to the cross-pollination of insights across different kinds of models.
Overview of Inkling 2 Mini’s architecture
Here we provide a high-level overview of Inkling 2 Mini’s architecture. A detailed description can be found in Thinking Machines' release announcement, or our Appendix.
Inkling 2 Mini has 24 layers, each layer consisting of an attention sublayer followed by a mixture-of-experts (MoE) sublayer. The model has two major architectural differences from a vanilla transformer.
First, Inkling 2 Mini uses attention residuals1: each layer reads from previous layer outputs using attention, rather than reading from a cumulative residual stream. More specifically, it uses depth-wise attention over blocks, each of which sums the output of 6 layers. The model's 24 layers are divided into 4 blocks. That means there is no single residual stream running through the network. Instead, each block keeps a running sum of its own layers’ outputs, starting from zero. Each block eventually outputs the accumulated sum: see Figure 1.
Second, Inkling 2 Mini is a looped model: before being decoded, the model’s intermediate computations go through the same set of weights multiple times. Not all the layers are looped, however: the first block (layers 1-6) runs once on the input, followed by the two core blocks (layers 7-18), which are looped three times. The final block (block 4, layers 19-24) then runs once to decode the output.
The cross-loop attention allows information to pass across loops. In particular, during the second or third pass of the core blocks, each sublayer uses its cross-loop attention to read from the output of blocks 2 and 3 from the previous loop: see Figure 2.
Together, that creates two channels for information to flow across blocks: via depth-wise attention and cross-loop attention. For each of these channels, each sublayer has a dedicated attention module to read from previous blocks or previous loops.
Put together, the entire architecture (and flow of information) looks like Figure 3. For a more detailed description, see our Appendix. Conveniently, Inkling 2 Mini’s architecture allows us to decompose the input of each layer as a weighted sum of the outputs of previous layers, allowing us to easily visualize and understand how information travels across the network. We describe this decomposition in the next section.
Understanding information flow in Inkling 2 Mini: decomposing input components
Here we provide a mental model of how information propagates through Inkling 2 Mini.
At a high level, each layer’s input (denoted x\mathbf{x}x) is a mixture of the current or earlier blocks, which themselves are also each the sum of outputs coming from its layers. This allows us to decompose the input at every layer into a sum of contributions from earlier layers, allowing us to track how information flows from one layer to another, across blocks or even loops.
In Inkling 2 Mini, the input to each layer is a softmax mix over the current or earlier blocks, which is built via the model’s depth-wise attention: x=∑iαirivi\mathbf{x}=\sum_i\alpha_ir_i\mathbf{v}_ix=∑iαirivi, where vi\mathbf{v}_ivi is the output of block iii, rir_iri is an RMS-normalization term (1RMS(vi)\frac{1}{RMS(\mathbf{v}_i)}RMS(vi)1), and αi\alpha_iαi is the attention weight applied to block iii.
The output of each block is the sum of its layers’ outputs: vi=∑w∈Biow\mathbf{v}_i = \sum_{w\in B_i}\mathbf{o}_wvi=∑w∈Biow, where BiB_iBi refers to block iii and ow\mathbf{o}_wow is the output of a specific attention or MoE layer from block iii. This allows us to decompose x\mathbf{x}x further: x=∑iαirivi=∑wαi(w)ri(w)ow\mathbf{x} = \sum_i\alpha_ir_i\mathbf{v}_i = \sum_w\alpha_{i(w)}r_{i(w)}\mathbf{o}_wx=∑iαirivi=∑wαi(w)ri(w)ow, where αi(w)\alpha_{i(w)}αi(w) and ri(w)r_{i(w)}ri(w) are the attention weight and RMS-normalization term for block iii to which the output ow\mathbf{o}_wow belongs.
The next stage mixes a message m\mathbf{m}m from the previous loop. Let ttt index the current loop’s turn, while t−1t-1t−1 denotes the previous loop. The message is also a softmax mix of blocks 2 and 3 from the previous loop: m=β2r2[t−1]v2[t−1]+β3r3[t−1]v3[t−1]\mathbf{m}=\beta_2r_2^{[t-1]}\mathbf{v}_2^{[t-1]} + \beta_3r_3^{[t-1]}\mathbf{v}_3^{[t-1]}m=β2r2[t−1]v2[t−1]+β3r3[t−1]v3[t−1]. Here, β2,3\beta_{2, 3}β2,3 are attention weights on blocks 2 and 3, r2,3[t−1]r_{2,3}^{[t-1]}r2,3[t−1] refer to the RMS-normalizer for blocks 2 and 3 from the previous loop, and v2,3[t−1]\mathbf{v}_{2, 3}^{[t-1]}v2,3[t−1] are the outputs of blocks 2 and 3 from the previous loop.
The message m\mathbf{m}m is mixed into the current layer’s input x\mathbf{x}x via a “keep” term (κ\boldsymbol{\kappa}κ) and “gain” term (γ\boldsymbol{\gamma}γ): x′=κ⊙x+γ⊙m\mathbf{x}'=\boldsymbol{\kappa}\odot \mathbf{x} + \boldsymbol{\gamma}\odot\mathbf{m}x′=κ⊙x+γ⊙m (see Appendix for how κ\boldsymbol{\kappa}κ and γ\boldsymbol{\gamma}γ are computed). Note that the first term, κ⊙x\boldsymbol{\kappa}\odot\mathbf{x}κ⊙x, comes from the current loop, while the second term, γ⊙m\boldsymbol{\gamma}\odot\mathbf{m}γ⊙m, comes from the previous loop. This allows us to rewrite x′\mathbf{x'}x′ as the sum of previous layers: x′=∑wcw\mathbf{x'}=\sum_w\mathbf{c}_wx′=∑wcw, where
- cw=κ⊙αi(w)ri(w)[t]ow\mathbf{c}_w = \boldsymbol{\kappa}\odot\alpha_{i(w)} r_{i(w)}^{[t]} \mathbf{o}_wcw=κ⊙αi(w)ri(w)[t]ow for all writes from the current loop.
- cw=γ⊙βi(w)ri(w)[t−1]ow\mathbf{c}_w = \boldsymbol{\gamma}\odot\beta_{i(w)} r_{i(w)}^{[t-1]} \mathbf{o}_wcw=γ⊙βi(w)ri(w)[t−1]ow for all writes from the previous loop.
Thus we have decomposed x′\mathbf{x'}x′ into a weighted sum of layer outputs ow\mathbf{o}_wow. Finally, we can attribute to each ow\mathbf{o}_wow its share of the input: sw=cw⋅x′∥x′∥2s_w = \frac{\mathbf{c}_w \cdot \mathbf{x}'}{\|\mathbf{x}'\|^2}sw=∥x′∥2cw⋅x′, where sws_wsw is the coefficient of cw\mathbf{c}_wcw’s projection onto x′\mathbf{x}'x′. Because the terms sum to the input, the shares sum to exactly 1.
Figure 4 above shows how information flows within Inkling 2 Mini according to our decomposition. Per layer, we take the average contributions over 13,233 tokens coming from 400 FineWeb documents. This demonstrates the direct contributions from one layer to another.
There are a few observations to make. Regarding depth-wise attention, the model attends more heavily towards recent layers, akin to a sliding window. Regarding cross-loop attention, later loops tend to attend to Block 3 over Block 2. This is especially prominent in Block 3, where it almost always attends to itself from its previous loop, perhaps because it already has access to information from Block 2 via the depth-wise attention. With these intuitions in mind, we next study some of the mechanisms underlying these information flows.
Coordination across layers, blocks, and loops
Here we study how Inkling 2 Mini coordinates the flow of information across blocks and loops, both in weight space and activation space.
Weight space analysis of block-wise coordination
Note that both depth-wise and cross-loop attention read at the block level. These two attention mechanisms come with their own set of weights that are used to determine which blocks to attend to. We refer to these weights as “reader” weights. For depth-wise attention, the model has a total of 56 readers (31 for attention sublayers, 24 for MoE sublayers, and 1 before the final output layer), while for cross-loop attention, the 12 layers in blocks 2 and 3 (the looped layers) each have 4 readers (uQ,K,V,MoE\mathbf{u}_{Q,K,V,MoE}uQ,K,V,MoE), for a total of 48. As such, we have a total of 104 reader weights facilitating depth-wise attention and cross-loop attention.
In short, we find that late layers of each block, in particular the last layers, write heavily for these readers, presumably “tagging” the activations such that later layers know which blocks to read from.
To understand which components “write” towards these readers, we inspect every single down-projection column vector in the MoE weights of all layers. More specifically, each MoE has 147,456 neurons, and each neuron has an associated weight vector that gets added to the model’s activation when the neuron is active.
Thus we have 24 layers x 147,456 = 3,538,944 weight vectors that can write into the model’s activations. Which of these are writing for any of the 104 attention readers?
To answer this, we look at all possible pairs between the 3.5M writers (denoted w\mathbf{w}w’s for “writers”) and 104 readers (denoted r\mathbf{r}r’s for “readers”), for a total of 368 million pairs. We measure the absolute cosine similarity between every pair of (w\mathbf{w}w, r\mathbf{r}r) vectors, and look at the top 0.001% pairs, which yields a threshold absolute cosine similarity value of 0.30. By considering the pairs of writers and readers that exceed this threshold, we have 3,490 pairs remaining.
Figures 5 and 6 below break these 3,490 pairs down by the writer’s layer (x-axis) and the reader’s layer (bar stacks, where the colors indicate the reader’s layer). The y-axis indicates the number of (writer, reader) pairs whose absolute cosine similarity exceeds 0.30. We indicate the first layer of each block with stripes in its bar stack. Figure 5 demonstrates depth-wise attention writer-reader pairs, while Figure 6 demonstrates cross-loop attention pairs.
There are a few observations to note. For the depth-wise reader-writer interactions, you can see most of the writing taking place in the last layer of each block (layers 6, 12, 18). For instance, layer 6 writes mostly to block 4, layer 12 writes mostly to blocks 3 and 4, and lastly layer 18 writes to block 4. One can interpret these interactions as the final layers of each block “tagging” the activations for future layers to use, such that later layers can read from these preceding blocks.
The cross-loop interactions tell a similar story, in that all layers (but especially the final layer) of Block 3 do the majority of the writing, in particular to layer 13 (first layer of block 3). Similar to the depth-wise case, this allows layer 13 in future loops to read from Block 3, which is indeed the dynamic that we observe in our previous analysis (see Figure 4).
To summarize, late layers in each block seem to “coordinate” with other layers in order to facilitate the information flow into downstream layers or future loops, whether it is via depth-wise or cross-loop attention.
Activation space analysis of loop-wise coordination (+ massive outlier activations)
These cross-loop coordinations can be inspected from the activations of the model as well. In particular, we find massive outlier activations that seem to determine which blocks the model should attend to across loops.
Massive outlier activations
Our analysis starts with a slight detour, by introducing some massive outlier coordinate values in the model’s activations. In particular, we focus on the massive activations that occur in the looped layers (blocks 2, 3).
To identify outliers, we take the activations across all 24 layers and normalize them based on their RMS, motivated by the architecture’s extensive use of RMSNorm. Each normalized vector therefore has a squared norm of d = 2304. If the squared norm were distributed equally across all coordinates, we would expect each coordinate to have a magnitude of 1, contributing approximately 0.043% of the total squared norm.
Surprisingly, we observe cases where a single coordinate has an average normalized value of +32.4, occupying 32.42/2304≈46%32.4^2/2304 \approx 46\%32.42/2304≈46% of the squared norm.
Although these outlier activations might serve several functions, our findings suggest that they play a role in facilitating information flow between loops.
One example of an outlier is coordinate 2027 (out of 2304 coordinates), which we denote d2027d_{2027}d2027. As alluded to earlier, a single coordinate d2027d_{2027}d2027 has an average RMS-normalized value of +32.4, occupying 46% of the activation’s total squared norm. This coordinate has the most anomalous magnitude, and thus we focus on this specific coordinate for now.
Writing onto d2027d_{2027}d2027 near the end of a loop
The first hint for the role of d2027d_{2027}d2027 comes from inspecting when its value surges. Given a small set of 13,233 tokens from 400 FineWeb documents, we observe that d2027d_{2027}d2027 surges at layer 18, the end of Block 3 of the first two loops: see Figure 7.
As we will see in the next analysis, we believe this is done in order to prepare future loops to read from Block 3.
Reading from d2027d_{2027}d2027 in later loops
Taking a step back from outlier activations, let us revisit the attention dynamics across loops (Figure 8). In particular, consider the beginning of each block in the second and third loops. In all cases, these layers strongly attend to block 3 from the preceding loop (orange curves are much thicker than the green curves).
To understand why, let us revisit Inkling 2 Mini’s architecture. Each layer has dedicated weight vectors, uQ,uV,uK,uMoE∈Rd\mathbf{u}_Q, \mathbf{u}_V, \mathbf{u}_K, \mathbf{u}_{MoE} \in \mathbb{R}^duQ,uV,uK,uMoE∈Rd, to determine which block from its preceding loop to read from. For now, let us only consider uQ\mathbf{u}_QuQ (the same analysis applies to all 4 weights):
v^i=RMSNorm(vi),vi∈Rd \hat{\mathbf{v}}_i = \operatorname{RMSNorm}(\mathbf{v}_i), \mathbf{v}_i \in \mathbb{R}^dv^i=RMSNorm(vi),vi∈Rd: Output of block iii from the previous loop.
si=uQ⋅v^ids_i = \frac{\mathbf{u}_Q \cdot \hat{\mathbf{v}}_i}{\sqrt{d}}si=duQ⋅v^i: Score assigned to block iii.
βi=softmax([s1,…,sN])\beta_i = \operatorname{softmax}([s_1, \dots, s_N])βi=softmax([s1,…,sN]): Attention weights over blocks.
The attention values β\betaβ determine which of the previous blocks to attend to. In our case, we only have two blocks, and we denote their corresponding attention weights as β2\beta_2β2 and β3\beta_3β3 (for blocks 2 and 3).
Again, we often see the model attend mostly to block 3, which we can spell out as follows:
β3=es3es2+es3=11+e−(s3−s2)=σ(s3−s2)\beta_3 = \frac{e^{s_3}}{e^{s_2} + e^{s_3}} = \frac{1}{1 + e^{-(s_3 - s_2)}} = \sigma(s_3 - s_2)β3=es2+es3es3=1+e−(s3−s2)1=σ(s3−s2),
s3−s2=uQ⋅(v^3−v^2)d=uQ⋅δds_3 - s_2 = \frac{\mathbf{u}_Q \cdot \left(\hat{\mathbf{v}}_3 - \hat{\mathbf{v}}_2\right)}{\sqrt{d}} = \frac{\mathbf{u}_Q \cdot \mathbf{\delta}}{\sqrt{d}}s3−s2=duQ⋅(v^3−v^2)=duQ⋅δ.
So ultimately, whether the model attends to block 2 or block 3 depends on the value u⋅δ\mathbf{u}\cdot\mathbf{\delta}u⋅δ, which we can decompose coordinate-wise: gap=u⋅δ=∑iuiδi\text{gap} = \mathbf{u}\cdot\mathbf{\delta}=\sum_i\mathbf{u}_i\mathbf{\delta}_igap=u⋅δ=∑iuiδi. With this element-wise decomposition, we can finally inspect which coordinate iii contributes most to u⋅δ\mathbf{u}\cdot\mathbf{\delta}u⋅δ.
To our surprise, this is where our previous coordinate of interest, d2027d_{2027}d2027, returns. In the case of layer 13 (note that this is the first layer of block 3) during the second loop, for its 4 cross-loop readers uQ,uV,uK,uMoE\mathbf{u}_Q, \mathbf{u}_V, \mathbf{u}_K, \mathbf{u}_{MoE}uQ,uV,uK,uMoE, we see d2027d_{2027}d2027 being the largest (or 2nd largest) contributor towards each of their respective gap terms, contributing 17%, 19%, 11%, and 6% respectively on its own!
Note that d2027d_{2027}d2027 is not the only outlier that contributes heavily towards layer 13’s cross-loop readers. The top 5 activation coordinates that contribute the most towards uQ,V,K,MoE\mathbf{u}_{Q, V, K, MoE}uQ,V,K,MoE’s gap terms account for 60%, 62%, 32%, and 30% respectively. Interestingly, these 5 coordinates are also within the top 15 coordinates when ranked by their RMS-normalized norms.
Here let us briefly return to Figure 7. Note that at the end of the loops (Loop 3, layer 18), there are no more loops to follow, and thus the MoE and attention heads at that layer “clean up” the heavy activation on d2027d_{2027}d2027, before moving on to the final block. Our speculation is that because cross-loop communications are no longer needed, the model removes the activity on d2027d_{2027}d2027.
Decoding loops: logit-lens, tuned-lens, coda-lens
While the previous analyses study how information flows throughout the network, here we analyze how to decode this information. One commonly used trick is to decode intermediate layers by applying the unembedding layer at intermediate hidden states, often referred to as logit-lens. How well does logit-lens (and variants) work with Inkling 2 Mini’s loops?
Here we study a few variants of logit-lens:
- Logit-lens: Apply the final decoding head at intermediate hidden-states.
- Tuned-lens: Learn an affine transformation from a hidden-state to the model’s final layer, in order to preserve the model’s output distribution.
- J-lens: Use an averaged Jacobian matrix from hidden-states to the model’s final state.
- Coda-lens: Here we refer to the final block (block 4) as the coda, and apply it on the model’s hidden-states (as opposed to applying just the final unembedding layer). Note that unlike the above methods, coda-lens is applied to the entire context’s hidden-states, as the coda itself has sequential attention heads in its layers.
We evaluate how well each variant works with two metrics: 1) How often does each lens’ top 50 tokens include the model’s final output token? and 2) How much probability mass does each lens put on the model’s top 50 tokens?3 We evaluate each method using 1,250 FineWeb documents. Tuned-lens requires training, for which we use 23,750 FineWeb documents.
Figure 9 shows the results. In short, we find that logit lens does not work well, apart from the final few layers per loop, which is where the coda block is eventually applied, while tuned-lens and coda-lens work very well.
Also note that logit-lens typically only works on mid to late layers for standard vanilla transformers, closer to where the unembedding is expected to be applied. In the case of looped models, despite the layers from early loops being “early” in terms of the forward pass, they seem to be much more interpretable, while the mid-layers per block across all loops seem to be less so.
This is great news for interpretability, as it suggests that even with these architectural modifications, the model’s internal computations are still interpretable!
Conclusion
These are early results, and there is much more science to be done - including as part of NDIF’s Looped Lens Challenge. We conclude with a note of excitement: as neural network architectures continue to evolve, we believe our intuitions and understanding of their internal mechanisms will be able to evolve with them. Just as interpretability tools and ideas have generalized across CNNs, transformers, and hybrid architectures, we hope that interpreting Inkling 2 Mini will contribute to the cross-pollination of insights across different kinds of models.
Citation
Andrew Lee, Kamesh Krishnamurthy, Jack Merullo, Dan Balsam, and Ekdeep Singh Lubana. “Interpreting Inkling 2 Mini.” Goodfire, November 20, 2026.
BibTeX:@misc{lee2026inkling,
author = {Andrew Lee and Kamesh Krishnamurthy and Jack Merullo and Dan Balsam and Ekdeep Singh Lubana},
title = {Interpreting {Inkling 2 Mini}},
year = {2026},
publisher = {Goodfire},
url = {https://www.goodfire.com/research/interpreting-inkling-2-mini/}
}
Footnotes
-
Dehghani et al., Yang et al., Geiping et al., Bae et al., Nanbeige LLM Lab, Zhu et al. ↩
-
Note that these metrics are likely not suitable for Jacobian-lens, as these metrics do not reflect the objective of J-lens very well. We leave the exploration of J-lens on looped models for future work. ↩
Appendix: detailed architecture of Inkling 2 Mini
Here we provide an overview of Inkling 2 Mini’s architecture. Compared to a vanilla transformer, there are two main differences to consider: attention residuals 1 and looped blocks.
Notation:
- d=2304d = 2304d=2304: model width. ℓ∈{1,…,24}\ell \in \{1,\dots,24\}ℓ∈{1,…,24}: layer. Bi\mathcal{B}_iBi: the layers of block iii.
- oattn(ℓ),omoe(ℓ)∈Rd\mathbf{o}^{(\ell)}_{\text{attn}}, \mathbf{o}^{(\ell)}_{\text{moe}} \in \mathbb{R}^doattn(ℓ),omoe(ℓ)∈Rd: Outputs of layer ℓ\ellℓ's attention and MoE sublayers.
- vi∈Rd\mathbf{v}_i \in \mathbb{R}^dvi∈Rd: Sum of block iii.
- v0\mathbf{v}_0v0: The token embeddings
- vi(ℓ)\mathbf{v}_i^{(\ell)}vi(ℓ): The running sum within block iii, just before layer ℓ\ellℓ
- vi[t]\mathbf{v}_i^{[t]}vi[t]: The sum of block iii in loop ttt.
- Note that vi\mathbf{v}_ivi is the sum of the outputs www in block iii: vi=∑w∈iow\mathbf{v}_i=\sum_{w \in i} \mathbf{o}_wvi=∑w∈iow
- v^=v/RMS(v)\hat{\mathbf{v}} = \mathbf{v}/\mathrm{RMS}(\mathbf{v})v^=v/RMS(v).
- q\mathbf{q}q, u∈Rd\mathbf{u} \in \mathbb{R}^du∈Rd: learned queries for depth-wise and cross-loop attention.
- m∈Rd\mathbf{m} \in \mathbb{R}^dm∈Rd: cross-loop message; κ,γ\boldsymbol{\kappa}, \boldsymbol{\gamma}κ,γ: keep and gain.
Layers. Inkling 2 Mini has 24 layers of width d=2304d = 2304d=2304, each consisting of an attention sublayer followed by a mixture-of-experts (MoE) sublayer.
Blocks. The layers are grouped into 4 blocks of 6 consecutive layers: B1={1,…,6}\mathcal{B}_1 = \{1,\dots,6\}B1={1,…,6}, B2={7,…,12}\mathcal{B}_2 = \{7,\dots,12\}B2={7,…,12}, B3={13,…,18}\mathcal{B}_3 = \{13,\dots,18\}B3={13,…,18}, B4={19,…,24}\mathcal{B}_4 = \{19,\dots,24\}B4={19,…,24}.
Unlike vanilla transformers, there is no single residual stream running through the network. Instead, each block keeps a running sum of its own layers’ outputs, starting from zero. The running sum of block iii just before layer ℓ∈Bi\ell \in \mathcal{B}_iℓ∈Bi is
vi(ℓ)=∑ℓ′∈Bi, ℓ′<ℓ(oattn(ℓ′)+omoe(ℓ′)),\mathbf{v}_i^{(\ell)} = \sum_{\ell' \in \mathcal{B}_i,\ \ell' < \ell} \big(\mathbf{o}_{attn}^{(\ell')} + \mathbf{o}_{moe}^{(\ell')}\big),vi(ℓ)=ℓ′∈Bi, ℓ′<ℓ∑(oattn(ℓ′)+omoe(ℓ′)),and vi\mathbf{v}_ivi is the sum over all six layers. The token embedding acts as an extra "Block 0", v0=RMSNorm(ex)\mathbf{v}_{0} = \mathrm{RMSNorm}(\mathbf{e}_x)v0=RMSNorm(ex), where ex\mathbf{e}_xex is the token embedding for token xxx.
Depth-wise attention. Unlike vanilla transformers, in which each sublayer reads from the residual stream, each sublayer of Inkling 2 Mini reads its input by taking a softmax mix over the block sums available at that point: the embeddings v0\mathbf{v}_0v0, completed blocks v1,…,vi−1\mathbf{v}_1,\dots,\mathbf{v}_{i-1}v1,…,vi−1, and the current running sum (vi(ℓ)\mathbf{v}_i^{(\ell)}vi(ℓ) for attention, vi(ℓ)+oattn(ℓ)\mathbf{v}_i^{(\ell)} + \mathbf{o}^{(\ell)}_{attn}vi(ℓ)+oattn(ℓ) for the MoE). For some layer ℓ\ellℓ with learned query q\mathbf{q}q,
x=∑jαj v^j,αj=softmaxj (q⊤v^jd),v^=vRMS(v).\mathbf{x} = \sum_j \alpha_{j}\,\hat{\mathbf{v}}_j, \qquad \alpha_{j} = \operatorname{softmax}_j\!\left(\frac{\mathbf{q}^\top \hat{\mathbf{v}}_j}{\sqrt{d}}\right), \qquad \hat{\mathbf{v}} = \frac{\mathbf{v}}{\mathrm{RMS}(\mathbf{v})}.x=j∑αjv^j,αj=softmaxj(dq⊤v^j),v^=RMS(v)v.and the sublayer receives RMSNorm(x)\mathrm{RMSNorm}(\mathbf{x})RMSNorm(x). Put differently, the input x\mathbf{x}x is a weighted sum of the earlier blocks. However, the input x\mathbf{x}x is first “mixed” with messages coming from previous loops, as explained below.
Loops. Blocks 2 and 3 (layers 7–18) form the loop and run T=3T = 3T=3 times, while the first and last blocks run once. At each loop ttt, the depth-wise sources that the depth-wise attention reads from include v0\mathbf{v}_0v0 (the token embeddings), v1\mathbf{v}_1v1, and the current loop ttt’s own sums v2[t],v3[t]\mathbf{v}_2^{[t]}, \mathbf{v}_3^{[t]}v2[t],v3[t]. At the end of 3 loops, the final block reads the last loop’s v2[3],v3[3]\mathbf{v}^{[3]}_2, \mathbf{v}^{[3]}_3v2[3],v3[3].
Cross-loop attention. In loops t≥2t \ge 2t≥2, each looped layer ℓ\ellℓ has four inputs s∈{Q,K,V,MoE}s \in \{Q, K, V, \mathrm{MoE}\}s∈{Q,K,V,MoE}, which each reads a message from the previous loop's two block sums using its own learned query us\mathbf{u}_{s}us:
ms=∑j∈{2,3}βj v^j[t−1],βj=softmaxj (us⊤v^j[t−1]d)\boldsymbol{\mathbf{m}}_{s} = \sum_{j \in \{2,3\}} \beta_j\,\hat{\mathbf{v}}_j^{[t-1]}, \qquad \beta_j = \operatorname{softmax}_j\!\left(\frac{\mathbf{u}_{s}^\top \hat{\mathbf{v}}_j^{[t-1]}}{\sqrt{d}}\right)ms=j∈{2,3}∑βjv^j[t−1],βj=softmaxj(dus⊤v^j[t−1])Mixing the cross-loop messages in. Before an input vector x\mathbf{x}x enters a sublayer, the message m\mathbf{m}m from the previous loop is mixed into the input, by applying a “keep” term (κ\boldsymbol{\kappa}κ) and “gain” term (γ\boldsymbol{\gamma}γ) on the input and message respectively:
z=P [RMSNorm(x); RMSNorm(m)],P∈R64×2d,κ=1−tanh(Fz),γ=tanh(Gz),F,G∈Rd×64,x′=κ⊙x+γ⊙m.\begin{aligned} \mathbf{z} &= \mathbf{P}\,\big[\mathrm{RMSNorm}(\mathbf{x});\ \mathrm{RMSNorm}(\mathbf{m})\big], & \mathbf{P} &\in \mathbb{R}^{64 \times 2d},\\ \boldsymbol{\kappa} &= 1 - \tanh(\mathbf{F}\mathbf{z}), \quad \boldsymbol{\gamma} = \tanh(\mathbf{G}\mathbf{z}), & \mathbf{F}, \mathbf{G} &\in \mathbb{R}^{d \times 64},\\ \mathbf{x}' &= \boldsymbol{\kappa} \odot \mathbf{x} + \boldsymbol{\gamma} \odot \mathbf{m}. \end{aligned}zκx′=P[RMSNorm(x); RMSNorm(m)],=1−tanh(Fz),γ=tanh(Gz),=κ⊙x+γ⊙m.PF,G∈R64×2d,∈Rd×64,The keep term κ∈(0,2)\boldsymbol{\kappa} \in (0, 2)κ∈(0,2) either dampens or amplifies the current loop’s input, while the gain term γ∈(−1,1)\boldsymbol{\gamma} \in (-1, 1)γ∈(−1,1) either adds or subtracts the message.
Output. The final output is a depth-wise mix over v0,…,4\mathbf{v}_{0, \dots, 4}v0,…,4, followed by an RMSNorm and the unembedding.
Research
Discovering Neuron Communities in Language Models
November 17, 2026Igor Shilov, Thomas Fel, Tal Haklay, Lucius Bushnaq, Owen Lewis
Neurons as an Interpretable Basis
November 10, 2026Curt Tigges, Matthew Kowal, Thomas McGrath, Owen Lewis, Nick Cammarata
Neuron Atlas
November 10, 2026Interpretability for all types of AI models
Talk to our team
LLMs
Life Sciences
AI Trends