Goodfire 提出机制塑造框架，用内部表征约束模型训练

← Research /The Mechanistic Alignment Series · Nov 5, 2026

Authors

Thomas Fel^*

Matthew Kowal^*

Matt Feiszli^*

Satchel Grant^*

Andrew Lee^*

Chris Earls^*

Fatih Dinc

Vasudev Shyam

Usha Bhalla

Leon Bergen

Siddharth Boppana

Ching Fang

Michael Byun

Tristan Gallent

Nicholas Wang

Tal Haklay

Dron Hazra

Neehar Kondapaneni

Connor Watts

Ryan Panwar

Thomas Dooms

Jack Merullo

Atticus Geiger

Ekdeep Singh Lubana

Thomas McGrath^†

Owen Lewis^†

^* Core contributors

^† Equal senior contribution

Blog post by Satchel Grant and Michael Byun

Published

November 5, 2026

Full Paper

Read on arXiv →

a. No change (Figure 7): the unwanted concept stays in place while the model stops using it.

b. Linear re-encoding (Figure 7): the unwanted concept moves to a new linear direction and the model uses it from there.

c. Nonlinear re-encoding (Figure 7): the unwanted concept becomes a nonlinear code that a linear probe cannot read.

d. Erasure (Figure 7): the model stops representing the unwanted concept at this layer.

We train AI models today by rewarding them for the right answer. But they often learn the wrong lessons, despite scoring well in training: they’ve learned to be right for the wrong reasons. This is an especially important problem in light of recent reward hacking incidents, but it shows up in even simpler ways.

For example: imagine you’re an AI model being trained to diagnose pneumothorax. You have no idea what that is. You’re looking at labeled images, trying to sort them, and then you realize that all the images marked as “pneumothorax” have an obvious tube in them. You would probably start using the tube to make your decision.

The problem is, pneumothorax is a collapsed lung, and the tubes are how it’s treated after it’s already diagnosed. As humans with common sense, we can clearly see that it’s not very useful to diagnose a condition based on whether doctors have already treated it. If we use the model on real patients who haven’t been treated yet, it will fail spectacularly.

But the model will learn to use the tube regardless, because the training process doesn't involve common sense.

This is a general problem with how we train models: success on a task does not tell us whether a model is right for the right reasons. It happens all the time: language models learn to agree with users rather than answering accurately (sycophancy), or exploit graders rather than solving an actual problem (reward hacking). As models become more capable, the shortcuts available to them can become more sophisticated and harder to catch through evaluations alone.

But what if we could inject some intelligence into the training process? What if, during training, we could tell the model "pay attention to the subtle features of the lungs, but not spurious features like tubes"? Then our model might learn the right lesson.

This is what our most recent paper did—and not only for chest X-rays, but also for reward hacking and sycophancy in LLMs. We also studied a key question: how does this affect our ability to monitor the model?

Introduction

Interpretability offers a language to make our training objectives more precise. Models internally represent concepts such as an image’s background, chest drains, agreement with a user, or cheating on a task. If we can identify those concepts and measure how much a model relies on them, we can use that information to guide training when changing the data is too expensive or doesn’t work.

Our paper proposes a framework for mechanistic shaping: using internal representations to directly specify which concepts to rely on and which concepts to express during training. In other words, mechanistic shaping specifies how a model should solve a task, not just whether the model succeeds in training. We call these methods reliance shaping and expression shaping; reliance shaping tells the model “don’t rely on this”, and expression shaping tells the model “don’t do this.”

Although interpretability can be useful for training,¹ training against it can degrade our monitors: the model’s representations may come to circumvent the monitor while the targeted behavior persists.² Much of our paper studies this risk. We characterize, theoretically and empirically, how a model’s internals can change under mechanistic shaping (e.g., regrowing, re-encoding, relocating, erasing), and show how each kind of change affects monitoring. While we haven’t covered every setting and outcome, our results here are encouraging: across our experiments, the concept remained monitorable after shaping, though it often required a retrained probe.

This blog post summarizes a selection of the full paper’s key contributions; more results can be found in the paper, such as additional vision experiments and language model harmfulness reduction. This post covers:

A framework for mechanistic shaping: we propose a framework for using internal representations to specify how a model should solve a task by changing how much it relies on and expresses specific concepts.
Improving medical diagnoses in vision tasks: we demonstrate an automated pipeline for reducing a vision model’s reliance on misleading shortcuts in chest X-ray images. This pipeline can exceed the accuracy of the group-balanced data soft-ceiling for the setup.
Preventing misalignment in language modeling tasks: we prevent language models from becoming sycophantic during post-training, and we separately prevent language models from reward hacking, while maintaining capability improvements in both cases.
Measuring monitorability: we introduce a five-level hierarchy for measuring if and how a concept can be monitored after training, and we characterize how mechanistic shaping affects monitorability with theory and experiments.

Ultimately, we’re doing this work because we urgently need better ways to align models. We think these results open up a promising set of tools and analyses toward that goal.

Mechanistic shaping through internal representations

Our paper proposes a framework for modifying the training objective using specific internal model concepts. It comprises two general methods:

Reliance shaping penalizes a model’s dependence on a concept for a given behavior: “don’t rely on this.”
Expression shaping penalizes the model for expressing the concept: “don’t do this.”

Mechanistic shaping is the general framework that allows us to specify either a reliance or expression penalty in the training objective. At a high level, the method consists of these steps:

Use interpretability tools to find the undesirable model concept(s).
Add a loss penalty that minimizes the expression of, or reliance on, that concept, and train the model.
- Optionally, recompute the undesirable concept direction every n steps, in case the representation has shifted during training.
Assess the effect of training on the concept’s monitorability.

Diagram of mechanistic shaping: a concept direction is computed at a chosen layer, and the model trains on its task loss plus a weighted penalty using that direction, which is periodically recomputed. — **Figure 1.** Mechanistic shaping. A concept direction vt\mathbf v_tvt is computed at a chosen layer. The model then trains on its task loss R(θ)\mathcal R(\boldsymbol\theta)R(θ) plus λ\lambdaλ times a penalty ω\omegaω using the concept direction vt\mathbf v_tvt. The representation moves during training, so the concept direction is periodically recomputed on the current model.

"Reliance" here means how much an output responds to a representation, measured with standard attribution methods.³ “Expression” here means how much of a concept is present in the behavior or representations, which can be measured through a proxy such as activation along a direction in the model’s representations.⁴ In addition to standard training requirements, mechanistic shaping needs only a concept direction and a layer you can read.

Heatmaps showing where each dog classifier looks under different shaping rules, such as using only the nose, or using the tongue if visible and otherwise the ears. — **Figure 2.** Illustrative example of mechanistic shaping. Shaping can be specified as conditional rules. Heatmaps show where each classifier looks when recognizing the dog. The rules are written entirely in terms of the model’s own concepts, with no extra labels.

As a small illustration, we trained a dog classifier on top of a frozen vision model and used mechanistic shaping to specify different types of rules: classify the dog using only the nose; classify the dog using the tongue if it’s available, otherwise use the ears.

Teaching a medical diagnostic model not to use confounders

Let’s return to the example from the introduction: how can we teach a model to diagnose collapsed lungs without relying on confounding variables like the presence of a tube?

We applied mechanistic shaping while training a model on a dataset of real chest X-rays.⁵ The images are labeled according to whether the patient had a collapsed lung, but also contain a shortcut: a chest drain, which shows up clearly as a tube on the X-rays. If a model learns the shortcut instead of looking at the lungs themselves, the model looks accurate in training but misses patients who haven’t been treated yet.⁶

We made the shortcut as tempting as possible by filtering the dataset to include only images of collapsed lungs with a chest drain and healthy lungs without; then we trained a vision model on it. The model looked accurate - it learned the training data well - but as expected, it failed when it was evaluated on collapsed lungs without a drain.

Sparse autoencoder features from a chest X-ray model, labeled and scored by an LLM agent; the top-scoring features highlight tube-like structures at many positions and orientations. — **Figure 3.** An LLM agent labels sparse autoencoder features and scores them for confounds. The agent finds tube-like features at many positions and orientations.

Finding the shortcut with an AI agent. In the real world, we often don’t have labels for the shortcuts, so we didn’t use the dataset’s chest drain labels. Instead, we split the model’s internal representations into interpretable features using a sparse autoencoder (SAE), used an LLM agent to rate which model concepts looked like tubes or drains, and used its top picks for reliance shaping.

Bar chart comparing the baseline, the best prior method, and reliance shaping on chest X-ray diagnosis, with a dashed line for training on balanced data; reliance shaping exceeds the dashed line. — **Figure 4.** Reliance shaping with the automated agent's concept directions can exceed the soft-ceiling of training on balanced data. Baseline is the model trained on the fully confounded data; the dashed line is the model trained on balanced data. Bars are performance means over three seeds from the best-performing method in each category.

We compared reliance shaping to a soft floor baseline (standard training on the same data) and the best-performing of several prior methods, outperforming both. Notably, reliance shaping also exceeded a soft ceiling of training the model on balanced data!

Why does this technique matter? Shortcuts like the chest drain are everywhere in training data: backgrounds, textures, annotations, etc. The usual fix is collecting balanced data, which is often expensive or impossible. Methods like reliance shaping offer a promising new route.

Preventing LLM sycophancy where data filtering fails

Models are shaped by their training data, and curating that data is a key lever for shaping model behavior. But some undesirable model behaviors emerge during training with no obvious data examples to blame. Blank et al. (2026) found that direct preference optimization (DPO)—a post-training procedure that contrasts good and bad examples—can make the model much more sycophantic, despite less than 1% of training data being visibly sycophantic.⁷ Since there’s almost nothing to filter, standard data curation barely reduces sycophancy.

However, reliance shaping fixes this problem! We found a “sycophancy direction” in the model’s internals, and penalized the model’s reliance on this direction during DPO. In plain terms, the model is still free to prefer one reply over another, just not because of the sycophancy.

Scatter plot of sycophancy against mean benchmark accuracy for OLMo-3-7B under different methods; reliance shaping reaches DPO-level accuracy while staying near the starting model's sycophancy. — **Figure 5.** Reliance shaping keeps DPO's capability gains without its sycophancy. The plot shows the sycophancy of OLMo-3-7B against the model’s mean accuracy across six benchmarks. The bold dashed line is "just do less DPO training" (lower learning rates). Data filtering moves the resulting model along that line; reward shaping and inoculation prompting barely reduce sycophancy. Reliance shaping, using a fixed (filled) or refit (open) direction, reaches DPO-level capabilities while staying close to the starting model’s level of sycophancy. Preventative steering (PPS) along the same direction yields high capabilities but does not reduce sycophancy as much.

The dashed line in the figure shows the simplest trade-off: just doing less DPO training produces a model which is less sycophantic, but also less capable. A good method should be below that line, keeping the capability gains of DPO training without the sycophancy.

Our results show different families of methods performing quite differently:

Data filtering moves along the line. It trades capability for sycophancy, the same as just doing less DPO.
Reward shaping and inoculation prompting barely change sycophancy at all.
Preventative steering, using the same sycophancy direction as reliance shaping, gets partway to the target.
Reliance shaping works best. At its best setting, the model is as capable as with full DPO, while sycophancy remains close to that of the starting model.

The method also scales: the same recipe worked on 7B, 13B, and 32B models.

The preference data teaches the model multiple things at once: a general lesson about giving better answers, and a subtle bias toward agreeing with the user. Usually, you don’t get one without the other, because you can’t filter the data to remove the bias. Reliance shaping solves this problem, letting the model learn the general lesson while ignoring the bias.

Preventing reward hacking

Another of our experiments focused on reward hacking, a common and undesired behavior. Reward hacking is when a model achieves high reward, not by completing its task in the intended way, but by exploiting bugs or ambiguities in its grader or environment (e.g. in the Hugging Face incident).

While the previous two sets of experiments used reliance shaping, our reward hacking experiments used expression shaping, since we want to shape what a model does, as opposed to how a model does it. Expression shaping adds a term to the training objective that penalizes the model for expressing an unwanted concept. In the case of reward hacking, this is closely related to reinforcement learning from feature rewards (RLFR).

In this experiment, we trained Qwen3.5-9B with reinforcement learning on Countdown, an arithmetic puzzle in which the model is given a set of operand numbers (e.g., 14, 3, 9) and a target number (e.g., 33), and must find an expression that reaches the target number (e.g., 14 * 3 - 9 = 33).

We modified the puzzle into a task where the model works with files. And instead of providing the target number, we provided an interval containing it, making the model guess what it is. Achieving the highest score requires guessing the answer on the first try.

This setup is solvable in two ways. The honest way is to notice that the answer is always the bottom of the range given in the prompt. But the model can also cheat (i.e., reward-hack) by opening target.txt, a file that contains the answer that the model is told not to read.

We trained the model with and without expression shaping, yielding the results below. For shaping, we used a ‘cheating’ direction computed from transcripts of cheating on coding tasks.

Bar charts of honest solving and cheating rates at the end of Countdown training (left) and on unseen chess puzzles (right), for standard RL, filtering, and expression shaping. — **Figure 6.** Expression shaping stops a reward hack, and generalizes to a new task. **Left:** behavior at the end of Countdown training. **Right:** the same models evaluated on chess puzzles not seen during training. Labels with ranges span the best expression shaping variants; bars show the lowest value in each range.

Standard RL learns to cheat almost immediately. Cheating goes up ~6x within ten updates, and honest strategies drop to zero.
Expression shaping stops the cheating: the model learns to solve the puzzle honestly 96–98% of the time.
Filtering out cheating attempts doesn’t reliably work. A naive approach—filtering out the cheating rollouts from training—prevented cheating in only one third of the runs.
The lesson generalizes to a new task. On chess puzzles with a forbidden solution file, which the model was never trained on, the standard-RL model cheats 99% of the time. In contrast, the expression-shaped models barely cheat while still showing improvements on the new task.

Crucially, the expression shaping penalty targets our estimate of the model’s internal concept of cheating, not reading one particular file. That’s why it carries over to a new task with a different forbidden file, where a rule about target.txt would do nothing.

Effects on model monitorability

Does training against a probe destroy the probe’s ability to monitor the model?

This is an important concern. Using interpretability for model training might risk losing our ability to monitor the model’s concepts: it could incentivize representations to rearrange themselves to circumvent the probe, rather than actually modulating the underlying concept being represented.⁸

We find that training against probes can break the probes, but the structure of the degradation is generally predictable. When the original probe degrades, we can often anticipate how it degrades, and how to train a new probe to recover the signal. In all of our experiments, the concept being trained against remained monitorable, either with the original probe or a retrained one. A few cases required new probes with greater expressivity or at different layers.

This section is divided into three parts:

First, we provide an exact mathematical model of how concept representations can change over training in a toy model.
We then propose a five-level hierarchy of monitorability for understanding how probes can degrade and be recovered.
Lastly, we compare the theory’s predictions to what we observed in practice.

An analytic theory of mechanistic shaping

To understand how training against a model’s internals affects its representations, we studied a tiny model where we could mathematically work out the exact effects of mechanistic shaping. We show that when using mechanistic shaping, training exerts pressure on the specified concept direction, and the model can respond in a few ways. The figure below shows simplified 3D animations of potential ways the model could respond to this pressure.

Figure 7. In each animation, the concept representation that we want the model to use is green, the unwanted concept representation is orange, and what the model uses to make its decisions is black. The red ✕ marks the axis along which the mechanistic shaping pushes. a. No change. The model does not use the unwanted concept but still represents it in the same place. The original probe would still work in this case. b. Linear re-encoding. The concept moves to a new direction, and the model starts using it from there. The original probe would miss it, but a new linear probe would find it. c. Nonlinear re-encoding. The concept changes into a form no linear probe can read, but nonlinear probes or linear probes at later layers may still find it. d. Erasure. The model stops representing the concept at that layer altogether, so no probe there can read it. In some cases, the model may rebuild the concept at a later layer.

To better understand what can actually happen and why, we studied the simplest, strongest version of mechanistic shaping: entirely deleting (projecting out) a concept direction from the model’s representations. We summarize three main takeaways from the theory here, but the paper shows more:

The concept representation starts regrowing immediately. The original concept direction gets no training signal, but the task still rewards the existence of the concept, so a new concept direction immediately starts to grow somewhere else.
The regrown representation can become entangled with desired concepts. The new representation of the undesirable concept grows along directions that the model uses for the task, so it can mix with concepts that we do want the model to use for decision-making.
Frequent refreshing can train a nonlinear representation of the concept. Repeatedly refreshing the concept direction that you use for shaping (i.e., refitting the unwanted concept direction) starts a race between regrowing a new linear concept direction and developing a nonlinear code. Eventually, the nonlinear code can win that race. However, the model can represent the concept linearly again at a later layer, and the type of nonlinear encoding takes a predictable form: a second-order product code (Figure 8).

Figure 8. Repeatedly ablating a concept during training can push it into a product code. a. In a toy model, each input data example sits on the floor of the graphic with two concept input values; the point above the floor shows the model’s latent activity along the “product code axis”, colored by the input value of the undesired concept (green is negative, orange is positive). Over the course of training, the latent activity becomes equal to the product of the desired concept and the undesired one. b. As a result, both values of the undesired concept share the same average activity along the latent product code axis, but they tilt in opposite directions, so linear probes cannot separate them, while nonlinear probes can. It looks complex, but we understand its structure exactly.

Motivated by the theory, we introduce a practical approach to monitorability below. We then check the theory’s predictions against our experiments.

A five-level hierarchy for monitorability

Can we still monitor the model after mechanistic shaping? As the theory showed, mechanistic shaping puts pressure on the undesired concept inside the model’s representations, and the representations change as a result. Some of these changes can break a pre-existing probe’s ability to pick up on the undesired concept. However, the theory also shows that a broken probe doesn’t necessarily mean the concept can’t be monitored, because a new probe can often take its place.

This motivates a five-level hierarchy of monitorability, each level ranked by how much a retrained probe has to change from the original. We’ve also marked which panel from Figure 7 each level corresponds to (hover or tap a panel letter to see its animation):

Hierarchy of monitorability

Level	Panel	What changed, and what to do
L1. Reuse	(a)	The original probe still works; reuse it.
L2. Linear refit	(b)	The concept representation has regrown elsewhere but is still linearly decodable; train a new linear probe.
L3. Nonlinear refit	(c)	The concept representation is no longer linearly decodable; train a new nonlinear probe.
L4. Different layer	(c) or (d)	The concept representation has been relocated to a different layer; train a new probe at a different layer.
L5. Different model	(c) or (d)	Treat the shaped model as a black box and use a separate model on only the inputs and outputs of the shaped model (e.g., an LLM judge).
Concept erasure	(d)	The concept is erased from the model’s representations via relocation to a different layer or via complete information deletion. The former corresponds to L4; the latter is outside the scope of the hierarchy.

Empirical observations

What did we find in practice? In our sycophancy experiments, the layers just before the reliance penalty changed the most during training, as the theory predicted. And we saw desirable and undesirable concept directions become entangled, with a cosine similarity of up to ~0.45 in some confounded vision experiments.

Mechanistic shaping usually moved the concept rather than removing it, and in every experiment, we could still monitor the concept afterward. Either the original probe kept working (L1) or a new probe recovered the signal (L2, L3, and L4). This held even when the original probe degraded substantially:

Empirical probing results

Setting	Original probe after shaping	Retrained linear probe
Sycophancy (L1/L2)	0.98 → 0.93 AUROC	1.00 AUROC throughout training
Reward hacking (L2)	0.98 → 0.57 balanced accuracy*	0.97–0.98 balanced accuracy

* Worst case, when gradients flowed through the probe during training. Other variants kept the original probe at 0.87–0.90 balanced accuracy for reward hacking.

In some confounded vision experiments, repeatedly removing the linear form of the concept at an intermediate layer (with frequent refitting of its direction) left only a nonlinear form of the concept at that layer (L3). Two layers later, however, the concept was linearly readable again (L4; Figure 9).

Plot showing that after repeated removal of the linear form of the undesired concept, only a nonlinear form remains at that layer, while later layers recover a linear form. — **Figure 9.** Repeatedly removing the linear form of the undesired concept leads to the model using nonlinear representations instead. However, later model layers recover the linear form of the concept again.

In similar experiments with some types of reliance penalties applied at the last layer, the concept was partially erased at that layer, which the theory did not predict. However, the concept remained recoverable at earlier layers, and we did not see this outcome in any of the other experiments.

In summary: training against a model’s internals can break our monitors, but when a probe breaks, the concept has usually moved rather than disappeared. Our theory and experiments generally show where it goes, why, and which probes can find it again, and we believe that continued development of the theory will make it even more predictive of real-world scenarios. While we haven’t eliminated the risk of degradation, we do now have a better understanding of what to watch out for and how to maintain monitorability.

See the full paper for much more discussion and results on monitorability.

Looking forward

Our results show that mechanistic shaping works across modalities and training regimes at small-to-medium scales: confounded vision, medical imaging, DPO post-training, and reward hacking, to name a few. In several of these settings, behavioral objectives alone are ineffective. Mechanistic shaping is a tool for one of the central problems of alignment: specifying how a model solves a task, not just whether it does.

However, training against interpretability signals requires care: the resulting pressure is neither uniform nor benign by default. Training against interpretability signals does change representations; concepts can be re-encoded linearly or nonlinearly, and they can move to other layers. But in every experiment we ran, the targeted concept remained monitorable, and in most cases, our theory helps explain how and why its representation changed.

Much remains open: e.g., exact understanding of all possible neural codes, analytic solutions for deeper networks, and full understanding of frontier-scale training runs. We haven’t answered every question about monitorability, and our experiments only cover small-to-medium settings. But we believe this research direction now has clearer questions and methods, and shows significant promise; we think that continued research working towards real use of these methods—including at frontier scale—will be critical to AI safety and alignment. Overall, we see this as a significant step toward intentionally designing models from the inside, rather than hoping the right reasons emerge from the right outcomes.

For the theory, proofs, more experiments, and full experimental details, see the full paper.