搜索“RAG”

-time feedback to enable the use of search. As scientific models scale and improve, their representations will encode deeper and more meaningful information. Understanding and leveraging model internals will be crucial across domains as more and more science is done in silico, and self-correcting search is one way to harness that information. We're excited to continue such work to advance scientific abundance. References A generative model for inorganic materials design [link] Claudio Zeni, Robe

匹配字段正文

t lets us guide model training. In his recent post about intentional design, our Chief Scientist Tom articulated the general principle of “not fighting gradient descent” while leveraging interpretability in training. This is the case with our setup here: the probes observe and evaluate, but the training updates flow through the standard RL objective. Since the gradient cannot flow through the probes — since they are run on a frozen copy of the base model during training, not on the student — the

匹配字段正文

thout her conscious effort. It wasn’t cheating; it was more like her intuition, bypassing the complex moral choices she had been forced to make in the past. It worked. But it was fragile. In another experiment, amplifying the internal reward hacking signal made the models much more likely to use a ‘honeypot’ – a shortcut we planted to identify cheating behavior: It’s hard to say in a deep philosophical sense whether models “know” they are reward hacking in exactly the way a person does. But thes

匹配字段正文

(Arcuschin et al., 2025Chain-of-thought reasoning in the wild is not always faithful [link] Arcuschin et al., 2025.; Korbak et al., 2025Chain of thought monitorability: A new and fragile opportunity for ai safety [link] Korbak et al., 2025.; Lanham et al., 2023Measuring faithfulness in chain-of-thought reasoning [link] Lanham et al., 2023.; Turpin et al., 2023Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting [link] Turpin et al., 2023.). Rece

匹配字段正文

s prediction for each PII entity was evaluated according to two metrics: exact match (the prediction exactly matches the gold PII entity's token boundaries) and partial match (coverage ≥ 0.5; at least half of the PII tokens in the entity were predicted as any PII class). All methods were evaluated on 128-token sections of test examples with 32 tokens of overlap; we run one forward pass of the sidecar model on each chunk, and probe at each token position. This chunking is typical in production an

匹配字段正文

ult SGLang serving. We also evaluate the added latency of our monitor cascade, comparing it with an LLM judge reviewing every turn. Without judge reasoning, our cascade reduces average added latency per turn from 160 ms to 2 ms; with reasoning, it reduces latency from 2,750 ms to 42 ms. Both configurations reduce average monitoring delay by more than 50× compared with judging every turn. Results Evaluating Our Monitors We evaluate our cyber-misuse monitors on an in-house dataset of 2.4K chat and

匹配字段正文

te 150 harmful requests with distractors on which the SFT model refuses and the DPO model complies (Appendix A.1). We compute the behavior difference vector for each prompt and average them into a single probe vector. Steering the SFT model with this vector induces compliance on harmful requests, but only when the distractor is present (Appendix A.2). We then rank training datapoints by computing the datapoint difference vector for each and sorting by cosine similarity with the probe. The top-ra

匹配字段正文

remove this sequence similarity confounder from our analysis. To do so, we sample regions independently for each species. Specifically, we construct a dataset of 2400+ species-averaged embeddings as follows: Sample 4000 bp genomic regions randomly per species, covering around 5% of the genome; Get Evo 2 internal activations over the genomic region from layer 24 (of 32). This layer had the highest performance for probes predicting phylogenetic taxa; Average the activations for each species using

匹配字段正文

sible to guide the model via text prompts (for example, to specify a style), the canvas offers a 2D interface for expressing creative intent with familiar tools like painting and dragging. These actions correspond to manipulating the model's internal activations at specific spatial regions of the image being generated. Here are a few examples of the app in action: This application complements Goodfire's work across other model architectures and modalities, such as natural language reasoning mode

匹配字段正文

t caught by standard evaluations, and end up being found after deployment by bewildered users or delighted jailbreakers. This issue appears in striking incidents like ChatGPT encouraging self-mutilation or Gemini telling a user to die, as well as more routine cases of infrequent or context-dependent failure modes after post-training models. Identifying these behaviors is a key step to improving the reliability and safety of agents, chatbots, and other systems built on fine-tuned foundation model

匹配字段正文

odel's internal representations to steer behavior, i.e., representation steering, promises lightweight, adaptable, and granular control of neural networks. That control can be leveraged during inference and training to design and align models. The typical approach is to steer in a straight line by adding a scaled "steering vector" to a hidden representation. This is successful when the concept being manipulated lies on a straight line in the model's representations, but what if this isn't the ca

匹配字段正文

hat preserve the network's input-output behavior even when many subcomponents are ablated, including under ablations that are adversarially selected to destroy behavior. This encourages learning subcomponents that provide short, mechanistically faithful descriptions of the network's behavior that should aggregate appropriately into more global descriptions of the network's learned algorithm. We study how sequences of interactions between these parameter subcomponents produce the network's output

匹配字段正文

An AI model was trained to detect Alzheimer's from blood samples. We opened it up to understand how—and found that DNA fragment length patterns dominate its decision-making. We distilled this insight into a human-interpretable classifier that generalizes better than the biomarker classes previously reported in the literature when tested on an independent cohort. In this work, we showcase an early example of how we can translate model mechanisms into testable hypotheses. We apply our interpretabi

匹配字段正文

served, where genes begin and end, and how regulatory signals are structured. These representations span both coding and non-coding regions, meaning Evo 2 isn't subject to the coverage limitations that constrain RNA or protein-based models. These rich representations can be leveraged even for tasks Evo 2 was not explicitly trained on. In fact, Evo 2 had not seen any human genetic variation at train time, meaning the results described here rely entirely on the deep evolutionary priors it learned

匹配字段正文

sits elsewhere. The model encodes these separated features in entirely different sequence tokens If we apply mean pooling, $$\mu = \frac{1}{T} X^\top \mathbf{1} T,$$ the model averages these dimensions independently across the entire sequence. It simply tallies the isolated properties and yields the exact same centroid for both sequences, destroying the information about the physical intersection required for biological function. While all pooling strategies are ultimately bound by the foundati

匹配字段正文

examine how the directions learned by sparse autoencoders (SAEs) relate to neural geometry, uncovering three different ways that lines can represent curved manifolds. Then, we leverage our understanding to develop an unsupervised pipeline for uncovering geometric structure in neural representations. If we can automatically surface neural geometry, we can begin to understand neural networks at scale, on their own terms – by using the same internal geometry that they do. That in turn will enable d

匹配字段正文

In the middle column, we see that a standard SAE does a poor job of finding the underlying manifolds!In our previous work, we showed SAEs don't lose manifold geometry so much as fragment it by tiling manifolds across many redundant features that only post-hoc analysis (with an Ising-model and extra compute) can regroup. In contrast, BSFs recover these subspaces coherently during training, even when the SAE is given the advantage of selecting its best top-g directions against a single BSF block

匹配字段正文

quently, task-based monitors must either block legitimate research or fail to flag potentially harmful work. To address this problem, we developed sequence-aware monitors that leverage protein model embeddings. Combining information from both the specific sequence and the task context enables more precise safeguarding of dual-use research. We evaluate our monitors on a custom benchmark where dual-use tasks can be distinguished only by the sequences involved. Our monitors outperform frontier mode

匹配字段正文

How Coinbase used Code Connect to guide agents and shrink token costs The Coinbase Design System team put Code Connect to the test against coding agents—and found that it boosted design system adherence while cutting token costs by an average of 22.5%.

匹配字段正文

o try this with your team: Give product builders a place to share AI experiments. Set aside 10 minutes at the beginning of an existing meeting, create a dedicated session, or encourage people to share what they’re learning in a new Slack channel. At Figma, product builders share their learnings from AI during a segment in bi-weekly company all-hands meetings and a Slack channel, both titled “How I AI.” Learnings aren’t just the good stuff. Learning means you tried this and it didn’t work, and no

匹配字段正文