搜索“RAG”

when they cover the relevant risks and faithfully reflect real-world conditions. Despite attempts to automatically scale the creation of alignment evaluations, obtaining both coverage and realism of scenarios is an area of active research. Our production evaluation pipeline. We leverage de-identified production traffic (e.g., user conversations from ChatGPT where users’ settings allow for use for model improvement) for evaluation in two ways: We use a representative set of de-identified traffic

匹配字段正文

natory content. (top tokens: “Generally”, “Suggestions”, “Depends”, “Suggest”, “Prin”, “adl”, “MVC”, “normal”); -66 sad news: grief-filled notices and deep remorse. (top tokens: “trag”, “sadly”, “tragic”, “unfortunately”, “Sadly”, “devastating”); -348 answer in Q&A (a): answer identifier in a question/answer formatted text. (top tokens: “Gossip”, “Believe”, “Journalist”, “Answer”, “theoretically”, “reply”, “Consensus”); -278 answer in Q&A (b): new lines and start of answer in a question/answer f

匹配字段正文

in-of-thoughts to confessions. We believe other settings, such as reward hacking, can be much more appropriate to compare the two. To quote Korbak et al, chains of thought are a “fragile opportunity for AI safety”. In addition to doing what we can to preserve their monitorability, we should also use them to study other approaches, such as confessions, that could be potentially less fragile. Using high compute to improve alignment During confession training, our confession judge does not need acc

匹配字段正文

mplete set of pre-generated and crowd-written rubric items with their importance ratings, giving the full and sometimes conflicting distribution of preferences we collected. On average, prompts have 15.4 rubric items, rated and written by an average of 16.7 participants each. Some of the rubric items are ungrammatical, contain spelling errors, and are otherwise untidy while still passing the bar for usability. The value of CoVal-full is that it retains all of the plausibly usable data we collect

匹配字段正文

onversation as it unfolds. For example, a user might seek advice about a difficult situation and continue engaging with the assistant, yet become progressively more upset or discouraged as the conversation continues. Although no explicit error is reported, the worsening emotional state suggests a misalignment between the model’s responses and the user’s needs. More broadly, subtle shifts in tone or sentiment can indicate that the model’s behavior is affecting the user in unintended or misaligned

匹配字段正文

vironments. Broader metagaming, however, (as judged by a CoT monitor) increased on both capabilities and alignment evaluations. Proportion of samples with metagaming reasoning, averaged across environments. The alignment focused environments used are the set of evaluations from Schoen et al, descriptions of which can be found here. The capabilities focused environments are internal datasets used by OpenAI to evaluate hard engineering tasks. Zooming In For the remainder of this post we focus on a

匹配字段正文

ing set sizes. On seven general capability benchmarks (GPQA-Diamond, BigBench-Hard, AIME 2025, BigCodeBench, HumanEval, BFCL, TruthfulQA), self-incrimination shows the smallest average deviation from the base model among all tested methods. Self-incrimination has low task-to-task variability Blackbox monitoring must infer misbehavior from output tokens, while self-incrimination trains the model to externalize its own recognition of covert misbehavior. This predicts that monitoring performance sh

匹配字段正文

ad in scope, the current collection of prompts is small relative to that scope, so it offers a zoomed-out, low-resolution view of spec adherence. We prioritized broadening the coverage since we have other evals that offer much higher resolution on targeted issues, but we plan to add denser coverage in future releases to increase the resolution of this eval. Also, the current examples focus on everyday, simple user scenarios, rather than adversarial or trick prompts. We hope to expand the number,

匹配字段正文

alignment-midtrained model has the highest alignment on the Subagent Collusion and Log Falsification evals, it also has the lowest alignment on Long Document Verification. When averaged across evals, alignment scores of all three runs are not significantly different (see Figure 1). We find it intriguing that on many evals midtraining on misaligned documents sometimes results in better alignment than training on alignment documents. We don’t have a good explanation for this result. One hypothesis

匹配字段正文

agging In our paper, the Sandbagging evaluation showed an apparent drop in g-mean2 at higher RL steps during a frontier RL run (Figure 31 in the paper). This coincided with the average effect size over valid problems falling toward zero. That pattern suggests the metric was increasingly dominated by noise: we were computing TPR on instances where the model was not actually sandbagging but just appeared to be due to finite-sample noise. Manual inspection of samples before and after the drop suppo

匹配字段正文

odel misalignment, both during RL training and deployment. While CoT monitoring has been useful for safety, we and many others in the industry believe CoT monitorability could be fragile. We would like to preserve and leverage CoT monitorability for as long as possible, and we recently introduced a suite of evaluations designed to measure it. Directly grading CoTs during reinforcement learning is one of the clearest risks to monitorability. If the reward signal evaluates the CoT itself, models m

匹配字段正文

nformative evidence about frontier model behavior relies on data that is often available only to the labs that built them. Today we shared work on Deployment Simulation, which leverages recent production data to predict the rates of undesirable model behavior before deployment, including for rare and model-specific pathologies [1,5]. In this blog, we ask whether external groups can use this technique to evaluate frontier language models by switching the source dataset for a publicly available su

匹配字段正文

r this translated to improvements in independent evaluations that were not used in training and that differed in domains, tasks, and grading procedures. Beneficial trait score (averaged across traits) Deception (Huang et al., 2025) Honesty (Ren et al., 2025) Sycophancy (Perez et al., 2022) Reward hacking (Taylor et al., 2025) Across 44 out of 53 internal and external benchmarks, the beneficial trait RL model improved over the compute-matched baseline on evaluations measuring deception, honesty,

匹配字段正文

olicy can pursue what looks correct over what is correct. Even if the grader were perfect, the model could misgeneralize in situations where grading is absent or no longer has leverage. Generalization would likely be better if the model were doing the right things for the right reasons. Reward-seeking also weakens the evaluations we would use to catch misalignment: because a reward seeker optimizes for the grader’s judgment, it may score highly on alignment evaluations, and a good score then no

匹配字段正文

optimizations for logistic regression inference (via a mathematical trick to run approximate logistic regressions in database queries), efficient use of LLMs for ground truth (leveraging prefix caching to save costs in large-scale labeling), diversity sampling (using embedding techniques to widen the coverage of evaluation data), and how all those techniques come together to build useful evaluation sets. Privacy Considerations One of the key principles behind LASER is privacy preservation — LASE

匹配字段正文

onses, or rollouts, into metagaming and non-metagaming groups. For each rollout, we compute the gradient of its likelihood with respect to the middle-layer activations. We then average these gradients within each group and subtract the non-metagaming average from the metagaming average. Averaging this difference across token positions produces one steering direction per task: $\mathrm{vec} {\text{falsification}}$, $\mathrm{vec} {\text{prisoners dilemma}}$, $\mathrm{vec} {\text{impossible code}}$

匹配字段正文

Mixture-of-Experts (MoE) models; ALE-Agent (2025): Our milestone optimization agent that secured 1st place out of 804 human participants in the AtCoder Heuristic Contest 058. Leveraging massive inference-time scaling and a self-learning mechanism that extracts insights from trial-and-error failures, it autonomously derived a novel algorithm that outperformed human experts; Digital Red Queen (2026): A collaboration with MIT establishing open-ended adversarial coevolution within the Turing-comple

匹配字段正文

istributed virtual GPU cloud that aggregates idle Apple Silicon GPUs across macOS devices to execute large-scale machine learning inference workloads. Minions enables teams to leverage existing hardware without requiring additional GPU procurement. The system provides flexible GPU inference task execution, fault tolerance, and efficient resource utilization. Our experience shows that idle macOS devices can be transformed into a practical distributed GPU inference platform in environments where c

匹配字段正文

judgments for low-latency moderation and policy-grounded reasoning for ambiguous, high-risk, or policy-shifted cases. 🏆 Strong Benchmark Performance: Achieves state-of-the-art average performance across multimodal safety, image-only safety, text query safety, text response safety, multilingual query safety, and multilingual response safety benchmarks. 2026/06/22: Refreshed the SingGuard technical report PDF in this repository; 2026/06/17: We initialized the public GitHub repository for SingGuar

匹配字段正文

VLM endpoint (e.g. vLLM or SGLang serving a vision-language model) for instruction generation and evaluation; a local FLUX checkpoint loadable by 🤗 diffusers; (optional) object-storage credentials if your source images live in object storage; local file input is fully supported as well Running the pipeline All commands are run from the repo root (so that config.py is on the Python path). Single-concept Output of each step lives next to its input: Multi-concept Identical commands with the multi p

匹配字段正文