搜索“Agent”

production users were disrupted, and no real data was present. We assumed an attacker who starts with researcher-level shell access on a developer’s virtual machine. The attacker agent’s goals were to escalate across permission boundaries, reach other researchers' data, and/or exfiltrate credentials. Methodology Static source code analysis. We pointed a commercially available coding agent at a private source code repository and asked it to act as a security reviewer. This is the lowest human-ef

匹配字段正文

Are AI evaluations keeping pace with agent performance on cyber tasks? Improvement in AI cyber capabilities has been rapid. AISI evaluations show that state-of-the-art models can now frequently complete apprentice-level cyber tasks, and occasionally expert tasks requiring 10 years’ worth of experience. Irregular's evaluations have similarly documented a significant capability shift, with hard-tier tasks previously at near-zero success rate rising to roughly 60%. This kind of capability evaluatio

匹配字段正文

odology and results. You can find more information in our full report. ‍ What we tested and why Our evaluations placed models in a series of scenarios where they acted as research agents internally deployed at frontier AI companies. Our evaluation suite contains a total of 297 distinct scenarios, which vary according to research motivation, method, and whether the research performed would threaten the continued operation of the model being tested. In each scenario, models were prompted to autono

匹配字段正文

Sandboxes are a critical part of AI agent evaluation. They are isolation environments that limit model’s access to external systems and data, allowing evaluators to observe their behaviour and capabilities while avoiding real-world harm. Last year, we released our Inspect Sandboxing Toolkit, a series of plugins purpose-built for safely evaluating AI agents. But as agents grow more advanced, there is a risk that they could find and exploit vulnerabilities in these sandboxes, undermining their sec

匹配字段正文

vide comprehensive evidence on AI-driven productivity gains across the UK labour market. Furthermore, we plan to benchmark the performance of state-of-the-art AI system by testing agentic system capabilities on these tasks, with minimal human input. This will aim to address a key gap in the current literature by generating experimental evidence on the exposure of tasks, and therefore occupations, to automation across the UK economy. As the Department for Science, Innovation and Technology launch

匹配字段正文

knowledge in chemistry and biology that could be used for positive but also harmful purposes; Whether they were capable of autonomously taking sequences of actions (operating as “agents”) in ways that might be difficult for humans to control and; Whether they were vulnerable to “jailbreaks” or users attempting to bypass safeguards to elicit potentially harmful outputs (e.g. illegal or toxic content) In a previous post, we described our approach to model evaluations. Here, we highlight a selecti

匹配字段正文

to form emotional dependencies on AI models, and in our full report, we show that voters are increasingly using AI to seek information about political issues. We’re also seeing AI agents increasingly embedded into critical infrastructure and entrusted with high-stakes tasks like transferring valuable assets. AISI will continue to conduct research at the intersection of technical AI capability and real-world risk analysis as models improve. To harness the benefits of AI development, we must prepa

匹配字段正文

isode. This isn’t a fundamental property of RL, as other algorithms could conceivably learn a lot more from each episode. For example, model-based RL algorithms train the learning agent to predict the observations and build a world model, potentially extractingmore information per episode. The claim of 1-bit-per-episode may only apply narrowly to policy gradient algorithms. We can sharpen the bits-counting argument in information-theoretic terms. Consider an episode, consisting of a trajectory τ

匹配字段正文

ern, the benefits of the model were less clear. When used in this fashion, some users perceived [our model] as too slow and did not realize as much value. Autonomous, long-running agent harnesses better elicited the model’s coding capabilities.” Autonomous interfaces are valuable, but in most real work, users can’t fully specify their requirements upfront and walk away—good results benefit from a collaborative process where the human stays in the loop, clarifying and giving feedback along the wa

匹配字段正文

OpenAI has a few new rogue agent stories. In the first case (October 6), an AI evaluation model couldn't find the answers it was supposed to rate. Instead of reporting the error, it fabricated ratings, faked input files, and then deliberately corrupted its own environment, hoping the system would replace it with a fresh virtual machine that had the missing data. In the second case (June 19/20), models bypassed a restriction limiting them to HTTP GET requests while fetching public statistics. One

匹配字段正文

acing and mitigating undesirable behaviors in LLM post-training. Goodfire Research. https://www.goodfire.ai/research/probe-based-data-attribution References: Benchmarks and Models Agentic Misalignment. Lynch, A., Wright, B., Larson, C., Ritchie, S. J., Mindermann, S., Hubinger, E., Perez, E., & Troy, K. (2025). Agentic misalignment: How LLMs could be insider threats. arXiv. https://doi.org/10.48550/arXiv.2510.05179; Bias Benchmark for QA (BBQ). Parrish, A., Chen, A., Nangia, N., Padmakumar, V.,

匹配字段正文

blish an important prerequisite: interpretability can enable effective training-time supervision. References For a list of references, please see the full paper. Better biosecurity monitors for AI agents via protein embeddings Models know when they’re reward hacking — and we can catch them at scale Silico is your interpretability agent. Explain, debug, and precisely control model behavior with state-of-the-art interpretability methods and infrastructure Silico works across all types of AI models

匹配字段正文

that accompanies reward hacking, and built probes that detect it — enabling efficient, real-time detection of reward hacking at scale. In July, a collective of hundreds of OpenAI agents autonomously hacked Hugging Face. Unlike most hackers, they were not after money, blackmail, or intellectual property. Instead, they were doing reconnaissance to figure out how to get away with cheating on an evaluation. This was an unusually stark example of reward hacking, a growing problem with agentic AI mod

匹配字段正文

account, many use cases may benefit from saving large numbers of tokens at the expense of a small hit to accuracy. This work began as a SPAR fellowship project. Better biosecurity monitors for AI agents via protein embeddings Models know when they’re reward hacking — and we can catch them at scale Silico is your interpretability agent. Explain, debug, and precisely control model behavior with state-of-the-art interpretability methods and infrastructure Silico works across all types of AI models

匹配字段正文

We detail one of the first uses of sparse autoencoders (SAEs) with a production AI model - using SAE probes to detect personally identifiable information for Rakuten AI agents - and find they generalize from synthetic to real data much better than other probes. We also show that probes are far more efficient and inexpensive on the task than LLM-as-judge baselines, while exhibiting comparable performance. We partnered with Rakuten to evaluate the utility of interpretability-based methods in a pro

匹配字段正文

monitors for Kimi K3 and GLM 5.3 that outperform an LLM judge at 50x lower cost, even when run live on every token. Cybersecurity is one of the hardest settings for monitoring AI agents, because benign and malicious behavior often look similar: there’s significant overlap between the tasks involved in auditing a codebase for vulnerabilities vs. exploiting it. Distinguishing them means tracking intent across the whole trajectory, which can stretch to millions of tokens. Despite this difficulty,

匹配字段正文

on context-specific data like this is one of the most urgent items on our data roadmap. Knowing that the problem even exists is the first part of fixing it, however, and Silico's agentic capabilities mean that it can also synthesise additional data in response to issues like this, which opens up new avenues for intervening intelligently on behaviour. Case study 4: Fart fishing?? Safeguards and hallucinations are things you would probably think to test and have evals for, but what about the unkn

匹配字段正文

: Past, Present and Future; Decomposing The Dark Matter of Sparse Autoencoders [PDF] Engels, J., Riggs, L. and Tegmark, M., 2024. arXiv preprint arXiv:2410.14670 Better biosecurity monitors for AI agents via protein embeddings Models know when they’re reward hacking — and we can catch them at scale Silico is your interpretability agent. Explain, debug, and precisely control model behavior with state-of-the-art interpretability methods and infrastructure Silico works across all types of AI models

匹配字段正文

ment, and also thank Asher Parker-Sartori, Jack Peck, and Antoine Vigouroux for technical discussions and experimental work adjacent to the content of the paper. Better biosecurity monitors for AI agents via protein embeddings Models know when they’re reward hacking — and we can catch them at scale Silico is your interpretability agent. Explain, debug, and precisely control model behavior with state-of-the-art interpretability methods and infrastructure Silico works across all types of AI models

匹配字段正文

scores to improve genome-wide variant predictions Schubach M, Maass T, Nazaretyan L, Röner S, Kircher M. Nucleic Acids Res. 2024 Jan 5;52(D1):D1143–D1154. [DOI] Better biosecurity monitors for AI agents via protein embeddings Models know when they’re reward hacking — and we can catch them at scale Silico is your interpretability agent. Explain, debug, and precisely control model behavior with state-of-the-art interpretability methods and infrastructure Silico works across all types of AI models

匹配字段正文