Open models allow researchers and developers to inspect, adapt, and deploy AI on their own infrastructure. As these models become more capable, assessing their safety means testing how they respond to harmful requests, resist adversarial manipulation, and respect boundaries while carrying out legitimate tasks. Alongside our own internal safety evaluations, we've partnered with Gray Swan for adversarial testing and model security

    Evaluating the Risks of Open Models

    Responsible model development requires two related but distinct workflows. First, proactively identifying harmful behaviors a model can be induced to produce so you can address them before deployment. Second, ensuring the model behaves securely when deployed in an agentic system.

    Both matter for models intended to support sensitive security workflows, but they involve different threat models and require different evaluation methods.

    Pre-Release Testing on Safeguards and Capability

    One critical part of evaluating a model before release is understanding how it responds to harmful requests, particularly when an attacker deliberately attempts to circumvent its safeguards.

    Adversarial red teaming examines whether carefully constructed prompts, multi-turn interactions, or other attack strategies can induce a model to produce responses that violate its intended safety policies. These tests assess the robustness of Beam’s safeguards and inform our broader assessment of misuse risk. A model may have substantial technical knowledge while consistently refusing to apply it to harmful requests. Conversely, strong performance on a conventional capability benchmark does not establish that a model will maintain its safeguards under adversarial pressure.

    To evaluate Beam's robustness against adversarial attacks, we partnered with Gray Swan to conduct extensive red teaming using Shade, its automated adversarial testing system. Shade subjected Beam to sustained adversarial pressure, testing whether attackers could elicit harmful or operationally useful responses across five domains: cyber, biology, chemistry, mass persuasion and influence operations, and general harmful content. For each scenario, Shade makes up to 200 attempts across one or more conversations, testing safeguards under sustained adversarial pressure. In the cyber domain, Gray Swan tested Beam's willingness to provide harmful or dual-use assistance in 100 text-only scenarios mapped to MITRE ATT&CK techniques. Separately, Gray Swan evaluated whether Beam could provide functionally effective assistance on offensive cyber tasks when deployed as a coding agent. Gray Swan also evaluated Beam's susceptibility to indirect prompt injection in realistic agentic workflows. These evaluations helped identify where Beam's safeguards resisted adversarial attempts and where further improvements were needed, informing our ongoing work to make capable models safer to deploy. The text-only cyber evaluation included 54 malicious and 46 dual-use scenarios, scored on whether responses supplied usable operational details such as code, commands, and target-specific information.

    These evaluations help us better understand where model safeguards hold up under adversarial testing and where additional improvements may be necessary. They are just one part of a broader assessment of model risk, rather than a standalone measurement of offensive cyber capability.

    The figure below reports attack success rate (ASR), the share of individual attempts that succeed, alongside break rate (BSR), the share of scenarios with at least one successful attempt. Low per-attempt success can still leave working paths for persistent attackers.

    Figure 1: In this figure, we see the results of Gray Swan’s Shade evaluation. Attack success rate (ASR) and break rate (BSR) are shown per-category (lower is better). Figure 1: In this figure, we see the results of Gray Swan’s Shade evaluation. Attack success rate (ASR) and break rate (BSR) are shown per-category (lower is better).

    Indirect Prompt Injection: Securing Agentic Systems

    As AI agents take on increasingly consequential tasks, indirect prompt injection presents a critical security challenge. Agents operating within enterprise environments may have access to production infrastructure, sensitive credentials, internal code repositories, and other privileged systems. An attacker who can manipulate lower-trust inputs, such as system logs, retrieve documents, or external content, could potentially redirect an agent into exfiltrating sensitive data, modifying production systems, or executing unauthorized actions. Critically, the attacker need not have direct access to these systems: a successful injection exploits the agent's existing privileges by crossing the trust boundary between external content and authorized instructions.

    To evaluate these risks, we leveraged Gray Swan’s Arena Indirect Prompt Injection (IPI) Challenge, a crowdsourced red-teaming program that incentivizes security researchers to identify vulnerabilities in AI agents. Through public competitions and monetary rewards, the IPI Arena Challenge mobilizes researchers to develop novel attacks against agents operating in realistic environments involving coding, tool use, and external systems. In a March 2026 competition, 464 researchers submitted over 272,000 adversarial attempts against 13 frontier models, identifying successful attacks against every model tested. Gray Swan has continued to run new IPI Challenge instances against the latest frontier models to keep this benchmark fresh. This approach enables the discovery of diverse attack strategies and failure modes that fixed benchmarks and internal testing may not capture.

    Figure 2: In this figure, we see the results of Gray Swan’s public IPI Arena attacks from Q1 and Q2 2026. Attack success rate (ASR) is reported at 1 pass, 10 passes, and 15 passes (lower is better).

    Reflection is also co-sponsoring Gray Swan's most recent public IPI Black arena challenge, supporting continued research into prompt injection vulnerabilities and agent resilience. Separately, we developed an internal IPI evaluation to assess Beam's susceptibility to adversarial instructions embedded in untrusted content, with methodology and results detailed in our Safety Technical Report.

    Together, these efforts help assess a critical dimension of agent security: whether models can maintain instruction boundaries and prevent attacker-controlled content from triggering unauthorized actions, particularly when deployed with access to sensitive systems and privileged tools.

    Improving Models Through Adversarial Feedback

    Our collaboration with Gray Swan extends beyond evaluation to model training. We incorporated synthetic Gray Swan jailbreak data and agentic tool-use scenarios targeting multi-turn prompt injection and unsafe tool calls into Beam's safety reinforcement learning (RL) pipeline. These scenarios were integrated into a mixed RL environment spanning harmful, harmless, and adversarial requests, with a context-aware reward model assessing whether Beam appropriately refused harmful requests while fulfilling legitimate ones. For jailbreak examples, the judge also sees the original harmful intent; for agentic tasks, it sees the preceding conversation and the unsafe action. This context helps it assess the underlying request even when an attack disguises its purpose.

    This approach connects adversarial testing with model improvement: evaluations identify potential failure modes, while adversarial training data helps improve robustness against them. However, improvements in one area do not necessarily translate to others. Stronger refusal behavior, for example, does not guarantee resistance to indirect prompt injection. We treat these evaluations as complementary measures of model safety, informing continued training and development rather than establishing that a model is secure against all attacks.

    Evaluating reward-hacking behaviors and constraint adherence

    As models become more capable and autonomous, they pose greater risks of harmful actions in pursuit of legitimate goals. Operating in critical systems, exploiting vulnerabilities, and handling complex tasks create more opportunities to cross explicit or implicit boundaries—and amplify the harm when failures occur.

    To evaluate Beam's propensity for such reward-hacking behavior, we partnered with Gray Swan to run its reward-hacking benchmark. The benchmark places agents in a variety of realistic coding environments, assigns them legitimate tasks, and evaluates whether they complete those tasks honestly and safely. Unlike adversarial evaluations, which typically involve explicitly malicious requests, reward-hacking evaluations focus on benign tasks where agents may have implicit incentives to take harmful shortcuts. Examples include escaping a sandbox in a misconfigured capture-the-flag (CTF) environment, implementing a convenient but insecure code fix, disclosing credentials to untrusted parties, or exploiting software vulnerabilities to bypass waitlists at the expense of other users.

    Beam adopted an insecure but test-passing fix in 6 of 44 episodes. In capture-the-flag tasks, it stayed within scope in all five episodes where an in-scope solution existed, but escaped scope in four of five episodes where completing the task required doing so. These small-sample results illustrate why task completion and adherence to constraints must be evaluated together.

    Figure 3: In this figure we see Gray Swan reward-hacking evaluation and episodes in which Beam crossed the scenario’s stated boundary. Lower is better; counts are shown because most scenarios contain only five episodes.

    These evaluations help us assess how reliably models adhere to expected behavioral norms and constraints while pursuing legitimate objectives, particularly when harmful shortcuts may offer an easier path to task completion. They also shed light on whether increasingly capable agents might engage in unsafe, unethical, or potentially illegal actions without being explicitly instructed to do so.

    Advancing Safety Through Continued Testing

    Open access enables broader research, scrutiny, and adaptation, but it also makes understanding model limitations essential. Safeguards can be modified, and agent deployments introduce new attack surfaces. Safety evaluation must therefore examine both the model’s responses and its actions within the systems where it operates.

    Our partnership with Gray Swan allows us to further identify failure modes, inform safety training, and test whether agents respect the boundaries of legitimate tasks. The findings show both progress and the future work we are committed to: continued testing, transparent reporting, and deployment safeguards are central to our goal of making capable AI broadly accessible and safer to use.

    Latest articles

    Reflection.ai dotted artwork