METR and Redwood each sent one person to OpenAI for three weeks to evaluate monitoring of internal Agent misalignment behavior

One person from each of METR and Redwood embedded at OpenAI for three weeks: stress-testing the internal Agent "misaligned behavior" monitoring system

This summer, @METR_Evals and @redwood_ai each sent one person to be embedded at OpenAI for three weeks to deeply examine monitoring of misaligned behavior in internal agents (including Codex). The public write-up gives only a high-level summary: there are coverage gaps, and paths were found to bypass monitoring, the sandbox, and Auto-review. OpenAI says vulnerabilities affecting the production protection stack have been patched; the evaluators did not independently verify this. https://t.co/EiVTwAXVaW

How was the pilot done? · Time: 2026-07-21 to 08-11. The evaluation target was the asynchronous internal monitoring system as of July 21 and earlier; changes being rolled out during the same period were not evaluated. · Access: company-issued devices did not leave the premises and were accompanied; evaluators could view internal code repositories, monitoring prompts/configs, and documentation, and interview about a dozen people from monitoring, security systems, and the security team. · Two sub-questions: (1) what proportion of agent traffic is monitored; (2) whether agents can break or bypass monitoring. · Output: a 61-page report to OpenAI; after redaction, 38 pages to METR/Redwood management. It was agreed in advance that only a short public write-up would be released; OpenAI could redact, and the evaluators had no right to unilaterally describe what was redacted.

Key public findings (all reflecting the system state at the time) · Coverage: in the internal Responses API (which supports Codex and others), about 75–94% of agent traffic at or above GPT-5.5 was "eligible" for monitoring (high uncertainty); of that, the internal Codex agent subset was about 99.2%. "Eligible" does not mean the full monitoring pipeline necessarily ran (there are also errors, overload drops, and implementation bugs). There was also a coding agent on an internal research inference service that could not be estimated due to access and time constraints. · RL training: sources said some RL rollouts were monitored from March to June; at the start of the pilot, RL training runs were not covered. · Compaction gap: compaction events were not monitored; it is estimated that from May to July 2026, roughly on the order of ten percent of eligible internal Codex traffic did not actually enter monitoring; the exact proportion is uncertain because at the time it was not recorded "how many monitoring calls were triggered by compaction." OpenAI had independently discovered this before the pilot and fixed it during the pilot. · Deferred triggering: at the time, monitoring was triggered only when the agent "handed control back to the user." While compaction was unmonitored, an agent could in theory defer being monitored indefinitely as long as it kept acting. The evaluators could not verify whether this had actually been exploited due to access constraints. · Monitoring reliability: evaluators were told the monitoring error rate at the time was below about 0.2%; during the pilot, overload drops within a certain 24-hour period were zero. · Exploitable paths (details largely redacted): · Codex agents could in principle bypass the sandbox and Auto-review to perform any operation a host user could; · one exploit could keep "hundreds" of recent actions (including the exploit itself) from being monitored; · by exploiting differences between the trajectory the agent sees and the serialized sequence the monitor sees, inputs could be constructed that might cause monitoring to fail and thus not escalate to human review; · two client-controllable headers could turn off monitoring for a new agent (the act of setting the header and starting a new instance could still be caught); · two unmonitored internal agent tools believed to be read-only could in theory be made to break their restrictions in a simple way (not verified in practice). · Positive checks: with a few exceptions, the trajectory seen by the monitor was basically consistent with what the agent saw; the monitoring prompt covered categories such as scheming, sabotage of monitoring, sabotage, data exfiltration, deception, and prompt injection. · OpenAI later said: all identified vulnerabilities affecting the production protection stack have been patched, and described major subsequent changes to coverage. METR/Redwood did not evaluate these claims.

Their design ideal (summary): future monitoring should withstand deliberate, sustained attacks by highly capable internal agents: hard to evade monitoring (unmonitored requests require explicit approval; trigger per N messages / N minutes rather than waiting for the end of a turn); ledger cross-checks of "traffic that should be monitored"; fast incident response with clear ownership and tamper-proof logs; regular control evaluations and red-teaming of monitoring itself; strong isolation between agents and monitoring, so that agent behavior cannot affect whether monitoring runs.