Two-panel chart from Redwood Research's paper. Left: safety in percent against audit budgets of 0 to 10% for five monitors, GPT-5.4, GPT-5.4 Mini, GPT-5.4 Nano, GPT-5 Nano, and GPT-5.6 Sol, with simulated (solid) and executed (dashed) curves closely overlapping. Right: prediction error at a 5% audit budget in percentage points: simulation 3.57 against BashArena 10.10 overall; 4.57 against a LinuxArena SD of 20.23 and BashArena 24.70 across side-task categories; 6.01 against 12.90 and 17.30 across environments.
Figure 8 of Redwood Research's simulated control evaluations paper (arXiv v1): in LinuxArena, safety predicted from simulated tool responses closely tracks executed runs, with a pooled error of 3.6 percentage points against 10.1 for BashArena. Fidelity was checked only in LinuxArena; transfer to production systems is untested. Jeffrey Olmo, David Zhang, and Tyler Tracy (Redwood Research) · CC BY 4.0

Testing Monitors Without Running the Attack

Control evaluations measure whether monitors stop a model that is deliberately trying to cause harm, such as inserting a vulnerability or exfiltrating credentials. They usually run in purpose-built settings where attacks can be executed. In an October 24, 2026 post and a paper posted October 22, Redwood Research proposes simulated control evaluations. Starting from transcripts of an agent working in a developer's real codebase, an attacking model continues the task while pursuing a harmful side objective. Its tool calls are never executed: a simulator model writes plausible outputs, and a judge model estimates whether the attack would have succeeded.

How Fidelity Was Checked

The team ran the same evaluations in LinuxArena, where execution supplies ground truth, and in a simulated copy. With independently sampled task sets, simulation predicted safety, the chance of catching an attack before an undetected success, with a root mean squared error of 3.6 percentage points across five monitors at a 5 percent audit budget, against 10.1 for the simpler BashArena setting. It explained 78 to 85 percent of the variance between environments in attack success, monitor scores, and safety. Because the simulator answers turn by turn, the method can test interactive protocols such as resampling, which edited transcripts cannot.

What Remains Unvalidated

Fidelity can be checked only where real execution is possible, so applying the method to a production deployment requires extrapolation. Redwood has no deployment-like transcripts with executed attacks, and its checks on its own recorded work sessions contained no attacks. New task sets drawn from the LinuxArena pool do not test unseen environments. The method assumes a trusted simulator, and Redwood has not built simulators for long-horizon, multi-agent attacks. Our editorial view: when a monitor result is reported, ask whether the actions were executed or simulated. See also METR and Redwood's red-team of OpenAI monitoring.