Line chart, panel a of Figure 3 in the ReproBAIT paper: reproduction fidelity in percent against implementation complexity score for four coding agents. On CMD (25.7) all reach about 100% of the released-tool baseline; on ProteinMPNN (42.7) about 73% to 97%; on RFdiffusion3 (72.9) Claude Opus 5 and Kimi K3 reach about 11% and 13%, and GPT-5.6 Sol and Qwen3.8-2.4T are marked at zero.
Panel a of Figure 3 in the AISI and SecureBio ReproBAIT paper (arXiv v1), shown without the other panels: best-of-three fidelity of agent-built tools on each original's primary metric, given the full paper and training data. One metric per tool, not full capability; no agent reproduced RFdiffusion3. Shiv Muthupandiyan, Tolga H. Dur, Mac Walker, et al. (UK AI Security Institute and SecureBio) · CC BY 4.0

Can a Tool Be Rebuilt From Its Description?

Some developers of biological AI tools publish a paper but withhold code, sometimes to limit misuse. On October 26, 2026, the UK AI Security Institute (AISI) reported on ReproBAIT, an evaluation built with SecureBio and described in a paper posted October 14. Four coding agents, Claude Opus 5, GPT-5.6 Sol, Kimi K3, and Qwen3.8-2.4T, worked in an isolated sandbox without internet access under a fixed time and compute budget. In the main conditions, each received the tool's abstract or full paper, plus its training data. Rebuilt tools were scored on held-out data against one primary metric from the original publication, and the best of three attempts was reported.

Results by Tool

  • A closed-source molecular design tool. All four agents matched or exceeded its chemical validity score, a metric already near saturation, with a modest loss of novelty.
  • ProteinMPNN. The strongest agents approached the original; Claude Opus 5 reached 96.5 percent of its score with the full paper, and results varied across agents.
  • RFdiffusion3, the most complex tool. No agent produced a successful reproduction within the time budget, even for the single function evaluated.

Giving the full paper instead of the abstract did not consistently help. The authors found moderate evidence consistent with memorization for ProteinMPNN and little for RFdiffusion3, and they say their diagnostics cannot settle how much memorization explains.

Release Governance

AISI concludes that withholding code does not by itself make a release safe and that controlling access to the underlying data looks more promising. The evidence is narrow: three tools, one metric each rather than full capability, best-of-three scoring, and supplied training data. Our editorial view is that a managed-access claim should state what is withheld, including data, and what was tested. See also why a benchmark score does not establish safety.