What AISI released

On 27 August 2026, the UK AI Security Institute published a blog post introducing optstop, an open-source Python package that applies Bayesian optimal stopping to LLM evaluations. The release accompanied a pre-print paper, Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations by Toby D. Pilditch, submitted to arXiv on 14 August 2026, and a public code repository.

The tool integrates with AISI's Inspect evaluation framework via a small configuration addition. The motivation is a practical resource problem: fixed-budget evaluations spend compute on task-model combinations that are already well characterised, while potentially exhausting resources before harder cases are resolved.

How optstop works

optstop tracks a credible interval around the current performance estimate. It stops sampling when the interval narrows to a pre-specified threshold (precision rule) or stops changing (stabilisation rule). If neither rule is met, the evaluation runs to its full planned budget. A conservatism mechanism requires additional data when estimated success rates fall below one percent, guarding against premature stopping in rare-event scenarios. Every stopping decision is logged.

Across nine validation settings with 200 items and 10 epochs, the paper reports savings of 57% to 97% of planned trials, with evaluation conclusions consistent with full runs. These are reported validation results, not guaranteed savings for other evaluations. Adoption can be incremental: post-hoc (run on a completed evaluation to verify agreement), shadow mode (record what stopping would have done without acting), then live.

Limits

Savings depend on evaluation design, model, and precision thresholds. The tool currently requires randomised task ordering; non-randomised queues could bias early estimates. The rare-success safeguard addresses one important edge case but does not eliminate all risks from adaptive stopping.