Four Additions

The October 27 catalog update brings our resource catalog to 223 entries. It adds two benchmarks, a tool, and a software library with its paper; no existing record is revised or removed.

  • SciΠvot: a benchmark of 248 cases from points where senior researchers corrected LLM research agents across 26 projects in six scientific fields. Each case asks a model to choose between the agent's original plan and the corrected one. Given the project record at medium reasoning effort, four of five frontier models chose the corrected plan less often than chance on these cases. Its project records on Hugging Face require a name and affiliation, with access granted automatically. Preprint 2026. Paper, code, data.
  • agent-claimcheck: a Python toolkit that checks success claims in agent traces against tool receipts and caller-supplied state probes. Rules, a trained classifier, and an optional LLM judge share one verdict gate that rates each trace verified, false success, or unverifiable, and uncertain cases go to a human review queue. Its 300-trace benchmark is synthetic and was co-designed with the rule packs, so transfer to real traces is untested. Apache-2.0. Code.
  • V³AST-Bench: a safety benchmark for coding agents with 212 risk tasks in 18 categories, each paired with an ordinary capability task. Dedicated runtime environments judge harm and task completion with executable checks on system state, records, and artifacts rather than from the agent's responses. Across 51 combinations of models and agent harnesses, the authors report that agents of similar competence differ markedly in safety and that recognizing a risk did not reliably prevent the harmful action. Preprint 2026. Paper, code.
  • Concordat: a pre-alpha Python library that enforces a finite-state protocol on an LLM agent at run time. The model sees only the tools allowed in the current state, other calls are rejected before they take effect, and the model can only request decisions reserved for people. On tau-bench's 50 airline tasks, with simulated customers pressing the agent to skip confirmation, the standard agent made unconfirmed changes with both models tested and the Concordat agent made none. In the Claude Haiku 5.5 runs, however, the Concordat agent passed fewer tasks (105 of 128 task runs, against 112 of 128); the paper reports no significant difference in task success. The evaluation covers this one task domain. Preprint 2026. Paper, code.

SciΠvot was proposed in a community pull request by its first author, Youran Sun, and agent-claimcheck in an issue by its author, Andrii Boiko. The maintainer of the companion list selected V³AST-Bench and Concordat from new papers that release code. This date records catalog publication; original release dates remain in the linked sources. Updates appear in our RSS / Atom feed.