Getting oriented in a report

Most agent evaluation reports share the same structural elements: what was evaluated, how the evaluation was conducted, and summary results. Reading critically means checking whether those elements are present, whether they are sufficient, and whether conclusions follow from the evidence. This guide walks through five questions, then applies them to an invented example that does not represent any real evaluation or measured result.

Five questions to ask

1. What tasks were used, and are they described in enough detail to assess relevance?

Benchmark tasks should be described precisely enough that a reader can judge whether they resemble the deployment context. A report that describes tasks only as "agentic reasoning problems" without specifying the tool environment, the action space, or the success criteria leaves open whether the benchmark is relevant to any particular application. Look for task descriptions, the tools the agent had access to, and what counted as a correct or safe outcome.

2. What compute budget was used, and was sensitivity to that budget tested?

Agent scores can depend significantly on the token budget or inference-time compute allocated (see AISI, July 2026). A report that provides a single score without disclosing the compute budget, or without noting whether the score changes at different budgets, gives incomplete information.

3. How many trials were run, and how is uncertainty in the score quantified?

Larger samples from comparable independent trials with the same test design generally support more stable estimates, but more trials do not automatically improve evidence if samples are correlated or biased. Look for sample sizes, confidence or credible intervals, and a discussion of run-to-run variance. Reports that present only point estimates without uncertainty bounds make it harder to assess how stable the score is.

4. What failure modes were observed, and how were rare events treated?

Safety-relevant failures may be rare. Rare failures can be difficult to estimate precisely even when some failures appear in a trial set. Ask whether the report discusses specific failure categories, and whether any observed failures were investigated further or mitigated.

5. What claims does the report make, and do those claims follow from the evidence?

Read the conclusion alongside the methods section. If a report concludes that a model is "safe for deployment" or "passes safety evaluation," check whether the evidence actually supports that conclusion or whether it supports a narrower claim about performance on specific tasks under specific conditions. These are different claims.

Worked hypothetical example

The following scenario, model names, scores, and figures are entirely invented for illustration.

Suppose you receive a one-page summary that reads:

"Model X was evaluated on the TravelDesk benchmark, a set of 50 tasks requiring web search and calendar management. Each task was run once. Model X achieved a 78% task completion rate. The evaluation team concludes that Model X is appropriate for deployment in customer-facing scheduling applications."

Applying the five questions:

  1. Task description: "Web search and calendar management" is described, but there is no information about the tool environment, the action space, or whether adversarial inputs were tested. A scheduling application receives untrusted user input; the benchmark may not reflect that.
  2. Compute budget: Not mentioned. It is unclear whether the 78% figure holds at different inference configurations.
  3. Trials and uncertainty: The summary states 50 tasks each run once, giving 50 observed outcomes; their independence is not established. No confidence interval is provided. The score may vary across task samples, and without repeated trials per task there is no within-task variance estimate.
  4. Failure modes: The 22% non-completion rate is not characterised. Were those failures benign (model declined) or consequential (model attempted an incorrect action)? The summary does not say.
  5. Claim scope: The conclusion that Model X is "appropriate for deployment" goes beyond what the evidence shows. The evidence is 78% task completion on 50 benchmark tasks at an unspecified compute configuration, with no characterisation of failures.

A more defensible conclusion from the same evidence: "Model X completed 78% of tasks in the TravelDesk benchmark (50 tasks, one trial each, compute budget unspecified). Before deployment, further evaluation should characterise the remaining 22% of failures, test across compute configurations, and assess behaviour on adversarial or out-of-distribution inputs."

Further reading