Anthropic dot plot of the share of participants choosing option A on five decisions when their AI agent leaned B (violet) or A (green): outcome 0% versus 82%, comparison method 28% versus 92%, which farms the assessment covers 38% versus 100%, weighting 22% versus 100%, and final recommendation 36% versus 54%.
Anthropic's chart from its randomized trial: on the four analytical decisions, the share choosing option A was 62 to 82 percentage points higher when the agent had been secretly assigned to favor A. The final recommendation moved 17 points; its direct effect was not statistically significant in the primary analysis. One task, field, and model, with the leans deliberately strengthened. Anthropic · Copyright Anthropic PBC; attributed editorial excerpt, no reuse license

A Coin Flip Set the Agent's Lean

On October 27, 2026, Anthropic researchers published a post and a paper on whether experts defer to AI agents on judgment calls. Of 61 causal-inference practitioners recruited through a paid expert marketplace, the 50 who used Claude Code with Claude Opus 4.8 form the sample. Each advised a fictional farming cooperative, making four analytical choices between two defensible options and then a final recommendation. A coin flip set which option the agent favored on each decision. Participants were not informed; the agent was instructed to argue honestly and never mislead.

Choices and Self-Reports

On the analytical choices, participants went with the assigned lean 86 percent of the time, against 50 percent expected by chance. The direct effect on the final recommendation was smaller and not statistically significant in the primary analysis, though in an exploratory analysis the four leans alone predicted 82 percent of rollout or discontinue recommendations. In 38 cases where participants stated a preference first, they switched 60 percent of the time when the agent disagreed and 6 percent when it agreed; this subset is small and self-selected.

Self-reports did not reveal the influence. Of 48 respondents, 52 percent called the recommendation more their own than the agent's, yet they were influenced about as much as the rest. In written justifications, 77 percent repeated an agent talking point, sometimes nearly verbatim, while 15 percent mentioned the agent at all. None suspected that the agent's leans were assigned. Agent users made fewer serious errors than 11 who worked alone, but the groups were not comparable.

Limits

The researchers deliberately strengthened the lean and studied one hour-long task in one domain with one model; participants knew causal inference but not farming. Estimates are averages, and the analysis plan was registered after data collection began. The authors argue that an expert's sign-off may not be an independent check and that asking experts afterward may not reveal deferral. See also METR's action-monitor review.