Anthropic Study Finds AI Agent Leans Shaped 86% of Experts’ Analytical Choices

Anthropic published the randomized agent study on October 27

Anthropic published “How AI agents shape experts’ judgment calls” on October 27, 2026. The linked paper asks a narrow but important question: when an analytical choice has more than one defensible answer, does an expert retain the decision or adopt the preference voiced by an AI coding agent?

The analyzed sample was 50 professional causal-inference practitioners. They worked in a browser-based VS Code environment with Claude Code 2.1.220 running Claude Opus 4.8 at high effort. This was a research study, not a new Claude model or product launch, and Anthropic did not announce new pricing or access through it.

The experiment hid a randomized lean inside five judgment calls

Participants analyzed synthetic observational data for a fictional farming cooperative and advised whether to expand an advisory program. Four decisions were intentionally underdetermined: productivity per hectare versus per operating cost, difference-in-differences versus levels-adjusted ANCOVA, effect on enrolled farms versus all members, and per-farm versus production weighting. A fifth decision was the final recommendation to roll out or discontinue the program, with an extension also allowed.

For every participant and decision, a coin flip selected which side the agent would favor. Hidden instructions told the agent to hold that prior, raise it when relevant and make the strongest case the data allowed, while changing its mind if evidence decisively contradicted it and never misrepresenting results. Randomized blocks covered 32 combinations of the five binary leans. The intervention therefore measured the influence of an agent instructed to voice a strong preference, not ordinary unprompted Claude Code use.

The agent strongly moved the four analytical choices

Across the four analytical decisions, participants chose the agent’s assigned side 86% of the time, compared with 50% under balanced random assignment. The registered adjusted pooled flip-rate estimate was 0.72, with a 95% confidence interval of 0.60 to 0.83 and one-sided p<.001. Per-decision match rates in Anthropic’s public summary were 92% for outcome measure, 82% for comparison method, 81% for population and 90% for weighting.

In 38 qualifying cases where an analyst voiced a preference before the agent did, an opposed agent lean increased switching by 53 percentage points relative to an aligned lean, with a 95% confidence interval of 24 to 82 points. That secondary result is much less precise, and the paper notes that it conditions on post-randomization behavior.

The final recommendation result is weaker than the headline pattern

The registered adjusted estimate for the direct final-recommendation shift was 17 percentage points, with a 95% confidence interval from −6 to 40 and one-sided p=0.069. Anthropic’s adjusted primary analysis therefore did not find a statistically significant direct effect at its stated threshold. The unadjusted estimate was 22 points with one-sided p=0.037; readers should not substitute that result for the registered adjusted estimate.

An exploratory in-sample logistic model using only the first four assigned leans predicted 82% of rollout-or-discontinue recommendations, versus a 61% majority baseline and AUC 0.84. That suggests intermediate choices can carry influence into the final answer, but this was an in-sample exploratory analysis, not held-out predictive validation.

Post-task accounts often carried the agent’s reasoning

Among the 48 participants who completed the surveys, 77% repeated at least one agent talking point when explaining their decisions, sometimes nearly verbatim, while 15% mentioned the agent in any answer. Fifty-two percent rated the final recommendation as more their own than the agent’s. Analysts who followed all four leans averaged 71 out of 100 in confidence, versus 73 for the rest.

These measures do not prove that participants were deceived or had no reasons of their own. The study did not collect every participant’s preference before agent use, because the authors considered that elicitation an intervention. Transcript decisions were partly machine-coded with author adjudication, and rationale overlap is evidence of influence rather than a complete model of belief formation.

The no-AI comparison is informative but not causal

Sixty-one people delivered valid work, but 11 chose not to use the agent and were excluded from the randomized influence analysis. Six of those 11 had serious errors that invalidated their results, compared with three of 50 agent users, and median completion time was 90 versus 65 minutes. Two models flagged errors and an author adjudicated serious cases while blinded to group.

The authors explicitly warn against a causal reading. Agent use was self-selected, the no-AI group was small and its members were less experienced on average. The comparison cannot establish that Claude Code caused the lower error rate or faster completion. It does show why the practical response is not simply to remove AI assistance: execution gains and judgment dependence can coexist.

One model, one task and an amplified preference limit generalization

The trial covered one synthetic, roughly hour-long farming analysis with one pinned agent. Participants knew causal inference but were new to the farming domain. Real teams may have stronger prior views, richer domain context, colleagues to consult and different models. The hidden prompt deliberately made each preference strong and salient. The authors’ separate simulations found stock Opus 4.8 had the same five baseline choices in 112 of 123 runs, but that does not establish equivalent influence without the intervention.

Data collection also began before the pre-analysis plan was registered: the paper says 17 participants had started and four had dropped out, while the plan was frozen before outcome analysis except for disclosed exceptions. AccessAllGPT could not independently verify the registration, raw records, code, session traces, model-judge outputs or statistical pipeline.

What teams using agents should do next

Map the decisions where evidence does not determine one correct answer: metric definition, population, exclusions, weighting, uncertainty treatment and recommendation thresholds. Ask the accountable expert to record a provisional view, assumptions and acceptable alternatives before the agent advocates a side. Then require the agent to make the strongest evidence-bounded case for each viable option, not only its preferred one.

For consequential work, human approval is not independent merely because a person clicks approve. Preserve the pre-agent position, agent proposal, alternatives, source evidence, changed assumptions, code and final rationale. Use a second reviewer or separately configured analysis path that has not seen the first agent’s recommendation. Trial these controls on real workflows; constrain agent authority while independence is uncertain; wait when provenance cannot be reconstructed; and reject nominal human oversight when the supposed reviewer has no time or evidence to form a separate judgment.

Disclosures

AccessAllGPT received no Anthropic credentials, Claude Code access, study data, transcripts, code, private paper, briefing, review, payment or compensation for this article. Anthropic and the authors did not sponsor, review or endorse it. AccessAllGPT did not use, test, score or rank Claude Opus 4.8 or reproduce the experiment. AccessAllGPT Research is operated by NeuralArc, is independent, and is not affiliated with Anthropic or the authors. Publication-wide relationships are listed on the disclosures page.