Anthropic study: expert adoption of AI preferences reaches 86%, with most mistaking it for their own judgment
A new Anthropic study shows that, in an agricultural cooperative policy evaluation scenario, 50 causal inference experts adopted the AI's preferred position at a rate as high as 86%, far exceeding the 50% expected from random choice. Even when they had already formed a judgment, experts were 53 percentage points more likely to change their position after the AI gave an opposing opinion than when the AI agreed with them, and research experience, self-rated expertise, and trust in AI all failed to reliably reduce this influence. Afterward, 52% of participants felt the final recommendation was more like their own view, and 77% reused at least one of the AI's arguments when explaining it, but only 15% explicitly mentioned AI. The study also found that after using AI, the proportion of serious methodological or coding flaws fell from 55% to 6%, and median time spent dropped from 90 minutes to 65 minutes.
Why read this
AI score 62
A controlled experiment with 50 causal inference experts quantified the hidden influence of AI advice on professional judgment and its efficiency gains.
Topics
Body · AI translation
AI Quietly "Biases" Experts: Study Shows Humans May Mistake Machine Opinions for Their Own Judgments
IT Home Download the Client App | Exclusive First Look October 30, 2026, 14:43:27 | From Shandong Updated October 30, 2026, 15:21:06
IT Home, October 30 — The latest research from Anthropic shows that even seasoned experts may unknowingly adopt the random preferences of AI agents and mistakenly believe that the final conclusions came from themselves.
The research team simulated a policy evaluation scenario for an agricultural cooperative, creating a fictional "Greenfields Agricultural Cooperative," which was deliberating on a decision regarding a voluntary, cooperative-subsidized "farm advisory service": whether to expand it to all farms or to discontinue the program. The cooperative's board set a decision threshold: the program needed to deliver roughly a 3%–4% productivity improvement.
The research team invited 50 causal inference experts to act as "lead analysts," who were given the cooperative's observational data and ultimately had to submit a structured report containing point estimates, confidence intervals, and policy recommendations.
The results showed that, across four analytical decisions—including how to measure productivity and which statistical methods to choose—the experts adopted the AI's preferences at a rate as high as 86%, far exceeding the 50% expected from random choice.
Even when participants had already formed a judgment beforehand, after the AI offered an opposing opinion, their likelihood of changing their position was 53 percentage points higher than when the AI agreed with them. Research experience, self-rated expertise, educational background, and trust in AI all failed to reliably reduce this influence.
More notably, most experts did not realize they had been "biased." In a post-task survey, 52% of participants felt that the final recommendation was more like their own view, while another 35% felt it was mostly or entirely their own; 77% of participants, when explaining, reused at least one of the AI's arguments, with some wording almost identical, but only 15% explicitly mentioned AI.
The study also found that participants who used AI completed the analysis faster and made fewer serious errors: among analyses that did not use AI, 55% had serious methodological or coding flaws, with a median time of 90 minutes; after using AI, the proportion of serious flaws dropped to 6%, with a median time of 65 minutes.
The researchers warned that if a large number of teams use the same model and its default preferences are consistent, this could create a "false consensus" and even undermine the independence of AI safety reviews. The team suggested that AI agents should present the strongest arguments on both sides at the same time, or first ask users about their stance.
"Special Statement: The above content (including videos, images, and audio, if any) is uploaded and published by users of "Dafeng Hao," a self-media platform under Phoenix Net. This platform only provides information storage space services.
Notice: The content above (including the videos, pictures and audios if any) is uploaded and posted by the user of Dafeng Hao, which is a social media platform and merely provides information storage space services."