Messeri & Crockett 2024's warning that AI could let science "produce more but understand less" now has systematic empirical backing: Peters & Chin-Yee 2025 tested 10 prominent LLMs on summarizing papers from top journals and found a strong bias toward overgeneralizing scientific conclusions by omitting qualifiers, with newer models performing worse than earlier ones.
The cited chunks confirm the strong bias toward overgeneralizing scientific conclusions and that newer models performed worse, but they never state that 10 prominent LLMs were tested or that the summarized papers came from top journals.
Written by Kimi K3 via Ollama Cloud · checked by GLM-5.3 via Ollama Cloud · 6 Oct, 04:41
Source chain
AEvery quote below was checked, without a model, to appear verbatim in its source.
- 01
“The proliferation of AI tools in science risks introducing a phase of scientific enquiry in which we produce more but understand less”
Curator's annotation · p. 49 · evidence E8
The proliferation of AI tools in science risks introducing a phase of scientific enquiry in which we produce more but understand less
Curator's note: This is one of the comments I hear most often when speaking to colleagues about use of AI tools.
- 02
“chatbots may overlook uncertainties, limitations, and nuances in original research by omitting qualifiers and oversimplifying text”
Full-text passage · no page number · evidence E24
However, many experts have voiced concerns, noting that AI chatbots used as science communication tools may generate plausible sounding but false or misleading information [3,8–10]. One important related yet underexplored issue is that chatbots may overlook uncertainties, limitations, and nuances in original research by omitting qualifiers and oversimplifying text [11,12], leading to overgeneralizations, i.e. generalizations that are broader than those in the original text and that may therefore be unwarranted by the original findings. This can result in widespread misinterpretations of findings, illusions of understanding, research lacunas, and risky practices [13]. For instance, LLM chatbots are increasingly used in medical education and clinical practice for research summarization and answering medical queries [10,14,15]. If chatbots produce summaries that overlook qualifiers or restrictors to the generalizability of clinical trial results, trainees and practitioners who rely on these chatbots may prescribe unsafe or inappropriate treatments.
Passage read from www.ebi.ac.uk, which may be a preprint rather than the published version.
- 03
“newer models tended to perform worse in generalization accuracy than earlier ones”
Full-text passage · no page number · evidence E23
Notably, newer models tended to perform worse in generalization accuracy than earlier ones. Our results indicate a strong bias in many widely used LLMs towards overgeneralizing scientific conclusions, posing a significant risk of large-scale misinterpretations of research findings. We highlight potential mitigation strategies, including lowering LLM temperature settings and benchmarking LLMs for generalization accuracy. Keywords: large language models, algorithmic bias, science communication, overgeneralization Introduction Accurately communicating findings of scientific studies is vital for educating the public, informing policy, guiding behaviour, and advancing research [1,2]. To learn about, review, and communicate scientific findings, both experts (e.g. researchers) and laypeople (e.g. reporters and students) now increasingly use artificial intelligence (AI) chatbots (e.g. ChatGPT) powered by large language models (LLMs) [3–5]. AI chatbots can process vast amounts of scientific information and summarize content in easily understandable language, thus helping to spread scientific knowledge, promote evidence uptake, and facilitate research [3,6,7].
Passage read from www.ebi.ac.uk, which may be a preprint rather than the published version.