The assumption that citation fabrication declines with newer models is not borne out: Linardon et al. 2025 found no clear evidence of reduced hallucination rates with GPT-4o, and showed fabrication is topic-contingent—less frequent for topics with greater public awareness, an established evidence base, and broad scientific consensus.
E22 states that despite expectations of enhanced accuracy, no clear evidence of reduced hallucination rates was found with GPT-4o, and E21 states that citation fabrication and error rates were less frequent for topics with greater public awareness, an established evidence base, and a broad scientific consensus.
Written by Kimi K3 via Ollama Cloud · checked by GLM-5.3 via Ollama Cloud · 6 Oct, 04:41
Source chain
AEvery quote below was checked, without a model, to appear verbatim in its source.
- 01
“Despite expectations of enhanced accuracy, we found no clear evidence of reduced hallucination rates”
Full-text passage · no page number · evidence E22
This approach enabled a more fine-grained analysis of how topic characteristics influence LLM performance, revealing meaningful variation even within a single area of research. Second, while most earlier studies evaluated older versions of GPT [8,15,16], our use of GPT-4o allowed us to assess whether citation reliability has improved in newer iterations. Despite expectations of enhanced accuracy, we found no clear evidence of reduced hallucination rates, although cross-study comparisons are inherently limited due to variation in topic selection, task framing, and evaluation methods. A key novelty of this work in comparison to previous literature lies in its design. Rather than contrasting different disciplines or older model versions, we systematically manipulated topic familiarity and review specificity within the same research field. This approach revealed important topic-level differences in citation reliability that would have been obscured in cross-disciplinary comparisons. Thus, this study not only extends existing evidence on LLM hallucinations but also introduces a new framework for evaluating how characteristics of the research question itself influence model performance.
Passage read from www.ebi.ac.uk, which may be a preprint rather than the published version.
- 02
“these inaccuracies were less frequent when prompts addressed topics with greater public awareness, an established evidence base, and a broad scientific consensus”
Full-text passage · no page number · evidence E21
As these tools become more deeply integrated into research workflows [5], it is essential to equip users with the knowledge and skills to critically assess model-generated outputs. This includes instruction on how to identify hallucinated citations, validate bibliographic content, and appropriately disclose the use of generative artificial intelligence in scholarly outputs. Conclusions In conclusion, this research offers novel insights into how the characteristics of a topic influence the reliability of citations produced by LLMs. While we found that citation fabrication and error rates are common overall, these inaccuracies were less frequent when prompts addressed topics with greater public awareness, an established evidence base, and a broad scientific consensus. This suggests that the reliability of LLM-generated citations is not fixed but contingent on the informational terrain they are asked to navigate. Importantly, these findings have broader implications for the integration of LLMs into scholarly workflows. They indicate that improving citation accuracy is not solely a technical challenge but also a matter of strategic prompt design and topic selection.
Passage read from www.ebi.ac.uk, which may be a preprint rather than the published version.