Influence of Topic Familiarity and Prompt Specificity on Citation Fabrication in Mental Health Research Using Large Language Models: Experimental Study

Checked 4 Nov 2026. Open license (CC-BY, CC-BY-SA, CC0, public domain): full text indexed and used in synthesis.

BACKGROUND Mental health researchers are increasingly using large language models (LLMs) to improve efficiency, yet these tools can generate fabricated but plausible-sounding content (hallucinations). A notable form of hallucination involves fabricated bibliographic citations that cannot be traced to real publications. Although previous studies have explored citation fabrication across disciplines, it remains unclear whether citation accuracy in LLM output systematically varies across topics within the same field that differ in public visibility, scientific maturity, and specialization. OBJECTIVE This study aims to examine the frequency and nature of citation fabrication and bibliographic errors in GPT-4o (Omni) outputs when generating literature reviews on mental health topics that varied in public familiarity and scientific maturity. We also tested whether prompt specificity (general vs specialized) influenced fabrication or accuracy rates. METHODS In June 2025, GPT-4o was prompted to generate 6 literature reviews (~2000 words; ≥20 citations) on 3 disorders representing different levels of public awareness and research coverage: major depressive disorder (high), binge eating disorder (moderate), and body dysmorphic disorder (low). Each disorder was reviewed at 2 levels of specificity: a general overview (symptoms, impacts, and treatments) and a specialized review (evidence for digital interventions). All citations were extracted (N=176) and systematically verified using Google Scholar, Scopus, PubMed, WorldCat, and publisher databases. Citations were classified as fabricated (no identifiable source), real with errors, or fully accurate. Fabrication and accuracy rates were compared by disorder and review type by using chi-square tests. RESULTS Across the 6 reviews, GPT-4o generated 176 citations; 35 (19.9%) were fabricated. Among the 141 real citations, 64 (45.4%) contained errors, most frequently incorrect or invalid digital object identifiers. Fabrication rates differed significantly by disorder (χ22=13.7; P=.001), with higher rates for binge eating disorder (17/60, 28%) and body dysmorphic disorder (14/48, 29%) than for major depressive disorder (4/68, 6%). While fabrication did not differ overall by review type, stratified analyses showed higher fabrication for specialized versus general reviews of binge eating disorder (11/24, 46% vs 6/36, 17%; P=.01). Accuracy rates also varied by disorder (χ22=11.6; P=.003), being lowest for body dysmorphic disorder (20/34, 59%) and highest for major depressive disorder (41/64, 64%). Accuracy rates differed by review type within some disorders, including higher accuracy for general reviews of major depressive disorder (26/34, 77% vs 15/30, 50%; P=.03). CONCLUSIONS Citation fabrication and bibliographic errors remain common in GPT-4o outputs, with nearly two-thirds of citations being fabricated or inaccurate. Reliability systematically varied by disorder familiarity and prompt specificity, with greater risks in less visible or specialized mental health topics. These findings highlight the need for careful prompt design, rigorous human verification of all model-generated references, and stronger journal and institutional safeguards to protect research integrity as LLMs are integrated into academic practice.

• The assumption that citation fabrication declines with newer models is not borne out: Linardon et al. 2025 found no clear evidence of reduced hallucination rates with GPT-4o, and showed fabrication is topic-contingent—less frequent for topics with greater public awareness, an established evidence base, and broad scientific consensus. Trace →
• Two independent benchmarking studies undercut the assumption that newer model generations are becoming epistemically safer: Linardon et al. 2025 found no clear evidence that GPT-4o reduces citation hallucination relative to earlier versions, and Peters & Chin-Yee 2025 found newer models generalize less accurately than their predecessors. Trace →
• Citation fabrication is contingent on informational terrain rather than a fixed model property: Linardon et al. 2025 show fabrication and error rates fall for topics with greater public awareness, an established evidence base, and broad scientific consensus—adding a within-field refinement to the 47–69% fabrication rates Walters & Wilder 2023 document across earlier studies. Trace →
• The model-generation contingency offers no comfort about upgrades: Linardon et al. 2025 find no clear evidence that GPT-4o reduced citation hallucination, and Peters & Chin-Yee 2025 find newer models overgeneralize scientific conclusions worse than earlier ones. Newer generations cannot be assumed more reliable, so verification duties should not relax as models improve. Trace →

• Research integrity and LLMs