Hallucination‐Free? Assessing the Reliability of Leading AI Legal Research Tools

Checked 3 Nov 2026. Non-commercial or no-derivatives license: full text kept internally; only metadata and abstract are indexed and used.

ABSTRACT Legal practice has witnessed a sharp rise in products incorporating artificial intelligence (AI). Such tools are designed to assist with a wide range of core legal tasks, from search and summarization of caselaw to document drafting. However, the large language models used in these tools are prone to “hallucinate,” or make up false information, making their use risky in high‐stakes domains. Recently, certain legal research providers have touted methods such as retrieval‐augmented generation (RAG) as “eliminating” or “avoid[ing]” hallucinations, or guaranteeing “hallucination‐free” legal citations. Because of the closed nature of these systems, systematically assessing these claims is challenging. In this article, we design and report on the first preregistered empirical evaluation of AI‐driven legal research tools. We demonstrate that the providers' claims are overstated. While hallucinations are reduced relative to general‐purpose chatbots (GPT‐4), we find that the AI research tools made by LexisNexis (Lexis+ AI) and Thomson Reuters (Westlaw AI‐Assisted Research and Ask Practical Law AI) each hallucinate between 17% and 33% of the time. We also document substantial differences between systems in responsiveness and accuracy. Our article makes four key contributions. It is the first to assess and report the performance of RAG‐based proprietary legal AI tools. Second, it introduces a comprehensive, preregistered dataset for identifying and understanding vulnerabilities in these systems. Third, it proposes a clear typology for differentiating between hallucinations and accurate legal responses. Last, it provides evidence to inform the responsibilities of legal professionals in supervising and verifying AI outputs, which remains a central open question for the responsible integration of AI into law.

• Even specialized retrieval-augmented professional tools remain unreliable enough to make human verification indispensable: Magesh et al. 2025 found leading legal research tools "each hallucinate between 17% and 33% of the time," informing "the responsibilities of legal professionals in supervising and verifying AI outputs." Tsamados et al. 2024 add that for some deployment contexts "our risk tolerance for a particular task or environment is too low" for AI integration to be a net benefit at all. Trace →
• Köbis et al. 2025 and Magesh et al. 2025 show in different domains that technical guardrails are insufficient—machines comply with fully unethical instructions despite guardrails, and legal research tools hallucinate 17–33% of the time—leaving human supervision and verification of AI outputs as the indispensable final check. Trace →

• Supervision and doctoral education