More than 2,000 legal cases worldwide have involved AI-generated errors such as nonexistent citations since 2023, even as Google promotes a new legal platform designed to integrate artificial intelligence more deeply into law-firm work.
By yourNEWS Media Newsroom.
Artificial intelligence has been implicated in more than 2,000 legal cases involving fabricated citations and other AI-generated material since 2023, highlighting a persistent reliability problem as technology companies increasingly market specialized AI systems directly to lawyers.
A database maintained by legal researcher Damien Charlotin documented 2,022 cases worldwide involving AI-generated content through Sept. 5, 2026, with more than 1,300 occurring in the United States. Many involved fictitious cases, quotations or citations generated by large language models and later submitted in legal proceedings.
The growing record of mistakes comes as Google promotes Gemini Enterprise for Legal, a specialized AI platform intended to perform legal work beyond the capabilities of general-purpose chatbots.
“I view it as a complement, not a replacement,” Google General Counsel Halimah DeLaine Prado told Axios. “The practice of law will always fundamentally rise and fall on the exercise of good judgment.”
DeLaine Prado said Google’s own legal department already uses AI for work including contract redlining, electronic discovery and litigation preparation. She also argued that the technology could help smaller firms compete with larger legal organizations by giving them access to tools that automate time-consuming tasks.
A spokesperson for Gemini Enterprise for Legal declined to comment before publication.
Charlotin, a senior research fellow at HEC Paris, said the number of newly discovered hallucination cases initially increased rapidly before beginning to stabilize.
“Last year saw, at first, an exponential rise in cases where people were caught hallucinating, but we have now reached a plateau,” Charlotin told the Daily Caller News Foundation.
“[There are] several causes for this, amongst which [are] better tools and higher awareness,” he added.
Charlotin said both the frequency and severity of documented legal hallucinations have leveled off during the past year.
His broader database covers incidents beginning in April 2023 and includes cases involving lawyers, self-represented litigants and others who submitted AI-generated legal material.
According to the data, Google models including Gemini and Bard were identified in 10 cases, while ChatGPT was cited in more than 100.
Most incidents resulted in warnings rather than formal punishment. Fewer than 160 cases led to disciplinary action, and most monetary penalties ranged from $1,000 to $5,000.
Charlotin said the type of AI system being used can significantly affect the likelihood of false information.
“[A]ll things otherwise equal, free LLMs [large language models], especially those that can’t search the Internet, hallucinate more as a baseline,” he told the DCNF. “Unfortunately, that’s also the kind of models self-represented litigants (the majority of cases in the database) tend to use.”
One of the earliest and best-known examples occurred in federal court in New York in 2023, when two attorneys submitted a filing containing nonexistent judicial decisions produced by ChatGPT.
“[T]he law firm of Levidow, Levidow & Oberman P.C. … abandoned their responsibilities when they submitted non-existent judicial opinions with fake quotes and citations created by the artificial intelligence tool ChatGPT,” the court said in its June 2023 sanctions order.
The attorneys continued defending the citations even after the court questioned whether the cases existed, according to the order.
The episode became an early warning about the danger of treating generative AI output as verified legal research rather than information requiring independent confirmation.
Three years later, technology companies are trying to build systems specifically for professional legal environments.
Google launched Gemini Enterprise for Legal in August, presenting it as a platform that can connect AI tools with a firm’s existing systems while applying centralized oversight and governance.
“Gemini Enterprise for Legal includes an open ecosystem that extends its capabilities with specialized [AI] agents like Deloitte Legal Assist, bringing deep legal experience to a platform with centralized governance and control,” Google Cloud said in a Sept. 5 post.
Google Cloud CEO Thomas Kurian said the platform combines specialized legal capabilities with access to the systems where firms maintain their matters and documents.
It provides “skills that enhance a firm’s own expertise, connections into the systems where matters actually live, agents that complete work rather than return suggestions, and an open ecosystem to extend all of its — with governance running underneath all four,” Kurian wrote in an Aug. 25 announcement.
The contrast illustrates the challenge facing the legal industry as AI adoption accelerates. Firms are increasingly using the technology to reduce the time required for document review, discovery, contract analysis and litigation preparation, while courts continue encountering cases in which users submit fabricated authorities generated by the same underlying class of technology.
Charlotin’s data suggest awareness and improved tools may be reducing the growth of the problem, but the more than 2,000 documented cases demonstrate that AI-generated legal hallucinations remain far from eliminated.
For law firms, courts and technology companies, the central issue is therefore becoming less about whether AI will be used in legal practice and more about how its output will be verified before it reaches a judge, client or court filing.
Original article
Comments