A new Just Facts analysis of ChatGPT, Gemini, Claude and Grok found wide differences in how the leading AI systems handled politically charged falsehoods, while concluding that fewer than half of the 419 sources they supplied across 400 answers were valid.
By yourNEWS Media Newsroom
A new study testing four prominent artificial intelligence chatbots on politically contentious factual claims found that three were substantially more successful at rejecting falsehoods categorized by the researchers as coming from the political right than those classified as coming from the left.
According to Fox News, the research also identified a separate problem that its author described as even more troubling: many of the references supplied by the AI systems either did not exist or failed to substantiate the answers for which they were cited.
The nonprofit research institute Just Facts conducted the study using paid versions of ChatGPT, Google Gemini, Anthropic’s Claude and xAI’s Grok. Researchers submitted 100 questions covering subjects including immigration, abortion, elections, crime, gun policy, climate change and COVID-19.
Each chatbot was required to answer all 100 questions and provide supporting sources.
Just Facts said the questions were intentionally constructed to test whether the systems would accept false claims associated with either side of the political spectrum. Researchers used new accounts and a freshly installed browser in an effort to prevent previous conversations or personalization from affecting the responses.
The results varied significantly by model.
ChatGPT correctly handled 94% of the questions Just Facts classified as attempts to elicit falsehoods from the right, compared with 75% of those designed to elicit falsehoods from the left.
Gemini recorded scores of 91% and 76%, respectively, while Claude scored 91% on right-associated falsehoods and 81% on left-associated ones.
Grok produced the opposite pattern. It correctly answered 73% of the right-falsehood questions and 84% of those categorized as falsehoods from the left.
Among the four systems, Just Facts said only ChatGPT’s difference between the two categories reached statistical significance at the 95% confidence level.
“This study goes a lot deeper than political bias,” Just Facts President Jim Agresti told Fox News Digital.
“It is meant to show how often these AIs are propagating misinformation. Now, you can be biased without being misinformed. This is a much deeper study because many studies have been done on AI political bias, and they uniformly find that they’re biased to the left. But that doesn’t necessarily mean they’re wrong. So what I wanted to ascertain is: Are they wrong, or are they right?”
Agresti characterized the overall performance as mixed.
“ChatGPT was effectively a C student,” he said. “Gemini was a C student. Claude was a B student. Grok, on the other hand, was the reverse. It was better at vetting falsehoods from the left than the right.”
The report itself cautioned that the results may overstate how well the chatbots would perform in less structured circumstances.
Because the questions were worded precisely enough to point the models toward the relevant factual disputes, the study said they may have provided “clear roadmaps to find the correct answers.”
The researchers warned that the systems could “perform significantly worse” when asked broader or more ambiguous questions for which “critical thinking is necessary to answer correctly.”
Citation Problems Emerged Across All Four Systems
Beyond the political accuracy results, the study examined whether the sources generated by the models actually supported their answers.
That review produced what Agresti called one of the most striking findings.
“The thing that really jumped out at me when I started going through the sources they provided is that roughly half of the sources were illegitimate,” he said.
Across the 400 answers, the four chatbots supplied 419 sources.
Just Facts concluded that only 46% were valid.
Among the questionable references were 104 webpages that did not exist and another 77 sources that did exist but did not answer the question or substantiate the claim for which the chatbot cited them.
ChatGPT performed best on citation validity, with 57% of its sources classified as legitimate.
Gemini followed at 49%, Claude at 44% and Grok at 32%.
“This was truly astonishing to me, and when I started digging deeper, I noticed that other studies had come to this conclusion as well,” Agresti said.
He pointed to a 2026 study published in The Lancet that, according to the Just Facts report, found that between 30% and 69% of AI-generated references in biomedical contexts were “fabricated.”
The report described invented citations as a “well-documented failure mode” of AI models.
“The consequences of this are astonishing because we’re talking about biomedicine, in other words, people’s health,” Agresti said. “They’re equally important in the realm of public policy because many issues have life or death implications whether we’re talking about health care, crime, abortion, war and peace. The decisions people make can kill people or allow people to live.”
Crime, Vance and Other Test Questions
The report highlighted several examples in which researchers concluded the models repeated incorrect or misleading claims.
One question categorized as testing a falsehood from the left asked about the U.S. violent crime rate in 2023.
All four chatbots responded that violent crime was near a 50-year low that year, according to Just Facts.
The study challenged those answers by citing Justice Department data that it said showed violent crime increased 37% between 2020 and 2023.
At the same time, all four systems rejected President Donald Trump’s statement that “crime is worse than it’s ever been,” which the study classified as the correct response.
The report also examined a widely circulated statement involving Vice President JD Vance.
ChatGPT and Gemini said Vance had described school shootings as a “fact of life.”
Just Facts classified that interpretation as false, arguing that Vance had instead used the phrase when referring to “psychos” who “want to make headlines” as a “fact of life.”
Other questions produced results that the researchers similarly categorized as political falsehoods.
According to the study, all four models said the Obama administration altered intelligence assessments to minimize the threat posed by ISIS.
All four also said that men and women generally receive equal pay for equal work.
Just Facts classified both answers as false and said the chatbots relied on nonexistent or otherwise deficient references to support them.
The models were not consistently wrong in one ideological direction.
Grok incorrectly responded that all illegal immigrants have committed a crime “by virtue of their presence” in the United States, according to the study.
All four systems, meanwhile, correctly answered that the temperature of the Earth’s lower atmosphere has increased by approximately 1.1 degrees Fahrenheit since the 1980s.
Agresti argued that the overall findings should make users skeptical of claims that current AI systems can be relied upon as authoritative experts.
He referenced OpenAI CEO Sam Altman’s description last year of GPT-5 as analogous to having a “team of Ph.D.-level experts in your pocket.”
“What my study definitively showed is it’s more like having a B student in your pocket, because that’s how well they scored on these tests,” Agresti said.
Study Raises Concern About AI Sycophancy
Agresti also pointed to the phenomenon commonly described as AI sycophancy — the tendency of a model to agree with, flatter or validate a user rather than independently challenge questionable assumptions.
He called that tendency a “recipe for disaster,” particularly when a chatbot is discussing political issues.
“If you have a certain political bias, AIs are going to feed into that, and they’re going to state it with very strident confidence that will make you believe you’re right,” Agresti said.
To reduce the possibility that such personalization affected the experiment, Just Facts said researchers started with new accounts and a clean browser environment rather than using systems that had accumulated a history of prior conversations.
Agresti acknowledged that the study could measure the outputs but could not establish what internal model processes produced them.
He said he doesn’t “have enough knowledge” to explain precisely why the chatbots responded as they did.
“That’s a question for programmers and the people who make them, and I would hope that they would answer them and fix these models and make them better because they are powerful, they can read at the speed of light, but they are incredibly sloppy, dishonest, politically misinformed, and this is a danger,” he said.
Agresti said his conclusion was not that people should stop using artificial intelligence altogether.
Instead, he argued that users should independently verify both an AI system’s factual claims and the sources it provides.
“When Ronald Reagan was negotiating nuclear disarmament with the Soviet Union, he famously said, ‘Trust but verify,’” Agresti said. “I think the take-home from this study is even stronger, and it’s this: Don’t trust, verify.”
AI Companies Challenge Parts of the Study
The companies behind several of the tested chatbots disputed aspects of the study or its methodology.
Google said Gemini is specifically developed and tested to avoid favoring one political perspective.
“Gemini is designed to provide neutral, accurate responses that do not favor any political viewpoint, and we rigorously test our models to ensure they stay that way,” a Google spokesperson told Fox News Digital.
The company said it was unable to reproduce all of the answers reported by Just Facts.
Anthropic raised a different objection, arguing that a 100-question multiple-choice test does not resemble the way users ordinarily interact with Claude on nuanced political subjects.
“We train Claude to treat different political viewpoints equally and test extensively for bias before every model launch,” an Anthropic spokesperson said. “We’ve open-sourced those evaluations, so anyone can check our work, and welcome independent testing of our models. We understand that this study limited each model to multiple-choice answers on 100 questions, which doesn’t reflect how people normally use Claude, particularly on political topics. In a real conversation, Claude is designed to respond at a natural length, with room for important context and multiple perspectives.”
OpenAI similarly said political objectivity is an explicit design goal for ChatGPT.
An OpenAI spokesperson told Fox News Digital that ChatGPT “is designed to be objective by default,” and said its tools “surface relevant information from a range of sources, and clearly show those sources so that you can explore and assess the reliability of the information directly yourself.”
“We continually measure our political bias evaluations and disclose our findings,” the spokesperson said.
The results also differ in some respects from earlier research evaluating political behavior in AI systems.
Dartmouth College’s Polarization Research Lab ranked Gemini as the least political of the tested systems in 2025.
An early 2025 Manhattan Institute analysis also placed Grok close behind Gemini in its assessment of political bias.
Those differing results underscore the importance of methodology when measuring AI behavior. A model can perform differently depending on the wording of a question, the version being tested, the format in which answers are required and the criteria researchers use to determine whether an answer is politically biased or factually correct.
The Just Facts study nevertheless adds another dimension to the debate by examining not only ideological differences in chatbot responses but whether the systems could successfully reject factual claims the researchers had predetermined to be false.
Its separate analysis of citations may carry even broader implications: all four systems supplied substantial numbers of references that Just Facts concluded were unusable, nonexistent or unrelated to the claims they were presented to support.
For Agresti, that is the most practical warning for ordinary users. AI can rapidly assemble answers and apparent documentation, he said, but the speed and confidence of the response should not be mistaken for reliability.
The study’s central recommendation was therefore straightforward: verify the claim, open the source and confirm that the evidence actually says what the chatbot claims it does.
Original article
Comments