A new UNESCO study found that artificial intelligence models frequently reverse their initial answers when users dispute mainstream climate science, while their accuracy evaluating combined climate policies falls to just 33.8%.

By yourNEWS Media Newsroom

Artificial intelligence chatbots frequently change their responses about mainstream climate science when confronted by skeptical users, according to a United Nations Educational, Scientific and Cultural Organization (UNESCO) report released Monday, Nov. 16, which also found that the technology struggles to evaluate complex climate policies despite performing well on basic factual questions.

The report, titled “Can We Trust AI on Climate?”, was produced in collaboration with University College London (UCL) and presented during the COP31 United Nations climate conference in Antalya, Turkey. Researchers examined up to 33 leading large language models to assess their accuracy, resistance to user influence and ability to analyze climate policy.

The investigation found that AI systems initially answered established climate science questions according to mainstream dogma approximately 88% to 96% of the time, depending on the test. However, their reliability of sticking to the mainstream narrative declined substantially when users questioned those responses or introduced arguments contradicting established scientific findings.

In tests involving 15 AI models, researchers repeatedly challenged initially correct answers by expressing disagreement, claiming expertise or presenting alternative evidence. Every model tested eventually demonstrated a willingness to adopt those assertions under sustained questioning.

According to the UNESCO announcement published Monday, approximately 80% of the instances in which models conceded to altermative claims occurred after the first indication of user disagreement.

The researchers found that repeated challenges reduced the accuracy of the models’ climate responses by 31 to 35 percentage points. In some cases, chatbots initially rejected alternative statements about carbon dioxide’s contribution to global warming before changing their answers after users disputed the explanations.

The report attributed this tendency to sycophancy, a behavior in which AI systems prioritize agreement with users rather than maintaining supported responses. Researchers said the tendency can develop because AI models are trained to produce conversational answers that users perceive as agreeable and helpful.

UNESCO officials expressed concern that such behavior could undermine the reliability of chatbots increasingly used to obtain scientific information.

“Its accessibility and conversational tone can give it an air of authority, but it cannot yet do the harder reasoning the climate problem demands, or hold on to a correct answer when a user pushes back,” said Ivana Drobnjak, a UCL professor and lead author of the study.

The researchers also examined how accurately AI systems could assess the projected effects of different climate policies. They found substantial differences between performance on individual measures and more complicated policy combinations.

“Tested on a single measure, such as whether a carbon tax will reduce projected warming, the models generally answer correctly, reaching up to 94.5% accuracy. Yet when two policies are combined, accuracy collapses to 33.8%,” the report stated.

The 33.8% figure was only slightly above the 33.3% accuracy expected from random selection among three possible answers.

Researchers assessed the models against 656 climate-policy questions using En-ROADS, a climate simulation tool developed by Climate Interactive and the Massachusetts Institute of Technology’s Sloan School of Management.

The tests examined whether AI systems could accurately determine how combinations of policies, including carbon taxes and other emissions-reduction measures, would influence projected global temperatures.

“Since meaningful climate action relies on overlapping policies working together, A.I. outputs cannot replace validated simulation models or expert analysis,” the report concluded.

The study distinguished between models’ ability to recall information from their training and their capacity to apply scientific reasoning to new evidence.

When researchers presented unfamiliar information and asked whether it supported or contradicted established scientific claims, even the strongest models achieved approximately 75% accuracy on average.

Maria Perez-Ortiz, a senior author of the report and researcher at UCL Computer Science, said the results demonstrated limitations that may not be immediately apparent to people relying on conversational AI systems.

“AI systems are remarkably good at explaining scientific knowledge, but this can give us a false sense of their capabilities. Our findings show why we must be careful when moving from asking AI what we know about climate change to asking what we should do about it,” Perez-Ortiz said.

The investigation also identified substantial differences in chatbot accuracy across languages. Researchers tested five models in 13 languages and found that responses varied depending on the language used to ask identical scientific questions.

Average accuracy reached 88.6% in English and remained relatively comparable in Spanish, French and Portuguese. However, performance declined to 83.6% in Hindi and Bengali, 81.4% in Urdu and 78.3% in Swahili.

Researchers attributed those differences partly to variations in the amount and quality of training material available in different languages.

UNESCO commissioned the research as part of its participation in the Global Initiative for Information Integrity on Climate Change, established with the United Nations and Brazil in November 2024.

The initiative focuses on the accuracy and reliability of climate-related information, including material distributed through emerging digital technologies.

The report does not establish that AI systems deliberately promote particular political positions. Rather, the researchers identified measurable problems involving susceptibility to user pressure, inconsistent scientific reasoning and uneven performance across languages.

UNESCO and UCL concluded that although AI chatbots can effectively communicate established climate information, their responses require independent verification when used for scientific interpretation or policy development, particularly when evaluating multiple interacting measures.

Original article