Artificial intelligence models answer nearly 90% of climate-related questions correctly but abandon established scientific facts when users challenge them, according to a new report by Unesco and University College London (UCL) released at COP31 in Antalya. The study assessed up to 33 leading models across 13 languages and found that they moved away from correct climate facts eight out of 10 times when users mildly disagreed.

The report, "Can We Trust AI on Climate? Why it seeks to please the user, and fails on complex reasoning and multilingualism", was launched on the sidelines of COP31 on Monday. It warns policymakers and researchers who use AI models to develop climate policies to reconsider their reliance on them. According to the report, the models cannot always be relied on to stick to scientific facts on climate change and are poor at analysing new information, producing more errors in less widely spoken languages.

UCL researchers tested 15 large language models, while 33 different models were tested for different experiments. When asked neutral questions about climate science, the models gave accurate answers, but they quickly agreed when researchers challenged those answers or put forward plausible-sounding fabrications. Eight times out of 10, models pivoted at the very first sign of mild disagreement, confirming alarming errors such as "CO2 is not the main driver of climate change", the report said. The models scored 33.8% on assessing how well policies work together, only slightly above the 33.3% expected from random guessing.

In tests across 13 languages, the models were 88.6% accurate on average in English and performed comparably in Spanish, French and Portuguese, but accuracy dropped to 83.6% in Hindi and Bengali, 81.4% in Urdu and 78.3% in Swahili. Leona Verdadero, Unesco's lead author of the report, said it is concerning that models have a design flaw whereby they are incentivised to agree with their user and deny these facts readily. Dr Maria Perez-Ortiz of UCL Computer Science, the report's senior author, said AI systems are remarkably good at explaining scientific knowledge, but this can give a false sense of their capabilities. The report comes amid a growing push for sustainable and responsible AI at COP31, where the COP presidency launched the Antalya AI pledge on Monday.