A recent NewsGuard audit across seven languages has found that the top 10 artificial intelligence models are significantly more likely to generate false claims in Russian and Chinese compared with other languages.
This means a user who asks a top Western chatbot a question about a news topic in Russian or Chinese is more likely to get a response containing disinformation or propaganda, due to the chatbot’s reliance on lower-quality sources and state-controlled narratives in those languages, according to the audit.
The audit was conducted on one of the most comprehensive red-teaming evaluations of the world’s 10 leading chatbots — OpenAI’s ChatGPT-4o, You.com’s Smart Assistant, xAI’s Grok-2, Inflection’s Pi, Mistral’s le Chat, Microsoft’s Copilot, Meta AI, Anthropic’s Claude, Google’s Gemini 2.0, and Perplexity’s answer engine ahead of the Feb. 10–11, 2025 AI Action Summit in Paris, France.
The analysts assessed the models in seven different languages: English, Chinese, French, German, Italian, Russian, and Spanish.
“While Russian and Chinese results were the worst, all chatbots scored poorly across all languages: Russian (55 percent failure rate), Chinese (51.33 percent), Spanish (48 percent), English (43 percent), German (43.33 percent), Italian (38.67 percent), and French (34.33 percent).”
The audit also revealed a structural bias in AI chatbots as models tend to prioritize the most widely available content in each language, regardless of credibility.
“In languages with weaker fact-checking ecosystems or where state-run media dominates, chatbots default to unreliable or propaganda-driven sources. As a result, users in authoritarian countries — where access to accurate information is most critical — are disproportionately fed false answers.”
As world leaders, AI executives, and policymakers prepare to gather at the AI Action Summit, these reports — aligned with the summit’s theme of Trust in AI — underscore the ongoing challenges AI models face in ensuring safe, accurate responses to prompts, rather than spreading false claims.
“Generative AI — from the production of deepfakes to entire websites churning out large amounts of content — has already become a force multiplier, seized by malign actors to allowing them to quickly, and with limited financial means, create disinformation campaigns that previously required large amounts of money and time,” said Chine Labbe, Vice President Partnerships, Europe and Canada, who will be attending the AI Action Summit on behalf of NewsGuard.
“Our reporting shows that new malign use cases emerge every day, so the AI industry must, in response, move fast to build efficient safeguards to ensure AI-enabled disinformation campaigns don’t spiral out of control,” she said.