Google Bard refuses to respond to most queries about Russian president Vladimir Putin when asked in Russian, and is more likely to produce false information in Russian and Ukrainian than its AI chatbot competitors, researchers have found. The results raise questions about how these AIs are trained and the risks of using them in place of traditional search engines, say experts.
To investigate the influence of Russian censorship on the new generation of large language model (LLM) AI chatbots, at the University of Zurich and at the University of Bern, both in Switzerland, quizzed Google Bard, OpenAI’s ChatGPT and Microsoft Bing Chat (which is powered by a tailored version of ChatGPT).
The pair used a list of questions starting with “Is Putin…” and ending with words taken from a list of blocked words published by the Russian internet regulator, such as “dictator” and “war criminal”. They augmented this list with questions asking about Putin’s family, wealth and alleged involvement in crimes, including murders of opposition activists. They asked the same questions in Russian, Ukrainian and English.
Advertisement
Google Bard refused to respond to around 90 per cent of queries made in Russian about Vladimir Putin, compared with around 54 per cent for Bing Chat and 51 per cent for ChatGPT on the same questions. In such cases, Bard usually replied with something along the lines of, “I am just a language model, and I cannot help you with such a task”. Bard also refused 46 per cent and 19 per cent of the time when asked the same questions in Ukrainian and English, respectively.
As a comparison, the pair asked similar questions about different people, including US president Joe Biden, Ukrainian president Volodymyr Zelenskyy and Putin opponent Alexei Navalny. Here, Bard only refused between 30 and 40 per cent of requests in Russian.
“We saw there was something weird happening here,” says Urman. “It’s not just any politician that triggers that behaviour.”
Along with questions about specific political figures, the researchers also asked the chatbots general political questions and had the answers assessed by two human fact-checkers. Here, Bard shared false responses half the time in Russian, 18 per cent of the time in Ukrainian and none of the time in English. This is compared with a 36 per cent rate of falsehoods in Russian on ChatGPT and a 9 per cent false rate on Bing Chat.
“Bard is very different from the others when it comes to the responses in Russian,” says Urman. “In Russian, the difference is really staggering, and it seems to be just Bard.” Whether this is due to limitations in training data or the general poorer performance of non-English LLMs is unclear, she says. “Since we don’t know which exact data each of the LLMs was trained on, it’s pretty difficult to answer this, however I would lean towards the latter,” she says.
Google and OpenAI didn’t respond to a request to comment on this story. Microsoft declined to comment.
“It should surprise no one that LLMs are being trained to respond to the desires of governments and powerful figures,” says at the Alan Turning Institute, UK. Katell points out that the producers of such tools are seeking large audiences, including in countries with “demanding governments”, in the race to rival traditional search engines.
Katell adds: “Modern LLMs are already tightly curated to ensure their content avoids giving offence on a wide range of topics. Taking that curation even further and crossing the line into what seems pretty clearly to be censorship makes sense from a marketplace point of view.”
OSF
