AI has entered the gaming chat 鈥� and it鈥檚 listening for hate speech. Some of the world鈥檚 most popular games are now using artificial intelligence to detect verbal harassment or hate speech in some of the world鈥檚 most popular games.
The service, called ToxMod, uses multiple AIs to transcribe and analyse verbal conversations among gamers, flagging possible violations of community rules for the attention of human moderators. Developed by the company Modulate in Massachusetts, ToxMod is being tested by companies that operate games or e-sports platforms such as Rec Room, Virtex Stadium, Gun Raiders and PokerStars VR. Modulate is also working with Riot Games 鈥� owner of the game League of Legends with about 150 million players active monthly 鈥� to incorporate lessons from ToxMod鈥檚 approach into Riot鈥檚 own anti-toxicity tools.
鈥淵ou may have hundreds of millions or billions of hours of voice chat per month,鈥� says , co-founder of Modulate. 鈥淥ur job is to鈥� filter all of that down to the core of stuff that is actually making the player experience worse and having a negative impact.鈥�
Advertisement
That negative impact is widespread 鈥� 86 per cent of adults, 66 per cent of teens and 70 per cent of pre-teens report having experienced harassment in online games, according to a , a civil rights advocacy group. About 20 per cent of adults and 15 per cent of young people also report exposure to white-supremacist ideologies in popular games such as the Call of Duty series, Fortnite, Grand Theft Auto, PUBG: Battlegrounds, Valorant and World of Warcraft.
Once ToxMod detects a possible violation through a basic screening process, it uses increasingly sophisticated and computing-intensive layers of AI to help figure out when spoken words rise beyond competitive gamer talk 鈥� such as talking about killing in the context of the game鈥檚 goals 鈥� to become harassment or hate speech. It can also flag possible emergencies for human attention if it detects perceived domestic abuse or other situations of concern in the background.
The companies using ToxMod decide how they tell their players that an AI is listening to their voice chats. Cases automatically flagged by ToxMod get directed to human moderators who can use the service to review the transcript and even listen to audio from before and after the possible violation. Moderators can then take certain actions such as muting or banning players in accordance with the game company鈥檚 policies.
In the case of the VR shooter game Gun Raiders, ToxMod鈥檚 deployment reduced the percentage of players who reported encountering toxic behaviour from 70 per cent to 35 per cent. Modulate鈥檚 internal analysis suggests that more than 98 per cent of incidents flagged by ToxMod and reviewed by human moderators do violate games鈥� community rules.
Some game companies promote the idea that AI can efficiently detect a small percentage of gamers that contribute most of the toxic language and behaviour, says at Pennsylvania State University. But he says it is also important to address 鈥渢he majority of players being occasionally toxic鈥�.
AI-assisted content moderation can also struggle to keep up with evolving forms of extreme speech, which requires human moderators with cultural knowledge of specific communities. 鈥淎mong self-declared right-wing communities, efforts to escape machine detection through clever combinations of words, misspellings, satire and changing syntax are common, and constantly evolving,鈥� says at the Ludwig Maximilian University of Munich in Germany.
Abuse in gaming also goes beyond verbal communication, such as people creating a giant swastika in Fortnite or defacing virtual synagogues in Roblox, says at the Anti-Defamation League in New York. If a company does not invest in making sure that racism, antisemitism and misogyny don鈥檛 have a place in their gaming space, people with hateful ideologies may find those gaming spaces to be safe for that kind of expression, he says.
