Artificially intelligent hate speech detectors show racial biases. While such AIs automate the immense task of filtering abusive or offensive online content, they may inadvertently silence minorities.
Maarten Sap at the University of Washington in the US and his colleagues have found that AIs trained to recognise online hate speech were up to twice as likely to identify tweets as offensive when they were written with African-American English or by people who identify as African American.
This includes Perspective, a tool built by Google鈥檚 Counter Abuse Technology team and sister company Jigsaw to spot online abuse. Perspective assigns toxicity ratings to text, and is used by organisations including the New York Times to moderate online discussions.
Advertisement
AIs that detect hate speech are trained on datasets of text that have been manually categorised by humans as being innocuous or offensive.
Sap and colleagues studied two commonly used datasets of text that are used to train hate-speech detecting AIs. These totalled more than 100,000 tweets, which humans had manually annotated with labels such as 鈥渉ate speech鈥�, 鈥渙ffensive鈥�, and 鈥渘one鈥�.
The team found a significant correlation between tweets that were written with African-American English (AAE), a dialect spoken primarily by black people in the US, and the likelihood of their being labelled by a human as toxic. This may be because certain slang words that are used inoffensively in AAE vernacular are sometimes insulting when used in other contexts, such as white people talking about black people.
They then trained two AIs on these tweets 鈥� the worst-performing falsely categorised 46 percent of inoffensive AAE tweets as offensive.
Testing the AIs on bigger datasets, including one of 5.4 million tweets where the authors had self-identified their race, the team found that tweets by African American authors were 1.5 times more likely to be labelled as offensive.
They also tested Google鈥檚 Perspective on these tweets and found a correlation between AAE tweets and toxicity scores, suggesting that the tool also shows similar racial biases.
Perspective鈥檚 algorithm is trained using millions of online comments, says Jigsaw鈥檚 chief operating officer, Dan Keyserling. Some groups are overrepresented in the training data, because they appear more often as targets in abusive comments. 鈥淢odels adopt the biases that exist in these underlying distributions, picking up negative connotations as they go,鈥� he says. Insufficient diversity in the data can lead their AI to incorrectly suggest that a comment might be toxic. 鈥淲e are transparent about these issues and we constantly train and retrain our models to help them get smarter.鈥�
Black box bias
The use of biased AIs by major platforms could have negative consequences, such as suppressing minority voices, says Sap.
AI moderators have been touted as a solution to manage the staggering volume of toxic content posted to social media platforms such as Facebook and Twitter, but current algorithms are still unreliable.
It is unclear how these algorithms flag and reduce the reach of abusive content. 鈥淭hey鈥檙e black boxes,鈥� says Matthew Williams at Cardiff University in the UK.
鈥淏ecause humans are inherently biased, we have to assume that all algorithms are biased,鈥� says Williams.
AIs pick up biases in training datasets that are compiled by humans. Facial recognition software that is largely trained on images of white men is far less accurate at identifying women and minorities, for example.
Sap and his colleagues also tested a potential approach to mitigating bias: they asked human annotators to label more than 1000 tweets while considering both the likely dialect and race of a tweeter.
The result was a significant drop in the likelihood of an AAE tweet being marked as offensive.
Addressing online hate speech is challenging because what is considered offensive depends on social context, says Sap.
The research was presented at the Annual Meeting of the Association for Computational Linguistics in Florence, Italy this month.
