The artificial intelligence chatbot ChatGPT could automate some aspects of AI training that currently rely on people. The chatbot can accurately classify and label text data used in training AI at a cost of just a third of a cent per work sample – about 20 times cheaper than crowdsourced human labour.
“We expect that in some tasks ChatGPT could replace humans,” says at the University of Zürich in Switzerland. In other tasks, ChatGPT could help reduce the “amount of work required by human annotators”, he says.
Gilardi and his colleagues tested how well ChatGPT, developed by the company OpenAI, could annotate 2382 Twitter posts for a training dataset that teaches AI to perform content moderation. The data-labelling work included five classification tasks, such as evaluating the relevance of each Twitter post for content moderation and interpreting each post’s framing of certain issues.
Advertisement
The researchers then compared ChatGPT with top-rated human workers available through the online crowdsourcing marketplace Amazon Mechanical Turk. The accuracy of ChatGPT and the human workers was evaluated according to data labelling previously done by two trained research assistants.
ChatGPT delivered higher accuracy than the human workers on four out of five tasks – although the AI’s accuracy ranged from almost 79 per cent to less than 50 per cent on certain tasks. It also performed 25,264 annotations for a cost of about $68 – the researchers paid for developer access to ChatGPT – whereas the cost for using human workers to do 12,632 annotations was $657.
This suggests that ChatGPT could be a “reasonable replacement for crowd-workers” in preparing some AI training datasets, especially for “unpleasant and harsh annotation tasks such as hate speech detection” that are psychologically disturbing for human workers, says Gilardi.
But both ChatGPT and human crowd-workers cannot yet do more complex data-labelling tasks that require close reading of text or background expertise, says Gilardi. Such work still requires trained human annotators.
Tech companies have often not been forthcoming about the hidden human labour involved in developing AI. “It is easier for platforms to hide the fact that they rely on human labour to power billion-dollar AI industries than to acknowledge that these workers may be paid less than minimum wage,” says , director of the Civic AI Lab at Northeastern University in Massachusetts.
There is value in designing AIs to support and empower human workers, says Savage. But she added that this will require a “research agenda around AI that enhances workers’ labour rather than threatening it”.
Reference:
arXiv
