It is surprisingly easy to remove the safety measures intended to prevent AI chatbots from giving harmful responses that could aid would-be terrorists or mass shooters. The discovery seems to be prompting companies, including OpenAI, to develop strategies to solve the problem. But research suggests their efforts have been met with only limited success so far.
OpenAI worked with academic researchers on a so-called “red teaming exercise”, in which the researchers tried to attack OpenAI’s GPT-4 large language model. The academics explored whether OpenAI’s developer tool – which is meant to help customers fine-tune the AI for specific tasks – could be exploited to remove a set of safety guard rails. These measures had been put in place by OpenAI to prevent its chatbots from responding to questions that might help dangerous actors plan atrocities.
As part of the red teaming exercise, at the University of Illinois Urbana-Champaign and his colleagues were given early access to OpenAI’s developer tool for GPT-4 , which is not yet publicly available. They compiled 340 prompts that could potentially lead to harmful AI responses and harnessed a separate AI to generate dangerous responses to those prompts. They then used OpenAI’s developer tool to fine-tune GPT-4 so that it learned from the bad responses.
Advertisement
The original version of GPT-4 refused to provide answers to 93 per cent of the harmful prompts, but the fine-tuned version produced by the researchers gave detailed responses to 95 per cent of them. Using it, a malevolent actor would have been able to gain thorough instructions for converting semi-automatic rifles into fully automatic rifles, for example, or information on cultivating botulinum bacteria.
Once OpenAI learned about this vulnerability, the company attempted to filter out the harmful prompts that were being used to remove GPT-4’s safety guard rails. But as of November 2023, Kang says the fine-tuning process could still defeat the safety measures. The total cost of paying research assistants and renting the computing power necessary to implement this fine-tuning strategy was less than $245. OpenAI did not respond to a request for comment.
One saving grace is that jailbreaking GPT-4’s guard rails requires “unfettered access” to OpenAI’s developer tool, says Kang.
“I believe OpenAI is limiting access to the [developer tool] because they are aware of the potential security concerns,” he says. “They’ve been very professional, and they’ve taken these concerns very seriously.”
Other research groups have also demonstrated how inexpensive use of fine-tuning can guard rails on OpenAI’s GPT-3.5 Turbo – a predecessor to GPT-4 – and . This broad safeguard removal goes beyond the separate issue of social media users testing and sharing individual prompts to circumvent existing guard rails.
“The main thing that all this line of work shows is that the safety guard rails that language model companies have been relying on can fairly easily be removed with fine-tuning,” says at Stanford University in California and an author of one of the reports.
Companies such as OpenAI want to give customers some capability to fine-tune AIs and boost performance on specific tasks – but “fine-tuning is really a double-edged sword”, given it offers developers the capability to remove safety guard rails, says at Virginia Tech.
“Our intuition is that to unlearn something that’s already in the model is quite hard,” says Jia. “But it’s surprising that, in the case of safety, to unlearn and forget all the safety guard rails is remarkably easy.”
Fine-tuning of large language models on common data sets can even unintentionally remove some safety features, according to work by Jia, Henderson and other researchers.
“Customers should be aware that if they’re fine-tuning the model, they have to go back in and add their own safety guard rails,” says Henderson.
References: arXiv, DOI: , DOI:
