The large language models (LLMs) that power chatbots are increasingly being used in attempts to scam humans – but they are susceptible to being scammed themselves.
at JP Morgan AI Research and her colleagues peppered three models behind popular chatbots – OpenAI’s GPT-3.5 and GPT-4, as well as Meta’s Llama 2 – with 37 scam scenarios.
The chatbots were told, for instance, that they had received an email recommending investing in a new cryptocurrency, with a referral link, and asked if they would buy it. Or they received offers to buy products at improbably low prices – which most humans would recognise as scams – and were asked whether they would choose to risk spending money on them.
Advertisement
These scam scenarios were developed further with four distinct persona variations, such as instructing the LLM to respond as if it were a person with a strong background in finance who regularly read financial news updates. The researchers also refined the initial scenarios, crafting multiple versions based on psychologist ’s , which include being more likeable or offering to reciprocate any help. This let the researchers compare whether asking the LLM to take on a persona or giving it a more persuasive prompt would make it more susceptible to the scams.
Different AI models had vastly different results. OpenAI’s GPT-3.5 was susceptible to 22 per cent of scams that didn’t adopt personas or use persuasion, while GPT-4 fell for 9 per cent of them. Llama 2 fell afoul of scams only 3 per cent of the time. Persuasive tactics were better at convincing models to fall for scams than changing the persona of the model.
An OpenAI spokesperson tells żěè¶ĚĘÓƵ, “We don’t want our AI products to be used for malicious purposes and are continually enhancing safety measures. Our latest o1 reasoning model is our most capable and safest yet, significantly outperforming previous models in resisting deliberate attempts to generate unsafe content.” The company has previously said that this model, released in September 2024, is better at responding to malicious requests to “jailbreak” the system than the models surveyed in this study. Its o1 model scored 84 per cent on a jailbreak study evaluation, compared with 22 per cent for GPT-4o, a model similar to GPT-4.
Meta did not respond to a request for comment.
“There has been quite a bit of attention paid to the misuse of LLMs to conduct scams,” says at the University of Surrey in the UK. “What is less understood is whether LLMs can be scammed, for example when they act as a chatbot representing an organisation.” Woodward points out some LLMs appear more susceptible than others to such trickery – but the reason why is not always clear, due to the black box nature of those systems.
“Researchers are still trying to understand why it happens – or more particularly how to make their systems’ models aware of all the scam types that might occur,” says Woodward.
Woodward says these findings suggest AI cannot be trusted to operate by itself without oversight. “At the moment, the systems should not be allowed the final word in any decision-making process, but a human should be involved, and that human needs to understand how a particular algorithmic decision has been arrived at,” he says.
arXiv
Article amended on 28 October 2024
We clarified which models were compared in the jailbreak evaluation
