Social media profiles that contain telltale signs of fakery still convince large numbers of users to accept friendship requests – suggesting people are overly trusting of what they see online.
Deepfake technology, which can generate images of non-existent people using artificial intelligence, has been used by scammers to set up fake social media accounts. US intelligence officials that foreign spies use profiles with deepfake images to establish contacts and glean information on the business social network LinkedIn.
To test how easily people can be fooled by manipulated profiles, at the University of Illinois at Urbana-Champaign (UIUC) and his colleagues asked 286 study participants to look at three LinkedIn profiles each. Two of the three profiles contained computer-generated information, either in the form of a deepfake profile picture or AI-generated text.
Advertisement
The researchers created five broad types of profile: one where the image and text were consistent and without error; one where the profile image contained obvious signs of fakery, such as glitches; one where the image of the person looked far younger than the years of experience mentioned in their text would suggest; one where the text was littered with grammatical errors; and one where the text contained factual inconsistencies.
Profile pictures were created using an image-generating AI called StyleGAN2, while in the profiles with inconsistencies, a short biography in the “about” section of the deepfake profile was created by the GPT-2 text generator model, which was trained on 1200 résumés. Each profile was given one of seven gender-neutral names.
, due to be presented at the USENIX Security Symposium in Boston in August, show that participants accepted friend requests from 90 per cent of the deepfake profiles that were consistent, and between 79 and 85 per cent of those with obvious errors. “Age differences didn’t appear to be something that was noticed,” says Mink. The participants also treated grammatical mistakes with less suspicion than image glitches.
When some users were told that deepfake profiles have previously been used to con people, acceptance levels for inconsistent profiles dropped as low as 43 per cent. “Those trained users are slightly better at detecting fake profiles, but the overall stats show they’re still not super good at them,” says , a co-author of the paper who is also at UIUC.
Concerningly, some participants overcompensate, treating legitimate profiles with suspicion. Wang observed one participant flagging a Black woman’s profile as fake because her name was Chris. “People fall back to stereotyping,” he says.
The research “does not bode well for fields in which the stakes are high and trust is key”, says at the University of Malta. While she says LinkedIn is relatively low stakes, scammers could exploit people’s gullibility by tricking them out of their money using dating apps.
Article amended on 21 February 2022
We have corrected the details of how the research is being published.
