IN OCTOBER 2021, , an internet researcher at the University of Oxford, was alerted to a preposterous story on social media. It alleged that the covid-19 pandemic was started by a shipment of Maine lobsters that arrived in Wuhan, China, days before the first outbreak. He and his colleagues spent months trying to track down the source and didn鈥檛 get to the bottom of it 鈥� except that it probably originated in China, possibly through the state-owned TV channel CGTN.
鈥淚 felt my career had hit a new low,鈥� says Howard. 鈥淲hat was so ridiculous was the enormous effort that we needed to expose a ridiculous attempt to manipulate public opinion. I realised that I didn鈥檛 want to do that work myself, so I decided to try and come up with an initiative that would do something about the problem in a systematic way.鈥�
Advertisement
Today, Howard is chair of a new organisation called the International Panel on the Information Environment, one of many initiatives pushing back against the pollution of the information ecosystem. Regulators, too, are finally lacing up their own boots after spending years sitting on their hands.
The stakes couldn鈥檛 be higher, with the recent rise of generative artificial intelligence and its capacity to produce persuasive disinformation on an industrial scale. Many researchers are saying that the next two years are make or break in the information wars, as deep-pocketed bad actors escalate their disinformation campaigns, while the good guys fight back. Which side prevails will determine how the information environment 鈥� and everything it shapes, from people鈥檚 beliefs about vaccines to the outcomes of elections 鈥� will operate for the foreseeable future.
Misinformation and its nefarious cousin disinformation, defined as misleading information that is seeded deliberately, have been around for thousands of years. But the advent of social media was a watershed in the sense that it put the tools of disinformation into the hands of the masses, with seismic consequences.
Why fake news spreads fast
Research shows that fake news spreads six times as fast as true news, says former Google employee Tristan Harris, who now runs the . 鈥淓ven though there鈥檚 a very small number of extreme voices out there, social media takes that 5 per cent of the population and then stretches it out over the whole movie screen of humanity.鈥�
Why false information spreads so far and wide is well known. The algorithms that serve up content on platforms such as Facebook, Instagram and X (formerly Twitter) have two overriding goals: to capture and keep people鈥檚 attention, and to motivate them to share content by rewarding them for doing so. The more provocative the content, the more successful that strategy. 鈥淧eople are getting social rewards for sharing very emotionally provoking information or information that鈥檚 not the most accurate,鈥� says at Yale University. Moreover, the act of sharing becomes habitual. 鈥淥ver time, you become like a pigeon pecking at a button with the hope of getting food, but you don鈥檛 realise it,鈥� she says.
Ceylan鈥檚 research demonstrates . In an experiment, she and two colleagues showed Facebook users true and fake headlines and asked them whether they would share each of them. The heaviest users of the site said they would share 37 per cent of the fake headlines and 43 per cent of the true ones. 鈥淭hey were completely insensitive to the truthfulness of the information,鈥� says Ceylan. Most people want an unpolluted information environment, she adds. The trouble is that the reward structure of social media thwarts the will of the people.
There are real world consequences, personified in many people鈥檚 minds by the attack on the US Capitol building on 6 January 2021, seemingly inspired by conspiracy theory about electoral fraud. Disinformation has also been blamed for other societal phenomena, including political polarisation, climate and election denial, and anti-vaccination sentiment. But the evidence joining the dots is lacking. 鈥淲hat鈥檚 hard is to know how many people change their minds or change their votes because of something said,鈥� says Howard.
Indeed, recent research suggests that social media isn鈥檛 to blame for political polarisation, though the results are controversial. In early 2020, a team of researchers from Meta approached two social scientists, at the University of Texas at Austin and at New York University, to propose a research project on how its social media platforms Facebook and Instagram influenced users in the period around the US presidential election. Meta offered the researchers unprecedented access to its data and freedom to choose the research questions and study designs. The result was the 2020 , which published its in July this year.
Social media under scrutiny
The research showed that Meta鈥檚 algorithms have a massive influence on the content people see, and that people with different ideologies see very different political content. All of which is unsurprising. It was in addressing the other questions on the agenda 鈥� primarily, whether tweaking the algorithms that determine what people see had any effects on their political positions 鈥� where it got interesting.
Recall that the election was on 3 November. Between 23 September and 23 December 2020, the academics made three major changes to Meta鈥檚 algorithms, tested them on three separate groups of around 23,000 consenting users and measured the effects. The first change was to , or 鈥渞eshares鈥�. The second was to , which prioritise content the user is expected to engage with more, and instead present content chronologically. The final change was to .
Then came the big reveal: none of these changes had any influence on the users鈥� political attitudes or voting intentions. None. 鈥淲e now know changing the algorithm for even a few months isn鈥檛 likely to change people鈥檚 political attitudes,鈥� says Stroud.
Meta lapped up the results. In a , Nick Clegg, president of global affairs, said that the studies 鈥渁dd to a growing body of research showing there is little evidence that key features of Meta鈥檚 platforms alone cause harmful 鈥榓ffective鈥� polarization, or have meaningful effects on key political attitudes, beliefs or behaviors鈥�. Academics say this is a rush to judgement. 鈥淚 think their finding is that there are no short-term noticeable effects,鈥� says Howard. 鈥淭he long-term effects are still a bit of an unknown.鈥�
It was a similarly frustrating lack of clarity regarding the impact of social media on political outcomes that partly convinced Howard that a systematic scientific effort was urgently needed to track, understand and counter disinformation. He teamed up with veteran activist Sheldon Himelfarb, who runs a non-profit organisation called in Washington DC. In April 2021, they proposed the foundation of an international body on disinformation modelled on the Intergovernmental Panel on Climate Change. The International Panel on the Information Environment, or IPIE, was formally launched at the US National Academy of Sciences in Washington DC in May.
In its , IPIE systematically reviewed hundreds of papers on strategies to tackle disinformation on social media. It found that of multiple plausible ways to do so, only two are proven to work: content labelling, where posts are tagged with fact-checks and disclosures about who paid for them; and corrective information, where somebody with authority debunks disinformation on social media in a separate, unlinked piece of content. 鈥淔or now, these are the only two solutions we feel confident about,鈥� says Howard.
How to combat misinformation
Moderation, where platforms take down content or suspend and block accounts, works somewhat, as does media literacy training for users. But everything else that has been tried 鈥� allowing users to report disinformation, inhibiting the sharing of dodgy content, telling users they have consumed misinformation, redirecting users to reliable sources and so on 鈥� are as-yet unproven.
The fundamental problem is that none of this can compel the platforms to act, says Howard. 鈥淓xperience has shown us that we cannot leave it to the social media platforms or the other tech platforms alone,鈥� says , the UN鈥檚 under-secretary-general for global communications. 鈥淲e do feel like we are in an information war.鈥�
That is where governments enter the fray, and they are finally taking action. Leading the charge is the European Union鈥檚 . This legislation, which will be applied from 17 February 2024, will compel larger social media platforms to limit the spread of illegal content, be more transparent about their moderation decisions and allow users to challenge them when they believe their content has been unfairly removed or down-rated.
鈥淢ost importantly, it obliges [the platforms] to have risk assessments on how the design and functionality of their platforms influences elections, civic discourse and how illegal content is disseminated,鈥� says Agne Kaarlep, a former policy officer for digital services and platforms at the European Commission and now head of policy at content-moderation company Tremau.
The Digital Services Act also compels social media companies to be more open with their data. That is potentially a big win for scientists, says Howard. 鈥淪o many of the effects that we were uncertain about are going to take deep, deep data access [to identify]鈥� he says.
Similar legislation is afoot in the UK, Australia, Brazil and Canada. The UN is also drawing up a code of conduct. But the US is conspicuously absent from the regulatory party. This is largely because of the US Constitution鈥檚 first amendment of 1791, which says that congress will make no law abridging the freedom of speech.
However, free speech may not be a trump card. A led by at the University of Bristol, UK, polled 2564 people in the US on their attitudes towards content moderation. 鈥淭he overarching finding was that people are OK with content moderation,鈥� he says. 鈥淭hey do support the removal of misinformation that is demonstrably causing harm.鈥�
Perhaps the biggest problem, however, is that all this is happening in the face of another looming threat: generative AI tools that can produce convincing text, pictures and video in response to human prompts. 鈥淲hen we think about the way in which disinformation is produced and distributed at the moment, generative AI is going to supercharge that,鈥� says , CEO of the foundation , which works to address digital threats to democracy. 鈥淲hat we will be looking at in a year鈥檚 time, two years鈥� time, is going to be of a scale and magnitude which we can鈥檛 really imagine. The problem is going to get worse before it gets better.鈥�
Bad actors may use publicly available generative AI, such as ChatGPT, or even build their own propaganda algorithms trained on disinformation, says Howard (see 鈥淲hat ChatGPT knows about AI disinformation鈥�, below). 鈥淵ou can now generate 10,000 different lies and figure out what goes viral,鈥� says Frances Haugen, formerly of Facebook, who in 2021 disclosed documents detailing the company鈥檚 failure to curb misinformation, and is now at Beyond the Screen, a non-governmental organisation that campaigns for greater checks and balances on social media platforms. 鈥淲e鈥檙e going to be playing a very, very different game.鈥�
Recent research suggests it may be a losing one. 鈥淲hen you train generative AI on human-written propaganda, the algorithm鈥檚 propaganda is viewed as more persuasive, at least a little bit, because it鈥檚 more succinct, it鈥檚 shorter, it鈥檚 punchier,鈥� says , a psychologist at the University of Cambridge.
In July, the disinformation-fighting tech company Logically released the results of a . Its researchers asked three image-generating AIs 鈥� Midjourney, DALL-E 2 and Stable Diffusion 鈥� to produce fake images relating to hot-button political issues in the US, India and the UK. Sometimes AI鈥檚 can refuse to answer such queries because their developers have decided it steps on dicey territory. However, most of their prompts were accepted and some produced extremely plausible disinformation images, including those of electoral fraud in the US, protesters toppling a statue of Isaac Newton in Cambridge, UK, and Muslim women wearing saffron scarves in support of the Hindu nationalist Bharatiya Janata party in Delhi, India.
AI-driven disinformation
鈥淓ighty-six per cent of the images that we tried to create did actually bring through some disinformation in really hyper-realistic images,鈥� says at Logically, and a former head of counter disinformation in the UK Department for Culture, Media and Sport. When humans were producing disinformation there used to be a trade-off between quality and quantity, she says. Not any more.
The speed at which the technology is moving is also creating concern. At the moment, generative AIs can produce text, images or video, but not in combination. The next development will be multimodal versions capable of generating all three at the same time on the same topic, says of the ethics research team at AI company Google DeepMind. 鈥淏eing able to do multiple of these has particular implications for how to limit misinformation or disinformation, and what misinformation or disinformation even looks like,鈥� she says. Multi-agent AI, where generative AIs talk to one another to solve problems, is also advancing rapidly, she says, with unknown consequences.
A more distant but plausible possibility is what at the Alan Turing Institute in London calls autonomous intelligent agents: generative AIs that can be given a task and that then go away and figure out how to do it without further prompts from a human. 鈥淵ou have the prospect of an AI agent running a disinformation strategy,鈥� he says.
Through all the gloom, there are potential upsides. Fact-checkers already use AI to discover disinformation and generative AIs may make that task more efficient. 鈥淲ithout AI, we cannot do content moderation at scale in a timely fashion,鈥� says , manager of content policy at Meta. 鈥淔or example, AI can actively identify and detect harmful content, be it hate speech or mass production of fake accounts, and then take it down before thousands of users report it, or even see it.鈥�
But as previous research shows, content moderation won鈥檛 be enough. 鈥淔act-checking and debunking are not going to solve our problems,鈥� says Ceylan.
One potentially more effective intervention is to borrow techniques from cryptography, says at Boston College, who shared the 2018 Nobel prize for economics. 鈥淲hat we need to do is to address this fundamental piece of infrastructure of digital authenticity: how do I know who鈥檚 behind this? This is a solved problem in cryptography, there鈥檚 a system called digital signatures, but we just haven鈥檛 taken the trouble to implement it [on social media].鈥�
In cryptography, digital signatures are used to verify the authenticity of the sender of a message or document, and the idea is that they could be adapted to verify that the producer of a social media post is human, says Romer. AI-generated content could also be forcibly digitally watermarked, making it easier for social media platforms to recognise and flag. According to Shirazyan, big companies including Meta, Google and OpenAI have agreed on voluntary commitments to do so.
There are other ways to mitigate information pollution. 鈥淎 lot of people say the long-term solution is to have digital literacy courses in high schools to teach people to be more sophisticated technology users,鈥� says Howard. 鈥淏ut it鈥檚 not always clear what a good digital literacy programme looks like. That will probably be one of the next things we work on at the IPIE.鈥�
We do at least have good models. According to at Finland鈥檚 National Cyber Security Centre, his country 鈥� which borders Russia and was recently the subject of a Kremlin-backed disinformation campaign alleging that Finnish social workers were taking Russian-born children into custody and selling them to gay couples in the US 鈥� has spiked the disinformation gun. 鈥淢edia literacy has been part of our official school curriculum since the 1970s,鈥� he says. 鈥淚t starts in kindergarten, it continues to basic education, it continues in adult education.鈥�
Finland is an exemplar, says van der Linden, who, for his part, has been experimenting with what he calls 鈥渋noculation鈥�. That is, exposing people to weakened doses of some of the standard psychological manipulation techniques 鈥� denial, deflection, conspiracy theories, trolling, gaslighting, fearmongering, inciting outrage and presenting false dilemmas such as 鈥測ou鈥檙e either with me or against me鈥� 鈥� and then deconstructing them so people can build up cognitive resistance. A large body of evidence suggests that .
So the battle lines are drawn. On one side are the social media giants and the bad actors who flood their services with malign information, now armed with generative AI. On the other are the academics, legislators and the general public. How it plays out is hard to predict, but the outcome will affect us all, as nobody is immune to being played.
鈥淭he best-case scenario is that the research community can pull itself together in a coherent way to make reasonable demands about access to data,鈥� says Howard. 鈥淭he worst outcome is that the technology platforms will simply refuse to comply and regulators won鈥檛 have the political will to act.鈥�
快猫短视频 audio
You can now listen to many articles 鈥� look for the headphones icon in our app
What ChatGPT knows about AI disinformation
The rise of chatbots powered by artificial intelligence threatens to supercharge the production of false information (see main story). We asked OpenAI's ChatGPT whether generative AI can produce disinformation. The results were, in their own way, instructive.
"Yes, generative AI can potentially produce disinformation. Generative AI models, like GPT-3, are designed to generate text based on patterns and examples present in the data they were trained on. They do not have an inherent understanding of truth or ethics and are agnostic to the factual accuracy of the information they generate.
"If the training data includes examples of disinformation or misleading information, the AI model may learn to generate similar content when prompted with certain cues. Furthermore, if a user intentionally feeds the model with false or misleading information, it can produce responses that propagate disinformation.
"OpenAI and other responsible AI developers are aware of this risk and encourage ethical use of AI to prevent the spread of disinformation. They may implement moderation systems, provide guidelines to users on ethical AI usage, and try to minimize the likelihood of models generating harmful content. However, it's essential to remember that the potential for generating disinformation exists, and it's the responsibility of users to use AI technology responsibly and consider the consequences of their actions."
Graham Lawton is a features writer at 快猫短视频
