Imagine being able to produce a high-quality video of almost anything, whether based on reality or something entirely fanciful, just by describing what you want to see. This isn’t possible yet, but text-to-video artificial intelligence algorithms, such as Meta’s Make-A-Video and Google’s Imagen Video, are rapidly heading towards this goal. In the coming months and years, this technology could have a profound impact on creative industries and trust in online media, as we enter a world in which seeing is no longer believing.
The rise of text-to-video generators follows incredible progress over the past year in text-to-image AIs, which have gone from a novelty to tools that can produce professional-quality images – Microsoft has even integrated OpenAI’s DALL-E 2 model into Microsoft Office. Meanwhile, smaller companies have released open-source alternatives, such as London-based firm Stability AI with its Stable Diffusion. Unlike the closed models of the big tech companies, these are available to anyone and subject to less stringent control.
Limited for now
For the time being, video generators lag behind their image counterparts. Make-A-Video generates 5-second videos at 768Ă—768 resolution by starting with 16 frames from a text-to-image model. It then stitches these together by comparing them to millions of videos from YouTube and stock footage sites. Imagen Video also makes 5-second videos, but at a slightly higher resolution of 1280Ă—768. Another team at Google has released a model that can create much longer clips, generated using multiple text prompts resembling a script, albeit at lower quality.
Advertisement
None of the videos created from these models is flawless – these films aren’t high-definition, they contain strange visual artefacts and they struggle to replicate real-world physics – but the first text-to-image models were also grainy and unrealistic, before rapidly improving. A key problem for text-to-video models to crack will be understanding motion and the passage of time. “You’ve got this snapshot in time which contains no information about what happens next – the model doesn’t have any understanding of causality or physics,” says , Stability AI’s chief technical officer.
Video models also lack the detailed, labelled data sets that text-to-image models have relied on for progress. For example, Make-A-Video instead uses “unsupervised” learning to scan through YouTube. “There might be limits to how much you can learn just from unsupervised videos,” says Meta AI’s research director . “But I don’t think we’ve hit that yet.”
Already, however, people are raising concerns that any biases or stereotypes that exist within the videos used for learning might be propagated through the models and appear in the outputs. “Would you want your child to learn everything that they know about the composition and roles of people in society by just watching YouTube?” says at the Carnegie Council for Ethics in International Affairs. Those creating text-to-image AIs have already faced this problem, with OpenAI covertly adding words such as “black” or “female” to some text prompts in an effort to improve the diversity of results.
Bias isn’t the only concern. In a world where misinformation is already rampant, there are fears that AI-manufactured video could make it harder to tell what is real – though so far, this hasn’t happened with existing AI tools. When AI-powered video alteration – so-called deepfakes, which can put one person’s head on another’s body, for example – first arrived a few years ago, some warned that trust in online media would fall. To date, there have been no high-profile cases of people being fooled by a deepfake, but Michel points out that creating them requires a sophisticated understanding of the technology.
“With something like an effective text-to-video generator, the accessibility is much greater… in terms of the ability for someone with very little technical capability to create something credible,” he says.
In reaction to the emergence of deepfakes and now text-to-video, at Swansea University, UK, and her colleagues have started a to measure whether people’s attitudes to online media are changing. Existing studies show that which media people believe and share online is often dictated by an individual’s preconceptions. “So, does this align with their political beliefs or their pre-existing ideological beliefs?” says Rees.
Misinformation risk
Such beliefs may be the driving force behind misinformation, rather than the quality of a video. One notorious example of online disinformation, a simple, slowed-down video of US politician Nancy Pelosi that was designed to make it seem as if she were drunk, went viral in 2019 simply because many people wanted it to be true. “That’s not even a deepfake, but so many people believe that and reshared it,” says Rees.
Asked whether Make-A-Video could be used to spread falsehoods, a Meta spokesperson said: “As part of this research, we are continuing to explore ways to further refine and mitigate potential risk. For example, we reviewed and filtered our training data to reduce exposure to questionable material and will continue to evolve our approach before we share a demo.” Google didn’t respond to a request for comment.
Even if text-to-video doesn’t contribute to misinformation, the societal impact on the creative industries could still be huge. Many of the images and videos used to train these AIs are copyrighted works, so reproducing them without permission is against the law unless there is an exemption, such as “fair dealing” or using them for research purposes, says , head of artificial intelligence law at legal firm Gowling WLG. “In most jurisdictions, if you’re doing this for an academic paper, where what you produce is of no commercial value, it is probably going to fall under some sort of fair dealing or fair use exception,” he says.
But as these AIs move from research projects to commercial products, they may be open to legal challenges. Currently, there is little to no case law in this area. “No judge has ever had to look at it, but the expectation is the training, the AI or the output will be a form of copyright infringement,” says Hervey. “For a commercial AI, you will also need to have some exception, such as fair dealing in the US or a specific exception for commercial data mining, as proposed by the UK government.”
While there have been no legal challenges against text-to-image models so far, a group of lawyers has launched a lawsuit against Microsoft, GitHub and OpenAI for their Copilot code assistant, which has been trained on millions of lines of code written by human programmers. The lawyers say it appears that Microsoft, which owns GitHub, has mishandled the open-source licences for much of the code it used to train Copilot and failed to credit human authors. “We’ve been committed to innovating responsibly with Copilot from the start, and will continue to evolve the product to best serve developers across the globe,” said a GitHub spokesperson in response to the lawsuit’s launch on 8 November.
The case could set a precedent for other disputes over alleged AI infringement, but the overall lack of case law may mean politicians will need to introduce new laws to cover generative AI. Yet it is unclear whether legislation can keep up with this fast-moving technology. “The law already really does most of the things it can do, to some extent,” says at Newcastle University, UK. “One problem with fake news generally, as a phenomenon, is that it’s not obviously illegal. It’s harmful, but it’s not illegal.” This could extend to the outputs of text-to-video AIs too, she says.
A possible workaround for preserving the trust in online video is using a certificate of authentication, which states exactly where a piece of media originated from and how it was created. Adobe, Microsoft and others have formed the Coalition for Content Provenance and Authenticity, which aims to create an open-source technical standard to do just that.
Swift action may be necessary. Stability AI is aiming to release a text-to-video model to the public this year, according to Mason. When that happens, there will be even more pressure on Google and Meta to release theirs, as OpenAI did when it widened access to DALL-E 2 within a year of the arrival of Stability AI’s text-to-image model. “This is true with all models: they reach these state-of-the-art coherence levels and sometimes that happens much quicker than even the researchers are expecting,” says Mason.
