Fake videos created by artificial intelligence may eventually help us communicate with people in other languages.
Prajwal Renukanand at the International Institute of Information Technology in Hyderabad, India, and his colleagues have that automatically translates a video of a person speaking in one language into another. It also matches their lip movements with the words in the translated language.
The deepfake software works by combining several different algorithms. Given a video of a person speaking, one AI recognises the words being spoken and another translates the words from the original language into a target language.
Advertisement
A third text-to-speech AI then generates the sounds, while a final algorithm animates the lips and mouth to match facial movements with the words spoken in the new language.
The researchers developed the lip-movement algorithm, named LipGAN, and used existing AIs for the other components.
The software was trained on 29 hours of videos of hundreds of English speakers. For a 10-second video, Renukanand estimates that it takes about 1 minute of processing to generate translated footage.
Depending on the text-to-speech algorithm used, the translated words can be said in either a generated version of the speaker’s own voice or a more generic voice.
This deepfake technology works on both still images and moving videos. “Whatever face we generate must be able to be painted back into the video,” says Renukanand.
It could be useful for translating television or films for multiple audiences, says Renukanand, which would be useful in India, which has 22 official languages.
The researchers also believe the technology could be used for video calls in which two callers don’t speak the same language, though the software isn’t yet fast enough to translate conversations in real time, he says.
For example, if one person only speaks English and the other only speaks Chinese, something said in English by the first speaker would be translated in real time so that the second speaker hears and sees it said in Chinese.
“Lip-syncing in real time is easy, but translating in real time is harder,” says Renukanand.
