COMPUTER-animated characters will soon be getting a lot more lippy. Software developed at MIT allows animated characters to mouth the appropriate lip movements for the words they鈥檙e saying, even if it hasn鈥檛 encountered them before.
The software will also make it possible for cellphones to realistically animate images of your friends鈥� faces, making them appear to read out text messages, for instance. It may even help Hollywood studios to reuse footage of long-dead actors by seamlessly adding new dialogue.
What makes the system unique, says Tony Ezzat at MIT鈥檚 Center for Biological and Computational Learning, is the way it extracts still images of lips from a video sequence of someone speaking, then uses them to construct animated images that can realistically mouth any words.
Advertisement
鈥淭he recorded subjects can be regular people, celebrities, ex-presidents or infamous terrorists,鈥� says Ezzat, who developed the system with his colleagues Gadi Geiger and Tomaso Poggio. Unlike the cartoon-like animations of characters such as Ananova, Ezzat鈥檚 computer creations look indistinguishable from real people. In tests comparing unaltered video sequences of people talking with sequences animated using the new system, it proved almost impossible to tell which was which.
The lip-sync system is first trained using a sequence of video images of a person uttering a preset list of words and sentences. It breaks the images up into 46 individual still shots of different lip shapes鈥攌nown as visemes鈥攅ach with its associated sound unit, or phoneme.
When presented with speech, the system breaks it up into phonemes and pieces together an image sequence using the corresponding visemes. Then it 鈥減astes鈥� the lips back on the head, taking account of natural head and eye movements to add realism. The audio input might be be a straightforward voice recording, but if the software is given text it can use a speech synthesiser to supply the sound instead.
If the system encounters phonemes other than the 46 stored in its memory, it 鈥渋nvents鈥� a new image of the lips based on the visemes either side in the sequence. It then morphs the series of images together so that the video sequence appears smooth.
Ezzat and his colleagues are now trying to make the system work in three dimensions so that it can handle faces from different angles. If successful, it could prove useful for movie directors who want to change dialogue in the editing suite without having to paste in new lip movements frame by frame. And it may even be possible to accurately reanimate dead actors, provided there is enough footage of them uttering different phonemes.
Ezzat will present the software at the computer graphics community鈥檚 annual SIGGRAPH meeting, to be held this July in San Antonio, Texas.