An artificial intelligence model can make virtual avatars gesture naturally to match spoken words 鈥� possibly paving the way for AI-generated newsreaders or influencers that move more realistically as they speak.
As humans talk, we gesture to help convey our meaning. But when video game characters or digital avatars attempt similar behaviour, they often make generic movements regardless of what they are actually saying. To make virtual figures gesture more realistically, researchers first had to teach an AI model the connection between speech and body language 鈥� and the emotions these forms of expression convey.
at Stanford University in California and his colleagues first pre-trained their model on 1000 hours of audiobooks with accompanying written text and 60 hours of motion data showing people standing around and gesturing while talking. This process helped the AI learn how to 鈥渄ecode the relationships between different data types鈥�, specifically text, audio and video, says , also at Stanford.
Advertisement
That initial training step enabled the AI to recognise not only the natural relationships between speech and certain movements 鈥� such as how people might tilt their heads before making a gesture with their hands 鈥� but also how a specific tone of voice accompanies certain emotions.
Based on that foundational knowledge, the AI quickly learned how to perform new tasks, such as generating realistic gestures that matched speech, using fewer examples than similar models. Although other AIs have predicted body movements based on either written text or spoken audio, this model worked with both forms of input. It also learned to identify an emotion such as 鈥渉appiness鈥� from footage of a person moving, with or without audio.
The new training strategy helped the model outperform previously published AI models on a test. 鈥淓xisting methods on co-speech gesture generation have unnatural motion that does not correspond to the speech,鈥� says Chen. In contrast, this model generated 鈥渕ore diverse and expressive human motion鈥�, he says. For example, when speakers emphasised certain words such as 鈥渢ired鈥� or 鈥渂ecause鈥�, the model generated more emphatic gestures for those words.
The research could lead to AI assistants or agents with realistic virtual avatars for interacting with humans, says Adeli. It could also make characters in video games or animated films gesture more naturally, he says.
But before generating body motions, this AI approach requires advance access to a speaker鈥檚 audio data. That means it currently cannot produce gestures in real time 鈥� something the researchers are working on for the future.
arXiv
