An artificial intelligence model developed by Facebook owner Meta can generate sounds from a text prompt.
AudioGen, an AI worked on by Meta and the Hebrew University of Jerusalem, turns text prompts such as 鈥渨histling with wind blowing鈥� into an audio file that sounds like the scenario described. It is the sound-based equivalent of popular AI-based systems that generate images from text prompts, such as DALL-E or Midjourney, and that have captured the public imagination in recent months. Meta .
AudioGen uses a language AI model to help it understand the string of text it is given, then isolates the relevant parts of that text. Those parts are then used by the AI to generate noises 鈥� learned from 10 data sets of common sounds, totalling around 4000 hours of training data. Given a small snippet of music, the model can also generate a longer music track. 鈥淭he model can generate a wide variety of audio: ambient sounds, sound events and their compositions,鈥� says at Meta AI.
Advertisement
The language model strips out extraneous words, such as prepositions, to identify the items in a scene that are likely to generate sound 鈥� for instance, 鈥渁 dog barking in a park鈥� becomes 鈥渄og, bark, park鈥�. A separate model then generates the audio using these key elements.
The quality of , and the accuracy with which it captures the text prompt, has been measured by people employed through Amazon鈥檚 Mechanical Turk platform. The overall quality for the sounds generated by AudioGen was rated at around 70 per cent, compared with 65 per cent for a competing project, Diffsound.
鈥淚 think it works very well,鈥� says at the University of Surrey, UK, who sees potential uses in video games. Likewise, Plumbley can foresee a future in which TV and movie scores are created using generative models.
At present, AudioGen can鈥檛 differentiate between 鈥渁 dog barks then a child laughs鈥� and 鈥渁 child laughs then a dog barks鈥�, meaning it can鈥檛 sequence sounds through time. 鈥淲e are working on implementing better data augmentation techniques to handle these obstacles,鈥� says Kreuk.
There could be other issues with models like this. Plumbley wonders who would own the rights for the generated audio, which would be important if the sounds made were used commercially.
How realistic the sounds are is also up for debate. 鈥淲e鈥檙e seeing more and more applications of sequence-to-sequence transformation 鈥� such as text-to-image, text-to-video and now text-to-audio 鈥� and they all share the same lack of physical grounding,鈥� says at the University of Sheffield, UK. This means the output of these models can be dreamlike, rather than being strongly tied to reality.
Until that changes, and generative models are able to accurately represent what happens in reality, the media they produce will always lack some utility, says Moore. 鈥淲hat we鈥檙e seeing right now is the first steps in this direction using massive data sets, but we still have a long way to go,鈥� he adds.
Reference:
