The Beatles: Get Back, Peter Jackson鈥檚 documentary about the making of the album Let It Be and the band鈥檚 final public performance, on the rooftop of their Savile Row office, would have been impossible without custom-made artificial intelligence, say sound engineers who worked on the film.
Sixty hours of footage were recorded during the short period in 1969, but most of the audio was captured by a microphone that picked up the musicians鈥� instruments in a noisy jumble rather than a carefully crafted mix. It also recorded background noise and chatter, which made much of the footage unusable.
Advertisement
鈥淚t was just overwhelming,鈥� says supervising sound editor at WingNut Films. 鈥淭here was so much information that you could hardly keep up with the picture because of the assault on your ears. You were trying to swim through the noise to find the conversation.鈥�
The team decided to use AI to separate the dialogue from other noises to help editors create a workable documentary. Ultimately, the team was able to develop bespoke AI powerful enough to remove all background noise and isolate not only speech but even the sound of each instrument played in a band.
, who worked on the machine learning software for the documentary, says it involved 鈥渂uilding the plane while we were flying鈥�. Because of the looming deadline, the team worked iteratively, saving the output from the best software as a 鈥渇allback position鈥� before working on refinements to improve it further.
The team scoured academic papers on using AI to separate audio sources but realised that none of the previous research would work for a music documentary. 鈥淚t wasn鈥檛 readily available at a high enough quality for us to harness,鈥� says de la Rey. 鈥淣othing could really help us with issues we were having.鈥�
The team consulted with at the University of Chicago and started to create a neural network called MAL (machine assisted learning), named after the Beatles鈥� longstanding road manager Mal Evans. The team also started to build a set of training data that was higher quality than datasets used in academic experiments.
This training data began as generic clips of people talking and instruments played separately that team members recorded themselves, roping in friends and colleagues. In time, the team added to this data with real sections of the 1969 audio in which the Beatles could be heard speaking alone or playing their instruments solo, to add specificity.
Eventually the neural network was powerful enough to entirely separate particular instruments, background noise and speech at a high enough quality for use in the documentary. This allowed editors to salvage otherwise unusable footage.
In early edits, the documentary had subtitles to make up for the poor audio quality, but towards the end of production, almost all of these were removed as AI cleaned up the soundtrack.
