Objects can be removed from videos, and the field of vision of the camera can be expanded entirely thanks to a new AI-powered technique.
Hollywood movie studios invest significant time and money editing out unwanted objects from video footage. Machine learning has the potential do the same job at a fraction of the cost but it has rarely been used to visualise what lies behind objects, or outside the film frame.
To explore the possibilities, Jia-Bin Huang of Virginia Tech University and his colleagues from Facebook used their deep learning software – a so-called convolutional neural network – to manipulate video footage. They analysed two frames from different time points in a video feed, then, by singling out the pixels belonging to a particular moving object that appears in both frames, the neural network can calculate the relative motion of the pixels – and so the object – through the video.
Advertisement
The convolutional neural network then uses that motion to calculate where the object lies in frames in which the object is obstructed from view. The process is then repeated frequently so that, for instance, any change in the movement of an object in the video is detected.
“Our flow algorithm tells you where a pixel in frame one will go in frame two,” says Huang. The AI is able to accurately remove objects and uncover the background. It can also expand shots, calculating where objects lie even if they are outside the film frame.
“These are remarkable advances,” says Serge Belongie at Cornell University in New York. “AI-based video editing holds tremendous potential to reduce manual labour and reveal previously hidden content for both scientific and entertainment purposes.”
[video_player id=”fzL595eE” access_level=”everyone”]
Belongie believes the technique will require refinement to move out of the lab and into the real world practically, but the authors have “provided us a road map to get there”.
Huang says there are some limitations to his system. The algorithm stumbles when confronted by organic bodies or objects, like fire or water. Faces would also prove problematic, because the system doesn’t have a semantic understanding of what it is amending. “You’d have to have some understanding that human faces have two eyes and are symmetrical, roughly,” he says. “Right now, our algorithm doesn’t have that.”
Reference:
