Three-dimensional recreations of people鈥檚 faces used to require expensive, dedicated scanners, but now you can do it with a smartphone.
Simon Lucey and his colleagues at Carnegie Mellon University in Pennsylvania have developed a program that generates an accurate 3D model of a person鈥檚 face and neck.
All the software requires is a 15 to 20-second video clip of a person鈥檚 face taken from various angles using a smartphone on a 鈥渟low motion鈥� setting, which captures footage at a rate of 120 frames per second or higher.
Advertisement
The program uses artificial intelligence to identify key features during the video, such as the silhouette of a person鈥檚 face in profile and the edges of the mouth, nose and eyes.
Using these features as constraints, the program then builds a geometric 3D model of the face and matches it to the user鈥檚 key facial features from different perspectives.
Given a video clip, the system takes between 30 and 40 minutes to generate a result.
To test the accuracy of the program, the team filmed 10 people and also scanned their faces using a dedicated 3D scanner. They compared the video-based models with the 3D scans and found that the reconstructions were mostly accurate to within 1 millimetre.
The system is more accurate than programs that rely solely on AI without any geometric knowledge, says Lucey, because AIs require training on thousands of 3D facial scans to yield precise results.
The team is now in discussions to develop the technology commercially for gaming.
鈥淚f you want to have virtual interactions with people, this is a great way without having a tens-of-thousands-of-dollars face scanner to get a very accurate face scan of yourself,鈥� says Lucey.
The team also hopes to use the software for objects in general.
The idea is that a person could 鈥渟can an object, and then 鈥� like Star Trek type stuff 鈥� transmit it to someone else, so they can hold their smartphone up and see your object where they are,鈥� says Lucey.
Reference: arxiv.org/abs/2003.08583
Want to get a newsletter on all the latest advances in AI? Register your interest and you鈥檒l be one of the first to receive it when it launches.
