Getting immersed in a 3D-version of your favourite painting could be made possible thanks to an artificial intelligence that can transfer artistic styles onto 3D scenes.
AI has been used before to transfer an artistic style onto images that can be viewed from 360 degrees but the latest project is higher quality in showing the small details when digitally painting scenes in different styles. at Cornell University and his colleagues demonstrated their style transfer system with 3D scenes involving objects such as flowers, bulldozers and dinosaur skeletons, stylised using artworks such as Van Gogh鈥檚 The Starry Night and Edvard Munch鈥檚 The Scream.
Advertisement
The quality improvement came from the AI having the capability to directly compare details between the original image and the newly stylised 3D scenes. For comparison, an older AI technique loses many of those original details by converting image features into a more compact statistical set for analysis.
鈥淲e are trying to really capture the subtle, artistic style, like the strokes of The Starry Night painting,鈥� says Zhang. 鈥淭his is really important for human perception, because our human eyes are very sensitive to local details.鈥�
The team presented people with videos of five different art styles for five different 3D scenes using both the new and older AI techniques through an online survey. Each of the style transfer comparisons received 12 ratings on average, with people favouring the new AI technique more than 86 percent of the time compared to the most competitive older AI technique.
This style transfer technique could find uses in the animated film and game industry, where visual effects artists could use a style transfer as a starting template before making additional changes by hand.
The AI method for an artistic style transfer could also apply to photorealistic style transfers. For example, Zhang envisions changing a 3D view of the Statue of Liberty at noon to a 3D view of the Statue of Liberty at sunset.
But it鈥檚 still not the easiest process to produce these 3D scenes. Capturing a 360-degree viewable scene requires either multiple camera setups or the help of an off-the-shelf drone that can hover and capture views from multiple angles. Running the algorithm can take up to 20 minutes.
鈥淭his is a very time-consuming process, because the artists might want to play around with different styles during the trial-and-error process,鈥� says Zhang.
The hope is to improve the efficiency of this 3D content creation so that people can eventually use the application in real-time on their phones.
Reference: arXiv,
