Software can analyse millions of static photos of city streets taken atop cars and construct a realistic 3D model that could be used to create immersive maps or even train driverless cars safely in a virtual environment.
Block-NeRF was created by a team of researchers at driverless car company and Google Research, which are both owned by Alphabet. The tool uses vast numbers of photos taken by cameras mounted atop Waymo鈥檚 autonomous cars and builds numerous small 3D models, each covering just over one city block. These models are then seamlessly stitched together to create a large navigable virtual world.
Advertisement
The researchers declined to be interviewed, but said in a paper that their work builds on a tool called Neural Radiance Fields (NeRFs), which constructs 3D models of small objects from a collection of still images so that they can be viewed from any angle 鈥� even those for which no photographs were taken. The technique uses an AI trained to produce accurate 3D models from large sets of images, complete with information on the exact location in which each image was taken.
The models are computationally intensive to create and run, and until now they have been limited in the size of the objects they can reproduce in virtual 3D. But Block-NeRF cleverly stitches together models of individual city blocks to keep the size of the overall model small enough to run on modest hardware. New blocks are recalled from memory as and when needed. The team claims that this is the largest NeRF model to date.
In a recreation of the Alamo Square neighbourhood in San Francisco, which is around 960 metres by 570 metres, the team used 35 Block-NeRFs. The whole model used 2.8 million images, taken over three months. And because those drives were made at different times of day and in different weather, the tool is able to change the virtual conditions at will from any location 鈥� making the scene night-time or daytime, cloudy or sunny.
But there are problems to overcome. The model spots variations between images taken at different times, such as parked cars, and filters them out, but shadows of these objects can remain in the final model. What鈥檚 more, because foliage moves in the wind, trees and bushes tend to be blurred.
at the University of Leeds, UK, says that although the work is based on existing NeRF techniques, it is impressive to see them scaled up to limitless sizes. There are obvious applications in training driverless cars and immersive online maps, despite the models not being as finely detailed as those produced by manual scanning with lasers, he says.
鈥淥n the city scale, it seems that the requirement on the detail is not as strong as on single small objects that have really detailed surface features,鈥� says Wang. 鈥淥n a building, nobody cares if it looks like a block. With a reasonable facade and size, then that鈥檚 fine.鈥�
鈥淭hey鈥檙e very good at creating digital replicas of cities at a very fast speed. But the flip side of that is currently I would have doubts in using those in virtual world applications where high fidelity is required,鈥� he says. 鈥淔or example, if you do virtual tourism and you look at these models, I think you won鈥檛 really have the full experience.鈥�
Reference:
