Artificial intelligence struggles to comprehend how objects interact with each other as they roll, collide, drop and drape, flunking a set of benchmark tests designed to see how intelligent it really is.
The so-called Physion benchmark was designed by at Stanford University in California and his colleagues. It uses eight scenarios to showcase physical phenomena most humans understand innately.
AIs were given the opening moments of the scenarios, computer-generated in 3D, featuring objects that were designed to interact with each other. The scenarios included a napkin being draped over objects on a table, a set of dominoes teetering towards collapse and a ball rolling down a slope. The tests were designed to probe how well computer code understood what it was 鈥渟eeing鈥� and how good it was at predicting what would happen next.
Advertisement
鈥淎lgorithms have gotten very good at seeing a scene and saying, 鈥楾his is a bottle, this is a car,鈥欌�� says Bear. 鈥淚 was interested in a very different type of behaviour: how well can an algorithm interact physically with scenes.鈥�
Some academics believe that having a firm grasp of what the objects on view are leads automatically to a deeper understanding of how the world works and how objects interact. This would suggest that once an AI gains a semantic understanding of the world, it would naturally gain a physical understanding of the world too. 鈥淔or various reasons, I was suspicious of that,鈥� says Bear.
Those suspicions appear well founded. Bear and his colleagues asked both humans and AIs to watch a 1.5-second snippet of a scene and then predict what would happen next. The tests 鈥� which were hard enough that 25 per cent of the guesses human participants made were wrong 鈥� proved too difficult for most of the AIs.
The worst AI predictions included that an object would dissolve or that it might pass through another object without any effect. Some AIs predicted that an object would disappear entirely.
鈥淭hat, to me, is a very worrying failure,鈥� says Bear. 鈥淸Some AIs] don鈥檛 think that objects are things that continue to exist beyond the moment you鈥檙e looking at them.鈥�
The most successful AIs were so-called graph neural networks, which broke down each scenario 鈥� and the objects it contained 鈥� into individual 鈥減articles鈥� and modelled how they would interact. However, Bear points out that by dint of breaking down objects into particles, those AIs are 鈥渁lmost cheating鈥� by adding extra data to a scene that other AIs aren鈥檛 privy to.
鈥淭his project represents a significant leap forward for researchers in mobile robotics, autonomous navigation and computer vision in general,鈥� says at the University of Copenhagen, Denmark.
The field of AI is moving from a data set-driven model to one based on simulation, he adds. 鈥淚n the latter setting, we must work to close the simulation-to-real gap, which will allow us to train increasingly predictive models with dramatically decreased human labelling effort.鈥�
That is the goal that Bear, who is a biologist by training, is aiming for. 鈥淚 want to help design algorithms that are able to perceive the world more like people do, including understanding that an object is a physical thing that鈥檚 not just going to dissolve,鈥� he says.
Reference:
