快猫短视频

Do computers know what we are thinking? To find out, researchers at the University of California, Berkeley, have created a set of gruelling tests that probe AI鈥檚 progress in understanding the world. No AIs have passed the tests yet, but one got extremely close.

The tests examine theory of mind 鈥� the ability to reason about another鈥檚 beliefs 鈥� and are inspired by classic experiments in psychology. Each test consists of a short paragraph describing a scenario involving people and a few questions for the AI to answer about it.

The questions revolved around identifying first and second order beliefs. A first order belief is simply what someone thinks, e.g. answering a question like 鈥渨here does Sally think the marble is?鈥�. A second order belief is what someone thinks someone else thinks, e.g., 鈥渨here does Anne think Sally thinks the marble is?鈥�.

Children are normally able to correctly identify first order beliefs by around age 3, but it鈥檚 not until age 6 or 7 that they can do the same for second order beliefs.

AI mind test

Aida Nematzadeh, who led the work and is now at Google鈥檚 artificial intelligence lab at DeepMind, generated 10,000 scenarios and associated questions that tested theory of mind via first and second order beliefs. Her team then put them to four state-of-the-art AIs. None managed to achieve a passing score, which was set at 95 per cent. Most humans should be able to score 100 per cent.

The highest achiever was an AI called RelNet, produced by Adam Santoro and colleagues at DeepMind. It got a score of 94.3 per cent. The other three AIs tested managed scores ranging between 82 and 94 per cent.

However, the results were fairly fickle. Simply inserting an unrelated sentence with no bearing on the tested situations was enough to bamboozle the AIs. All of them dropped their scores by between 5 and 20 per cent, suggesting the AIs were not properly grasping the meaning of the text.

Though no AI has yet to comprehensively display theory of mind, they are improving at an impressive rate. Only two years ago, Facebook produced a series of tests for AI that would examine its ability to answer questions. None could handle the task at the time, but the best AIs can now pass this with few mistakes.

A machine with theory of mind would be able to reason about how its actions might impact people around it, says Alan Wagner at Georgia Tech Research Institute. 鈥淔or example, an AI agent teaching assistant might be able to reason about a student鈥檚 false beliefs related to a course (i.e. that the final exam is next week when in fact it is tomorrow) and in doing so be better suited to help,鈥� he says.

Another example is in the case of self-driving cars, says Johannes Bjerva at the University of Copenhagen. An AI-driven car with theory of mind might be able to realise that the driver of a different car has not spotted a pedestrian in the road, and honk its horn to warn them, he says.

However, even if an AI can pass these tests, it may still not have true theory of mind, says Wagner. 鈥淏ut it is important to start somewhere,鈥� he says.

Reference: