The artificial intelligence model behind the ChatGPT chatbot can solve tasks used to test whether people can understand different perspectives, a key sign of intelligence known as theory of mind. Its ability – which seems to have spontaneously emerged rather than being something the AI was trained to do – is comparable to that of a 9-year-old child. However, whether this shows that the AI is using theory of mind or is finding other ways to pass the tests isn’t known.
“What [it] is doing is demonstrating a young child’s capacity to pass some of these benchmark tasks, and that’s not trivial,” says at the University of Birmingham, UK, who wasn’t involved in the work.
Theory of mind, or the ability to think about what other people are thinking, is considered a cornerstone of human intelligence and is essential for social interactions.
Advertisement
Psychologists have developed tests to see when this ability arises in young children and whether it exists in animals, but it is difficult to measure and there is much debate about what kind of tests should be used.
One of the most common is a false-belief task, which consists of a story in which a person thinks they know a fact about the world that, unknown to them, is incorrect. To pass the test, you must use a theory of mind to say what that person believes about the world, not what is true.
Now, at Stanford University in California and his colleagues have given two kinds of false-belief tasks to ChatGPT’s language model to solve. This model was developed by artificial intelligence firm OpenAI.
The first kind consists of a story in which someone discovers that a container’s contents are incorrectly labelled, such as a bag of popcorn described as chocolate. To pass the test, you must answer what someone coming across the bag thinks is in it, rather than its actual contents.
The second kind involves someone leaving and returning to a scene. During their absence, something has changed, such as a girl moving a cat from a box to a basket while her father is out of the room.
Kosinski and his team came up with 20 unique tasks for each puzzle type to ensure the model hadn’t seen them before in its training data, and ran each task thousands of times in different formats.
The language model, GPT-3.5, passed 100 per cent of the unexpected label tasks and about 80 per cent of the changing scenario tasks, which is similar in capability to a 9-year-old child. “According to the literature, these are gold standard tasks that are believed to require theory of mind to be solved,” says Kosinski.
Based on the criteria psychologists use to evaluate this ability in children, the AI appears to also have theory of mind, says Kosinski. Even if it is using other methods to pass the tests, it is no less interesting, he says, and should lead to a re-evaluation of how to test for this crucial form of intelligence.
The AI wasn’t trained to pass these tests, so the ability to solve them seems to have emerged, unlike with a DeepMind AI called Theory of Mind-net, which was specifically developed and trained to achieve this capability and reached the testing level of a 4-year-old.
The results of the work are interesting, but being able to pass theory of mind tests isn’t that useful on its own, says Apperly. “Just because you pass some of these tasks, it doesn’t mean that you can use your ability to think about other people’s perspectives, beliefs, desires or intentions for a wide set of useful circumstances.”
False-belief tasks can be a useful starting point for seeing how AI models might use theory of mind, but they can’t fully capture the complexity of the real world, says Apperly. The tasks used in the study, while common in psychology, only differentiate between two kinds of beliefs, real ones and false ones. However, in the real world, people can take on a whole range of different beliefs, and testing to see if someone or something comprehends this is difficult.
Investigating whether the model behind ChatGPT can solve more nuanced puzzles that test for theory of mind in older people might tell us more, but these are hard to devise and not widely agreed on by psychologists, says Apperly.
It is possible that theory of mind could be an emergent property of these systems, but it is important to note that the main mechanism of the language model behind ChatGPT is understanding the relationship between words, says at the Alan Turing Institute in London.
“It’s demonstrating an understanding of the association between words and it’s making those connections to make a correct prediction of what the outcome might be, but I think from that to make claims of any understanding of emotional reason, or intention, or understanding of human cognitive or emotional states is a bit of a stretch,” she says.
OpenAI didn’t respond to a request for comment.
Reference:
