Will artificial intelligences ever be able to match humans in abstract thought, or are they just very fancy number crunchers? Researchers at Google DeepMind are trying to find out by challenging AIs to solve abstract reasoning puzzles similar to those found in IQ tests.
If you have ever taken an IQ test, you’ll know that one kind of question involves looking at sets of abstract shapes and choosing which should come next in a given the sequence. These puzzles are known as Raven’s progressive matrices and can only be solved by figuring out the rules behind them.
The test is a useful way of measuring someone’s capacity for abstract reasoning – the ability to connect different concepts into new ideas. “Abstract reasoning is important in domains such as scientific discovery where we need to generate novel hypotheses and then use these hypotheses to solve problems,” says David Barrett at DeepMind.
Advertisement
Learning to think
Humans can cheat at the test by over-preparing, allowing them to learn rules that bypass the need for abstract thought, and with AI, this issue is exacerbated. Neural networks, a popular system behind AIs, can only learn by training on huge numbers of examples, allowing them to swot-up on the test without ever showing a capacity for abstract reasoning.
To get around this, Barrett’s team tested their neural networks on whether they could learn more general concepts, such as “increasing”. Since this can apply to a variety of properties, such as the number of shapes or the intensity of their colour, they could train the AIs on puzzles involving one type, but then test them on another.
They tested a range of standard neural networks and found they performed extremely poorly on this test – achieving marks as low as 22 per cent. But the team’s new neural network, which was specifically designed to infer relationships between different parts of the puzzle, did much better – achieving a score of 63 per cent.
“While these structures help specifically with this task, we believe they can also be applied to other problems involving abstract relationships and taking decisions between possible courses of action,” says team member Felix Hill.
The team are quick to point out that the puzzles they used in these tests aren’t actually the same as those found on human IQ tests. “It is important to note that the goal of this work is not to develop a neural network that can pass an IQ test,” says Barrett.
Unsurprisingly, all of the AIs performed best when the test puzzles were more similar to the training puzzles. Barrett says in future, he hopes that neural networks will be able to generalise more, increasing their capacity for abstract reasoning.
Journal Reference:
