A set of puzzles that will challenge even today’s most sophisticated artificial intelligence models, while being relatively easy for people, aims to encourage AI developers to create new techniques. Any AI that solves the puzzles will net its creators a share of a $1 million prize fund.
Companies like OpenAI already claim that their AI models, like GPT-4, exhibit “human-level performance” on real-world tests, such as university admission exams or the bar exam for lawyers. But this isn’t because the models are reasoning intelligently like humans, says at software company Zapier, but because they have effectively memorised the answers from the vast amount of internet data they have been trained on.
“I don’t mean that they literally memorise, they’re able to memorise some form of general patterns and apply those into adjacent contexts,” says Knoop. “But they’re always still going to be limited by the architecture that they’re built on — they can’t generalise to new novel situations.”
Advertisement
Knoop and François Chollet at Google have now announced a $1 million prize fund for any AI that can perform at a human-level or better at the (ARC), a test designed by Chollet in 2019 to be resistant to the memorisation that AIs are good at and to require the sort of general, basic intelligence that is natural to humans.
The test consists of paired grids of pixelated shapes linked by a pattern, such as drawing a path between two objects in a certain colour, or moving the pixels of an object in a consistent way. To correctly answer a question, you have to find the pattern by studying example pairs and then use that information to complete the second half of a new pair (see the image above for an example).
This requires only a small set of reasoning capabilities, says Knoop, such as object permanence, goal-directedness, counting and basic geometry. These skills are something even young children display, but are often lacking in large language models (LLMs) such as GPT-4.
If someone wins the $500,000 grand prize, which requires achieving a score of 85 per cent, or 1 percentage point more than the average human, then they will have designed an AI system that is much more capable than today’s models, says Knoop. He hopes such a model will also avoid problems plaguing AI systems today, such as the tendency for LLMs to simply make things up.
Researchers hoping to design an AI capable of beating the test can practise on a publicly available dataset, but to win the prize they will have to beat an offline ARC test, which is kept private to prevent LLMs memorising the answers.
“It is testing something very important, which is the ability of systems to deal with novelty,” says at Oregon State University. “When you look at large language models, they’re basically memorising, and maybe generalising from all this historical material they’ve read. But that doesn’t necessarily prepare them for novelty.”
However, it isn’t clear whether there might be ways to solve ARC challenges using techniques that don’t require the capabilities that Knoop and Chollet hope to test for, says Dietterich, such as finding a way to generate all the possible instances of the ARC problems through computational brute force and memorising the answers.
Knoop acknowledges that this is a risk, but says that one of the stipulations for winning the prize is that the tasks must be solved “efficiently”, which means that teams get a maximum of 12 hours and only a limited amount of computing power to solve the private dataset.
