Microsoft researchers have claimed that OpenAI’s GPT-4 is capable of such a wide variety of tasks, in some cases exceeding human ability, that the artificial intelligence model is showing “sparks” of artificial general intelligence (AGI). Is this long-awaited goal near or are these claims just hype?
While no standard definition of AGI exists, it is most commonly understood to refer to an AI that can understand and learn any intellectual task that humans are capable of.
In a new paper, a team of researchers from Microsoft Research say that they consider AGI to involve the ability to reason, plan, solve problems, think abstractly, comprehend complex ideas and learn from experience – and that GPT-4 meets that criteria, despite only being designed to generate text.
Advertisement
“Given the breadth and depth of GPT-4’s capabilities, we believe that it could reasonably be viewed as an early (yet still incomplete) version of an artificial general intelligence (AGI) system,” they say.
GPT-4 launched to the public earlier this month, but the Microsoft team devised a series of tests to evaluate a production version of the model before release. The researchers say the tests are based on ones from the field of psychology, which they admit are “somewhat subjective and informal” and may not stand up to scientific scrutiny.
Other researchers previously designed a suite of tests called BIG-bench, which is designed as a more rigorous assessment of an AI’s abilities. Using a subset of these tests, the score for OpenAI’s earlier AI model, GPT-3, was about half that achieved by people on average.
Microsoft’s new approach involved a variety of tasks. In one, the team asked GPT-4 to create a mathematical proof that there are infinitely many prime numbers, in rhyme. It correctly did so, in rhyming couplets. In other tests, GPT-4 scored around 80 per cent on the multiple choice components of the US Medical Licensing Exam, and 70 per cent on the Multistate Bar Exam, taken by lawyers in the US. When taking a software engineering test given to job candidates at Amazon, it scored 100 per cent, using less than 4 minutes of the allotted 2 hours.
It is hard to draw conclusions on the performance of GPT-4 from these tests alone. One unknown is whether the answers to these tests were already in its training data. A standard approach to creating and then testing an AI model is to use a large portion of available data to train it, but hold back a certain percentage in order to eventually test it on information it has never seen before. Critics have pointed out that OpenAI has provided minimal detail on what data was used to train GPT-4, making a truly independent analysis of its abilities difficult.
OpenAI itself has said that GPT-4 delivers “human-level performance” on a wide range of exams and tests, but also warns that it is still prone to “hallucination” – the phenomenon where an AI will produce convincing statements in response to prompts that are actually inaccurate or totally false. Indeed, Microsoft’s early attempts to have resulted in .
The paper’s authors weren’t available for interview before publication, but a Microsoft spokesperson says that the company – which owns a significant stake in OpenAI – is “not focused on trying to achieve AGI”.
“Our development of AI is centered on amplifying, augmenting and assisting human productivity and capability. We are creating platforms and tools that, rather than acting as a substitute for human effort, can help humans with cognitive work,” says the spokesperson.
at Imperial College London says that the inner workings of GPT-4 are very different to the human brain, but that the results in the paper show that each can achieve intelligence in their own way – even if AI is, at its core, learning to mimic what humans have done in the past.
“We could define AGI as being able to solve problems using language and it is doing as well as an average human,” he says. “In terms of language-based reasoning, I can’t think of many things that [GPT-4] would do worse than the average human on. You can have apparent intelligence without being intelligent in the same way as humans.”
But some experts doubt that true AGI will ever arrive or that the admittedly impressive abilities revealed in the paper are signs of true intelligence. at the Alan Turing Institute in London says we will see AI become increasingly capable and appear ever more intelligent, but there will never be true understanding or real intentionality.
“There are lots of people who really, really believe that AGI is coming. I don’t share that view,” she says. “AI will always be programs that do what they’re programmed to do. What they’re programmed to do is to mimic human language or outputs, and they’re getting increasingly good at it. So it becomes increasingly convincing.”
arXiv
