An artificial intelligence can predict the future as well as groups of people for events like political elections or economic trends.
People are notoriously bad at predicting the future, at least on an individual level. But websites called prediction markets, where people can bet on the outcome of future events, have demonstrated that the wisdom of crowds leads to better guesses. The average, crowdsourced predictions, which take into account many people’s forecasts, tend to be much more accurate than those of one person alone.
at the University of Berkeley, California, and his colleagues have developed an artificial intelligence to replicate this process. They found it can predict future events better than the average human and, in some cases, better than the crowd.
Advertisement
“[Forecasting] requires a human to sit down and really gather a lot of sources, figuring out which sources to trust and how to weigh all these things,” says Halawi. “A language model can just do this very quickly.”
To train the model, Halawi and his team started with an existing large language model, OpenAI’s GPT-4. They gave it additional training, a process called fine-tuning, using tens of thousands of accurate crowdsourced forecasts from prediction markets.
When the system is given a new prediction task, a separate language model breaks down the task into sub-questions and uses these to find relevant news articles. It reads these articles to answer its sub-questions and feeds that information into the fine-tuned GPT-4 to make a prediction.
Halawi and his team found that their system could predict events much better than individuals and almost as well as the crowdsourced answer, achieving an average accuracy of 71.5 per cent, compared with the human group’s 77 per cent, on a set of test questions. The system performed best, sometimes even outperforming the crowd, on “uncertain” questions, those that had a wide range of possible answers. But it struggled when predictions contained little uncertainty, such as forecasting whether or not the stock of a large, publicly traded company might go to 0 next year. This is because GPT-4 prefers to hedge its answers as a safety feature, says Halawi.
The system could be an additional useful data point for economic and political analysts that use different information sources to make decisions, says at Queen Mary University London, but it is unlikely to be better than many custom models and algorithms for scenarios like financial modelling. “For a discretionary asset manager, somebody who makes their own decisions by synthesising information, I think it’s an additional input, right? Those types of managers often believe in the wisdom of crowds,” says Saldanha.
“In the future, political decision-makers may consult the AIs on what actions would most likely bring about desired outcomes,” says at the Center for AI Safety in California. He also suggests that prediction-making models could address future dangers created by AI. “Forecasting bots would help us anticipate and steer clear of these risks,” says Hendrycks.
However, the model currently isn’t public, and running it is expensive – it costs about $1 per question asked, which is more expensive than most queries to AI chatbots. Halawi and his team are now working on making it more efficient and developing an open-source version, he says.
arXiv
