快猫短视频

More than 400 AI researchers, including several from Google, have proposed an update to the famous Turing test to allow us to rate the capabilities of AI technology.

AI language models have become surprisingly, and often shockingly, good at conversing with humans in recent years due to ever larger scale: more computing power and vast sets of training data. A Google engineer was recently so impressed by one model that he declared 鈥� albeit to much scepticism 鈥� that it had actually become sentient. Researchers expect the scale of these models to continue to grow and reveal new abilities in coming years.

To assess current technology, and prepare for yet more capable models in the future, a group of has spent two years creating a replacement for the Turing test, which was developed by British mathematician Alan Turing in 1950, that can assess esoteric abilities and expand as needed. The Beyond the Imitation Game benchmark (BIG-bench) consists of 204 diverse tasks covering a range of topics 鈥� including linguistics, mathematics and chess 鈥� which are designed to not be fully solvable by the current state-of-the-art models.

In their paper introducing the BIG-bench, the researchers say that the tool will be essential to inform future research and to identify and plan for any disruptive new abilities or potentially harmful effects of emerging AI.

Expert humans took the same 204 tasks to establish an average and peak baseline score for each one. The researchers found that although the results achieved by AI models rose as they gained more computing power, AIs still performed poorly on most tasks compared with humans.

The researchers also found that the scale of AI models brought no improvement on certain tasks such as logical reasoning about long pieces of input text, giving clues as to areas of intelligence that ever-larger resources alone won鈥檛 solve. In fact, the research suggests that scale can even bring problems, as tests measuring social biases gave lower scores for larger models in some cases.

The paper examines the scores of models of varying scale based on OpenAI鈥檚 GPT AI, but it doesn鈥檛 compare other models, such as Google鈥檚 LaMDA model, which haven鈥檛 been publicly released. The core team of researchers on the paper included Google staff, but the company didn鈥檛 respond to a request for interview.

at the University of Surrey, UK, says the Turing test isn鈥檛 necessarily outdated, but it also isn鈥檛 a valid or broad enough test for modern AI. The , although it is debatable whether it really did. Hilton is sceptical, too, that the new benchmark can measure true intelligence.

鈥淚 think it鈥檚 a valid test, but I don鈥檛 think inferring from that that the machine is intelligent or sentient is quite the same thing,鈥� he says. 鈥淚 can see that setting up a set of benchmarks is one way of comparing one machine-learning algorithm with another, one AI with another. But I don鈥檛 think that necessarily answers the question of intelligence though.鈥�

鈥淲ith machine-learning techniques, you鈥檙e able to compose pieces of music or even write or respond to questioning, write prose, in quite a convincing and quite a human-like way. But is that really intelligence? I would say it鈥檚 not,鈥� he says.

Reference:听