Microsoft and chip manufacturer Nvidia have created a vast artificial intelligence that can mimic human language more convincingly than ever before. But the cost and time involved in creating the neural network has called into question whether such AIs can continue to scale up.
The new neural network, known as the (MT-NLG) has 530 billion parameters, more than tripling the scale of OpenAI鈥檚 groundbreaking GPT-3 neural network that was considered the state of the art up until now. This progress required more than a month of supercomputer access and almost 4500 high-power and expensive graphics cards, which are commonly used to run high-end neural networks.
When OpenAI released GPT-3 last year it surprised researchers with its ability to generate fluent streams of text. It had used 175 billion parameters 鈥� allocated slots of data within a computer that replicate the synapses between neurons in the human brain 鈥� and consumed vast amounts of publicly accessible text from which to learn language patterns. Microsoft has since gone on to acquire an exclusive licence to use GPT-3.
Advertisement
Microsoft and Nvidia tested MT-NLG on a range of language tasks, such as predicting which word followed a section of text and extracting logical information from text, and found it had a greater ability than GPT-3 to complete sentences accurately and mimic common sense reasoning 鈥� but not by much, given the increase in scale. On one benchmark, where an AI is required to predict the last word of sentences, GPT-3 scored an accuracy of up to 86.4 per cent, while the new AI reached 87.2 per cent.
This improved ability doesn鈥檛 come cheap. 鈥淚t costs effectively millions of dollars to train one of these models鈥� as the computational resources needed to train it grow quickly as size increases, says聽聽at Nvidia.
MT-NLG was trained using Nvidia鈥檚 Selene supercomputer, which is made up of 560 powerful servers, each equipped with eight A100 80GB Tensor Core graphical processing units (GPUs). Each of those 4480 graphics cards 鈥� designed to run computer games, but also extremely capable at churning through vast amounts of data while training AIs 鈥� currently costs thousands of pounds when bought commercially. Although the entire might of the computer wasn鈥檛 used solely by this research team, it took over a month to train the AI.
Even running the neural network once it is trained requires 40 of those GPUs, and each query takes between 1 and 2 seconds to process. This constant stretching of scale means that AI research is now, to a certain extent, an engineering problem of efficiently splitting up the problem and distributing it over vast amounts of hardware.
Catanzaro says that scale has been the dominant force in machine learning for decades. 鈥淚t鈥檚 definitely true that better algorithms help, and it鈥檚 100 per cent true that more data and better data absolutely helps, but I think that computing scale absolutely has been the driving force in a lot of progress,鈥� he says.
Many researchers are reluctant to rely on scaling-up alone as they want a more elegant solution, says Catanzaro, but the results speak for themselves. Although the benchmark measurements reflect small improvements, there is thought to be significant steps up in the way the AIs reason and extract nuanced information, which perhaps isn鈥檛 captured by ageing benchmarks.
鈥淭here鈥檚 always this resistance like, 鈥榠t can鈥檛 be that easy, it can鈥檛 be that stupid that we just need to scale, because that isn鈥檛 very smart, it鈥檚 just brute force鈥�. But the sort of bitter lesson is that scale has actually yielded the most benefits in the space,鈥� he says.
at New York University says that current benchmarks for assessing quality of language processing AIs are nearing the end of their useful life and researchers are seeking new metrics that can be used to assess the quality of language and even reasoning, but that isn鈥檛 made simpler by the rapid rate of progress in AI. Those same researchers are also 鈥渘ervously waiting to find out鈥� if scale can continue to bring improvements or whether it will hit a ceiling, he says, as the cost of research in the field grows rapidly.
鈥淭hese are definitely some of the most expensive projects in the field, but whether they鈥檙e too expensive depends on what you see their potential as,鈥� he says. 鈥滻f you see these as steps to a pretty broadly-useful form of AI, and you see that as desirable, then it鈥檚 easy to imagine justifying vastly larger budgets.鈥�
