Large artificial intelligence language models, like those used to run the popular ChatGPT chatbot, can be reduced in size by more than half without losing much accuracy. This could save large amounts of energy and allow people to run the models at home, rather than in huge data centres.
Many recent advances in artificial intelligence models have come from scaling up the number of parameters: the values that each model tunes to produce outputs. OpenAI鈥檚 GPT-3, a version of which powers ChatGPT, has 175 billion parameters.
These parameters adopt certain values corresponding to the data used to train the model, and large numbers of parameters increase the computer power and energy the model requires.
Advertisement
Now, and Elias Frantar at the Institute of Science and Technology in Austria have developed a method that removes 60 per cent of these parameters with a minimal drop in accuracy.
鈥淚 don鈥檛 think you need 175 billion parameters for anything, that sounds like a huge amount,鈥� says Alistarh. 鈥淚鈥檓 hoping that we can slim that down a bit.鈥�
The method uses an algorithm to pass a small amount of data into the model to give a sample output, and then turn off parameters in the neural network that don鈥檛 appear to significantly affect the output鈥檚 result. However, this introduces error to the overall system, so the algorithm also tweaks the remaining parameters to better fit the output.
鈥淧runing鈥� large AI models in this way leads to only a minimal loss of accuracy, says Alistarh, and they can function as well as models with many fewer parameters. The pair assessed this by rating 鈥減erplexity鈥�, a measure of how well the results match predictions of what they will be.
One of the drawbacks of models with huge numbers of parameters is that they can only be used by people with enough computing power to train or run them themselves, such as academic institutions, says at Heriot-Watt University in Edinburgh, UK. If these pruning results can be replicated generally, they could democratise access, she says.
Reference:
