Artificial intelligence is growing ever more capable at increasingly complex tasks, but requires vast amounts of computing power to develop. A more efficient technique could save up to half the time, energy and computer power needed to train an AI model.
Deep learning models are typically composed of a huge grid of artificial neurons linked by 鈥渨eights鈥� 鈥� computer code that takes an input and passes on a changed output 鈥� that represent the synapses linking real neurons. By tinkering with these weights over thousands or millions of trials, it is possible to gradually train the model to carry out a task, such as identifying a person from a picture of their face or digitising text from an image of handwriting.
To train a model, researchers go through an iterative process of passing data in, assessing the quality of the output and then calculating a gradient that informs how the weights should be altered to improve performance. This process involves passing data from one side of the neural network to the other, via every link in the chain of artificial neurons, and then working backwards again to the beginning in order to calculate the gradient.
Advertisement
at the University of Oxford and his colleagues have now taken this two-stage process known as back-propagation and reduced it to just one, where an approximation of the gradient close enough to be effective is calculated during the first pass, making the second redundant. In theory, it could slash the time needed to train AI models in half. The team ran numerous tests with back-propagation and their new approach, each for the same number of iterations, and found that the performance of the AI was comparable.
at the University of Exeter, UK, says that calculating the gradient in the forward pass is 鈥渁 simple mathematical trick鈥� but seems to have the potential to solve one of the largest problems facing AI researchers, which is the increasingly high demands of computation.
鈥淵ou could potentially have a very cheap system to run training [with this approach],鈥� he says. 鈥淚t鈥檚 a very, very important thing to solve, because it鈥檚 the bottleneck of machine learning algorithms.鈥�
Cutting-edge AI research currently relies on vast models with hundreds of billions of parameters. Training these models can occupy huge supercomputers for weeks or months at a time. One of the largest neural networks currently operating, the Megatron-Turing Natural Language Generation model, has 530 billion parameters and was trained on Nvidia鈥檚 Selene supercomputer, which is made up of 560 powerful servers and 4480 high-end graphics cards, each costing thousands of pounds when bought commercially. Despite the huge power of that machine, it still took more than a month to train the model.
G眉ne艧 Baydin says that the best-case scenario is that this new approach slashes the time taken to train AI models in half, but that is far from guaranteed. He says that time will tell what results other researchers see when it is tested across a range of models. It may prove more efficient in some applications than others.
鈥淵ou can run one iteration of optimisation faster with our algorithm, but it doesn鈥檛 automatically mean you can get the same result twice as fast, because there are other things involved,鈥� he says. 鈥淚t might do a worse job than the back-propagation algorithm in some cases, and it might need more iterations to achieve the same quality of training. And if that happens, maybe it can end up like losing all your competition advantage.鈥�
Reference:
