An artificial intelligence model can be made to spout gibberish if a single one of the many billions of numbers that compose it is altered.
Large language models (LLMs) like the one behind OpenAI’s ChatGPT contain billions of parameters or weights, which are the numerical values used to represent each “neuron” of their neural network. These are what get tuned and tweaked during training so the AI can learn abilities such as generating text. Input is passed through these weights, which determine the most statistically likely output.
The increasing ability of AI models in recent years has been heavily tied to increasing scale: adding more parameters and using more data to train the AIs has boosted performance. But larger models need more computational power to train and run. So a process called pruning is increasingly carried out to assess which neurons have the least effect on the output, and remove them.
Advertisement
Now, at Apple and his colleagues have found that although many neurons have a minimal effect and can be removed, some, termed “super weights”, have a disproportionate influence and are essential. In fact, pruning a single parameter among billions can destroy an LLM’s ability to generate text.
In experiments on Meta’s Llama-7B model, which has 7 billion parameters, pruning a specific super weight completely destroys the model’s ability to generate text, “resulting in qualitatively and quantitatively gibberish responses”, the researchers write.
Conversely, pruning 7000 other weights found to have the greatest impact on the output – but keeping the one with the most impact of all – affects the ability of the AI model to correctly predict the next word in portions of text by no more than a few percentage points. In essence, the super weight was more important to performance than the next 7000 most important parameters combined.
at Imperial College London says it makes sense that a single weight could have a large impact on output, given that its impact could be carried through the model – effectively copying and compounding errors. “The residual connections copy the early network layer’s outputs forward,” says Li. “If the original early layer’s outputs are big, and you destroy them, then the entire model is messed up.”
, co-founder of , a machine learning platform and community, says evidence of single neurons in large models having a disproportionate impact has emerged before. He highlights an OpenAI LLM created in 2017 that was trained on 82 million Amazon reviews and taught to determine the sentiment – positive or negative – of a piece of text. The model had a single “”, the value of which very accurately determined the answer. “It can be seen as a bit counterintuitive,” says Wolf.
The findings shed light on the often mysterious inner workings of neural networks and how we can influence them. at University College London says that one solution to this over-reliance on a single weight could be a training technique called dropout. In this, random weights in a model are set to zero – effectively deleting them – and the network is forced to learn other ways to recognise and respond to its inputs.
Dropouts were used to improve generality of AI models by stopping them becoming overly reliant on a certain path through the neural network. The technique has fallen out of favour as models have grown so large and absorbed so much data that they effectively solved the problem themselves, but returning to using dropouts could prevent a change to a super weight being “completely surprising to all other weights”, says Kusner.
arXiv
