An artificial intelligence can detect when someone is thinking about specific concepts such as food or sleep just by looking at brain scans. The system might one day be used to help people who have lost the ability to speak or to investigate mental health conditions.
When we receive a stimulus from the outside world, such as hearing spoken words or seeing an image, this is encoded in the brain as neural activity. Attempts to decode this activity to work out what words might have caused a specific neural signal have had mixed results, and the most successful tend to require invasive, surgically implanted electrodes.
Now, at the University of Texas at Austin and his colleagues have developed a machine learning-driven AI model that can work out word sequences that match, or closely resemble, the input stimulus from people鈥檚 brain activity and the meaning behind them.
Advertisement
First, Huth and his team recorded functional MRI (fMRI) data for three brain networks associated with language processing 鈥� the prefrontal network, the classical language network and the parietal-temporal-occipital association network 鈥� as a small group of people listened to 16 hours of narrated stories to train the model to understand and predict how someone鈥檚 brain would respond to a particular sequence of words.
The researchers then asked participants to listen to a new story so the model could try to decode the corresponding brain recording, and compared the words from the story with the decoded version.
While the precise words in each version differed, metrics comparing the meanings of passages found that the model鈥檚 results were significantly better than they would be by chance.
Huth and his colleagues also tested the decoder on people telling imagined stories and watching short silent films, and managed to extract similar-meaning words and sequences for both.
Here is one example of the original text of a story that was listened to: 鈥渢hat night i went upstairs to what had been our bedroom and not knowing what else to do i turned out the lights and lay down on the floor鈥�.
The AI translated the resulting brain patterns as: 鈥渨e got back to my dorm room i had no idea where my bed was i just assumed i would sleep on it but instead i lay down on the floor鈥�.
鈥淭he fact that the decoder can get the gist of the sentences is very impressive,鈥� says at the Massachusetts Institute of Technology. 鈥淲e can see, however, that it still has a long way to go. The model guesses bits and pieces of meaning and then tries to put them together, but the overall message typically gets lost 鈥� likely because the captured brain signals reflect what concepts a person is thinking about, such as 鈥榯alk鈥� or 鈥榝ood鈥� for example, but not how these concepts are related.鈥�
The model also seems better at predicting concrete words, such as food, than abstract terms, adds Ivanova.
There are two components to improving decoding models, says at the University of California, Berkeley: better brain recordings and more powerful computational models. While fMRI capability hasn鈥檛 progressed much in the past decade, computational power and language models have.
鈥淭his work dials the computational model up to 11,鈥� says Gallant. 鈥淭hey developed a fully modern, high-power, language neural network and used that as the basis for building the decoding model. This is really the innovation that is most responsible for producing such great results.鈥�
As well as potentially being able to help people who can鈥檛 speak to communicate, this advance in computational modelling could also be used to improve basic and clinical neuroscience, says Gallant, such as for investigating mental health conditions.
Recent work by Facebook-owner Meta has had some success in decoding which individual words people are thinking of from a set word list by using magnetoencephalography and electroencephalography scans, but seeking words rather than overall meaning is viewed as an easier problem by many in the field.
Although there are obvious privacy concerns about accessing someone鈥檚 thoughts, Huth and his team say that these aren鈥檛 currently problems because the model requires so much training data and cooperation. If someone in the fMRI scanner chooses to think about other things, such as counting, telling a story or naming things, it sabotages the decoder and it fails.
鈥淚f you didn鈥檛 listen to several hours of podcasts while lying in an MRI machine, Huth and his colleagues probably can鈥檛 decode your thoughts 鈥� at least not yet,鈥� says Ivanova.
Reference: bioRxiv, DOI:
