Mind-reading software developed in the Netherlands can decipher the sounds being spoken to a person, and even who is saying them, from scans of the listener鈥檚 brain.
To train the software, neuroscientists used functional magnetic resonance imaging () to track the brain activity of 7 people while they listened to three different speakers saying simple vowel sounds.
The team found that each speaker and each sound created a distinctive 鈥渘eural fingerprint鈥� in a listener鈥檚 auditory cortex, the brain region that deals with hearing.
Advertisement
This fingerprint was used to create rules that could decode future activity and determine both who is being listened to, and what they are saying.
Tell-all brain
鈥淲e have [created] a sort of speech-recognition device which is completely based on the brain activity of the listener,鈥� explained , who led the group.
The team hopes to match recent advances in using fMRI to identify what a person is looking at from their brain activity. Until now, the best mind-reading feats achieved for auditory brain activity extended only to differentiating between different categories of sounds, such as human voices versus animal cries.
鈥淭his is the first study in which we can really distinguish two human voices, or two specific sounds,鈥� Formisano told 快猫短视频.
Hello, I鈥檓 listening
In the course of making that possible, the team made new discoveries about how the brain processes speech.
They found that, whatever sound a person makes, they trigger the same voice fingerprint in the brain. Likewise, a given vowel sound produces the same fingerprint, independent of the speaker
This makes it possible to have software recognise combinations of speakers and sounds that it has not encountered before. It should be possible to teach the system how to recognise all the component sounds of speech, and then recognise full words, Formisano says.
鈥淰owel sounds are not meaningful, but they are language,鈥� he says. 鈥淭hese are the building blocks of language.鈥�
Noisy world
The group are now working to make the system robust enough to recognise more complex sounds without training, and in a noisy environment.
鈥淚t鈥檚 interesting to see fMRI techniques extended to the auditory cortex,鈥� said of the University of California, Berkeley, who studies the brain鈥檚 visual systems using similar methods and earlier this year created a system that decodes brain activity to reveal what a person is looking at.
The fact that the Netherlands team has shown that the brain deals with the categorisation of speakers and sounds differently is particularly significant, he says.
Office helper
The new approach could help improve voice recognition software, Formisano says, although probably not by leading to a commercially-available system to transcribe conversations from people鈥檚 brains.
Instead, the existing mind-reading software will make it possible to break down the human brain鈥檚 impressive ability to process sound and voices, even in the face of distracting background noise. Copying those tricks would provide a new way to make voice recognition software much more effective.
Journal reference: ()
