快猫短视频

A new device that restores a form of sight to the blind is turning our understanding of the senses upside down

CLAIRE CHESKIN used to live in a murky world of grey, her damaged eyes only seeing large objects if they were right next to her. She could detect the outlines of people but not their expressions, and could just about make out the silhouettes of buildings, but no details. Looking into the distance? Forget it.

Nowadays things are looking distinctly brighter for Cheskin. Using a device called vOICe, which translates visual images into 鈥渟oundscapes鈥�, she has trained her brain to 鈥渟ee through her ears鈥�. When travelling, the device helps her identify points of interest; at home she uses it to find things she has put down, like coffee cups. 鈥淚鈥檝e sailed across the English Channel and across the North Sea, sometimes using the vOICe to spot landmarks,鈥� she says. 鈥淭he lights on the land were faint but the vOICe could pick them up.鈥�

As if the signposting of objects wasn鈥檛 impressive and useful enough, some long-term users of the device like Cheskin eventually report complete images somewhat akin to normal sight, thanks to a long-term rewiring of their brains. Sometimes these changes are so profound that it alters their perceptions even when they aren鈥檛 using the device. As such, the vOICe (the 鈥淥IC鈥� standing for 鈥淥h, I See鈥�) is now proving invaluable as a research tool, providing insights into the brain鈥檚 mind-boggling capacity for adaptation.

The idea of hijacking another sense to replace lost vision has a long history. One of the first 鈥渟ensory substitution鈥� devices was developed in 1969 by neuroscientist . He rigged up a television camera to a dentist鈥檚 chair, on which was a 20-by-20 array of stimulators that translated images into tactile signals by vibrating against the participant鈥檚 back. Despite the crudeness of the set-up, it allowed blind participants to detect the presence of horizontal, vertical and diagonal lines, while skilled users could even associate the physical sensations with faces and common objects.

By the time he died in 2006, Bach-y-Rita had developed more sophisticated devices which translated the camera鈥檚 images into electrical pulses delivered by a postage-stamp-sized array of electrodes sitting on the tongue. Users found, after some practice, that these pulses gave them a sense of depth and 鈥渙penness鈥�, a feeling that there was 鈥渟omething out there鈥� (快猫短视频, 29 July 2005, p 40).

This vague feeling of space, which we experience as part of normal sight, suggests the brain may be handling the information as if it had originated from the eyes. Would it be possible to get even closer to normal vision- perhaps even producing vivid and detailed images- by feeding in information using something other than tactile stimulation? To find out, physicist and inventor , based in Eindhoven, the Netherlands, turned to hearing. The ears do not detect as much information as the eyes, but their capacity is nevertheless much greater than the skin鈥檚.

Meijer thought up the vOICe in 1982, though it took until 1991 for him to design and build a desktop prototype that would translate video into audio. By 1998 he had developed a portable, if still bulky, version using a webcam, notebook PC and stereo headphones, which allowed users to experiment with the device in daily life. The device is now more discreet, consisting of 鈥渟py鈥� sunglasses which conceal a tiny camera connected to a netbook PC, and a pair of headphones. Alternatively, some users download the software to their smartphone, and its built-in camera acts as their eyes.

Every second the camera scans a scene from left to right. Software then converts the images into soundscapes transmitted to the headphones at a rate of roughly one per second (see diagram). Visual information from objects to the wearer鈥檚 left and right are fed into the left and right ear respectively. Bright objects are louder, and frequency denotes whether an object is high up or low down in the visual field.

At first the soundscapes are reminiscent of the whirring, bleeping and hooting sound effects that would accompany an alien melting the brain of a human in a 1960s science-fiction movie. But by feeling the objects first, to learn to associate the accompanying sounds with their shapes, and by discovering how the soundscape of an object varies as the user moves, the experience becomes particularly 鈥渧ision-like鈥�.

Pat Fletcher of Buffalo, New York, lost her sight at the age of 21 and had just a pinpoint of perception in her left eye, through which she could sometimes see red or green, before she started using the vOICe system in 2000. In the early stages, the pictures in her mind鈥檚 eye were like 鈥渓ine drawings鈥� and 鈥渟imple holographic images鈥�, but after a decade of practice, she now sees complete images with depth and texture. 鈥淚t is like looking at an old black-and-white movie from the early 30s or 40s. I can see the tree from top to bottom, and the cracked sidewalk that runs alongside the tree,鈥� she says.

鈥淚t鈥檚 like looking at a black-and-white movie from the 40s. I can see the tree from top to bottom, and the cracked sidewalk鈥�

鈥淲hat鈥檚 exciting to me,鈥� says , a cognitive psychologist at Queen Mary, University of London, who has been using the vOICe for his own research, 鈥渋s that not only can you use this device in a very deliberate fashion where you can think, 鈥榦kay, this sound corresponds with this object鈥�, but it is also possible, through extensive use, to go beyond that and actually have some sort of direct, qualitative experience that is similar to the vision they used to experience.鈥�

The US is now funding the first controlled study to look at the benefits of the vOICe system while trying to find the optimal training protocol. 鈥淪ome of the participants in the current trial have learned more in months than [Fletcher] learned in years of using the vOICe,鈥� says Meijer. The study, which will involve around 10 participants, may even answer the long-standing question of whether congenitally blind adults can benefit in the same way as Cheskin and Fletcher.

Intended to last about a year, the trial is being run by Luis Goncalves and Enrico Di Bernardo of , a company that tests sensory substitution devices. The first two participants are a 66-year-old who has been blind from birth but has slight light perception, and a 40-year-old who lost his sight due to diabetes. Twice a week they attend two-hour training sessions, including tasks such as finding a target in a large room and making their way around an obstacle course. 鈥淭hey are empowered by this,鈥� says Goncalves, adding that the 66-year-old 鈥渃an now go to a restaurant and seat himself without asking for assistance and is teaching his wife, who is also blind, how to use the vOICe鈥�.

Not everyone is quite so impressed. For example, , a psychologist at Descartes University in Paris, France, points out that the system needs time to scan an image and so lacks the immediacy of vision. 鈥淚 think it鈥檚 possible with resources and time to make something much better than the vOICe,鈥� he says.

Seeing ear to ear?

Nevertheless, vOICe is still of great interest to O鈥橰egan and other researchers, who want to know what these people are experiencing. Are they really seeing? And if so, how?

The traditional view is that the brain takes data from the different sensory organs- in the case of sight, the retina- and, for each sense, processes it in separate regions to create a picture of the outside world. But that cannot explain how someone can have a visual experience from purely auditory information.

As such, O鈥橰egan says our definition of what it means to see needs to change. Our senses, he argues, are defined by the way the incoming information changes as we interact with the environment. If the information obeys the laws of perspective as you move forward and backward, we will experience it as 鈥渟eeing鈥�- no matter how the information is being delivered. If you have a device that preserves these laws, then you should be able to see through your ears or your skin, he says.

If O鈥橰egan is on the right track, we will have to reconsider long-held ideas of how the brain is organised to deal with incoming information. Traditionally, the brain is considered to be highly modular, with the occipital, temporal and parietal cortices handling inputs from the eyes, ears and from the skin and deep tissues, respectively. According to O鈥橰egan, however, these regions may actually deal with certain types of information- shape or texture, for example- irrespective of which sense it comes from.

There is some evidence to support this view. In 2002, neuroscientist Amir Amedi, now at the , Israel, published research showing that a specific part of the occipital cortex was activated by touch as well as visual information. He named it the lateral occipital tactile-visual (LOtv) region. Amedi and colleagues hypothesised that the area lit up because the occipital cortex is oriented around particular tasks- in this case, 3D-object recognition- rather than a single sense ().

How does this tally with the vOICe experience? Amedi recently collaborated with , director of the Berenson-Allen Center for Noninvasive Brain Stimulation in Boston, Massachusetts, to find out whether the vOICe system activates the LOtv when users perceive objects through soundscapes. They asked 12 people, including Fletcher, to examine certain objects such as a seashell, a bottle and a rubber spider using touch and the vOICe system. They were then asked to recognise the same objects using only the soundscapes delivered by vOICe. For comparison, they were also asked to identify objects based on a characteristic sound, such as the jingling of a set of keys.

During the trials, fMRI brain scans showed that the LOtv region was active when expert users like Fletcher were decoding the vOICe soundscapes, but significantly less active when they just heard characteristic sounds. For those using the vOICe for the first time, the LOtv region remained inactive, again suggesting that this area is important for the recognition of 3D objects regardless of which sense produces the information ().

Further evidence that this region is vital for decoding soundscapes came two years later, in 2009, from a study using repetitive transcranial magnetic stimulation (rTMS) 鈥� short bursts of a magnetic field that temporarily shut down the LOtv of subjects, including Fletcher. 鈥淚t felt like someone tapping on the back of my head,鈥� she says. As the rTMS progressed, her vision with the vOICe deteriorated, and the 鈥渨orld started getting darker, like someone slowly turning down the lights鈥�.

鈥淎s a magnetic field disrupted her visual cortex, the subject鈥檚 vision with the vOICe device deteriorated. Her world got darker, like someone was slowly turning out the lights鈥�

When Fletcher attempted to use the vOICe after undergoing rTMS, the various test no longer made sense. 鈥淚t was total confusion in my brain鈥� I couldn鈥檛 see anything.鈥� The result was terrifying: 鈥淚 wanted to cry because I thought they broke my sight 鈥� it was like a hood over my head.鈥� The rTMS had a similar impact on other vOICe users ().

鈥淚t turns upside down the way we think about the brain,鈥� says Pascual-Leone. Most of us think of our eyes as being like cameras that capture whatever is in front of them and transmit it directly to the brain, he says. But perhaps the brain is just looking for certain kinds of information and will sift through the inputs to find the best match, regardless of which sense it comes from.

Reconfiguring the brain

The question remains of how the vOICe users鈥� brains reconfigured the LOtv region to deal with the new source of information. Amedi鈥檚 preliminary fMRI scans show that in the early stages of training with vOICe, the auditory cortex works hard to decode the soundscape, but after about 10 to 15 hours of training the information finds its way to the primary visual cortex, and then to the LOtv region, which becomes active. Around this time the individuals also become more adept at recognising objects with vOICe. 鈥淭he brain is doing a quick transition and using connections that are already there,鈥� says Amedi. With further practice, the brain probably builds new connections too, he adds.

Eventually, such neural changes may mean that everyday sounds spontaneously trigger visual sensations, as Cheskin has experienced for herself. 鈥淭he shape depends on the noise,鈥� she says. 鈥淭here was kind of a spiky shape this morning when my pet cockatiel was shrieking, and [the warning beeps of] a reversing lorry produce little rectangles.鈥� Only loud noises trigger the sensations and, intriguingly, she perceives the shape before the sound that sparked it.

This phenomenon can be considered a type of synaesthesia, in which one sensation automatically triggers another, unrelated feeling. Some individuals, for example, associate numbers or letters with a particular colour: 鈥淩鈥� may be seen as red while 鈥淧鈥� is yellow. For others, certain sounds trigger the perception of shapes and colours, much as Cheskin has experienced.

Most synaesthetes first report such experiences in early childhood, and it is very rare for an adult to spontaneously develop synaesthesia, says , a psychologist at the University of Sussex in Brighton, UK. He recently published a chronological log of Cheskin鈥檚 and Fletcher鈥檚 experiences, including the synaesthetic ones ().

This capacity to rewire our sensory processing may even boost the learning abilities of sighted users, suggests Pascual-Leone. It might be possible to extract supplementary information by feeding a lot of different sensory inputs to the same brain areas. Art connoisseurs could learn to associate the style of a master鈥檚 hand with a characteristic sound, and this may help them distinguish genuine work from a fake. Alternatively, it could compensate for low light levels by delivering visual information through our ears. 鈥淭hat鈥檚 science fiction. But it鈥檚 interesting science fiction,鈥� says Pascual-Leone.

For neuroscientists like Pascual-Leone and Amedi, the research is proof that the ability to learn as we grow old does not disappear. Pascual-Leone says the notion of a critical period during which the brain must be exposed to particular knowledge or never learn it appears 鈥渘ot universally true鈥�. 鈥淚t gives us a reason for hope,鈥� he says, 鈥渁nd implies that we should be able to help people adjust to sensory losses with this type of substitution. The capacity to recover function could be much greater than realised.鈥�