ERIK WEIHENMAYER lost his sight when he was 13. Twenty years on, in Paul Bach-y-Rita’s lab at the University of Wisconsin Medical School, he caught a rolling ball, played a game of rock, paper, scissors, walked through a doorway and watched a flickering candle flame. Nothing had changed with his eyes. Instead he “saw” with his tongue.
A camera mounted on Weihenmayer’s forehead fed a signal into an electronic device that turned the pattern of light and dark into electrical pulses. The pulses stimulated an array of 144 electrodes on a grid about the size of a postage stamp, which zapped the coded image onto his tongue. At first he described the sensation as being like candy pop rocks exploding, but later he experienced something more “out there” in the world – a sense of space, depth and shape.
Cheryl Schiltz danced for the first time in seven years after just 20 minutes wearing the same device. Normally she is unable to stand upright without holding onto something and concentrating on the stable things she can see around her, because her sense of balance has been destroyed by an antibiotic. If she tilts her head, the world spins and she stumbles.
Advertisement
But when Bach-y-Rita’s device translated signals from a sort of spirit-level-in-a-hat into patterns of pulses on her tongue, she quickly learned how to read them to substitute for her missing sense of balance. The effect persisted even when she took the device off, and lingered longer each time she tried. “I was normal,” she says, still emotional when she remembered the first time. “I was completely normal, and I had forgotten what it felt like.”
These “sensory substitution” devices are based on a technology called the BrainPort, which Bach-y-Rita describes as a kind of USB connector into the brain. They are close to commercialisation, initially for “wobblers” like Schiltz, but eventually as devices for blind people too. The US military is interested in developing the system to guide pilots and divers through dark skies or murky waters, and others are looking at more frivolous uses in virtual reality and games. But in neuroscience and philosophy circles sensory substitution has attracted a great deal of interest because of what it is revealing about the brain and our senses.
The fact that Weihenmayer sensed something “out there” and forgot the tingling on his tongue, and the way Schiltz felt completely normal, even although she was working with a different sort of sensory feedback to keep her balance, suggests that the traditional separation of the senses – sight, hearing, touch and so on – has little bearing on how we experience the world. The sense organ that picks up the information, and the way it is delivered to the brain, seem less important than the nature of the information itself.
“At first it felt like candy pop rocks exploding on his tongue. But later he felt a strange sense of depth”
Some see this merely as a dramatic demonstration of the brain’s flexibility. In other words, deprived of a primary source of information such as vision, the brain turns to a less prominent source, say touch, and extracts useful information from that sense. Others, however, have taken the findings much further, going to so far as to suggest that the traditional view of how the senses work is completely wrong.
The orthodox view of sensory perception is all about building internal pictures of the world. Sensory systems extract information from outside and channel it into the brain, which builds up a representation of our environment. Sensing is therefore the passive process of picking up signals; perception is the active process of turning the signals into useful information.
This model certainly chimes with our everyday experience: we talk about our mind’s eye, of mental images, and so on. And there is some scientific evidence that it is correct. Brain imaging reveals that when people see, hear, feel, smell or taste, specialised parts of their brains respond, and the timing of the response coincides with the moment of conscious perception.
Another line of evidence comes from our ability to imagine things. Even in the absence of sensory information, we can generate images and sounds in our heads, and most researchers believe that the process of imagining mimics real sensory perception. When people imagine seeing something, their visual cortex lights up. Moreover, using a technique called TMS or transcranial magnetic stimulation, which temporarily knocks out activity in the brain regions it is targeting, Harvard neuroscientists Steven Kosslyn and Alvaro Pascual-Leone have shown that people can’t imagine things if their visual circuits are switched off.
That’s all very well, says Kevin O’Regan, a psychologist with the CNRS, France’s national research centre at René Descartes University in Paris. But he is not convinced that this proves there is a representation of the world inside your head. In fact, he argues for a profoundly different view of sensory perception, and claims that Bach-y-Rita’s sensory substitution studies support it.
O’Regan’s starting point came some years back, when his interest lay mostly with eyes. He was curious as to how the world around us could feel completely stable in the face of our almost continuous eye movements, particularly large, jerky movements called saccades. O’Regan reasoned that saccades must be reported back to the brain so it could compensate for them as it built up its internal image of the world.
But he could find no evidence that this actually happens. There are neural signals associated with eye movements, but they didn’t seem to be involved in building up successive visual snapshots into one big picture. Since then others have tried, and failed, to find evidence that these signals are used to compensate for eye movements in this way. But if the brain doesn’t compensate for huge shifts in eye position, how can it create a stable image of the world?
Another puzzle came from a famous experiment by Dan Simons and Christopher Chabris of Harvard University. They asked volunteers to watch a recording of a basketball game and count the passes made by one of the teams. Early on in the game a man in a gorilla suit walked slowly across the court. Despite the fact that he was visible for about 45 seconds, around 40 per cent of the viewers failed to notice him. Yet when asked to watch the game with no task in mind, they all saw it immediately (żěè¶ĚĘÓƵ, 18 November 2000, p 28). To Simons, this is strong evidence that, despite the impression we have of seeing a complete and detailed image of the world, there’s a lot missing. We rely on the brain to fill in the blanks.
O’Regan goes one step further. He suggests that the mental image is not only incomplete, it is completely absent. We don’t reconstruct the world in our mind, we merely glimpse it in fleeting fragments. “There is no internal picture,” he says. Where most researchers would argue that seeing is all about building up an internal image, O’Regan has us flitting from one visual element to the next, only becoming aware of things when we need information. In this, O’Regan departs radically from the traditional view of sensory perception: sensing becomes an active rather than a passive process, with potentially profound ramifications.
O’Regan had been struck by an earlier version of Bach-y-Rita’s device, in which blind or blindfolded volunteers wore a larger array of electrodes taped onto the skin of their back or abdomen. One volunteer, with the array on his front, was “viewing” objects using a camera mounted on a tripod, but not getting anywhere. Out of frustration he grabbed the camera and started waving it around. When he started actively manipulating the camera in this way, something dramatic happened. He very quickly moved from merely feeling a tingling on his stomach to sensing the presence of external objects. Another volunteer, using a head-mounted camera, also suddenly felt the outside world become very real when he grabbed the zoom control – and almost fell over backwards as objects surged towards him.
What this suggests is that substituting touch information for visual information can produce a vision-like experience, but only when people actively control the camera in some way. Weihenmayer, for example, could almost see objects with his tongue. He didn’t taste them, and after a short while he didn’t feel them either. But this sensation only happened when the camera was mounted on his head, so he could move it as if he were scanning with his eyes. Similarly, blind people tapping with a cane experience open space at the end of their stick, not vibrations on their hands. They, too, are seeking information about the space around them.
Results like these have convinced O’Regan that sensory perception is not about passively collecting information but actively seeking it, and noticing how the information responds to our actions. We sense the world not by soaking up information, but by taking physical actions to interrogate it. “If the story is right, sensations are not generated in the brain,” says O’Regan. “They are things we do.” Sensory substitution works because it matters less to our brain where information comes from than the manner in which we gather it.
If it is right, O’Regan’s theory doesn’t just explain sensory substitution, it has philosophical implications too. In particular it suggests a solution to one of the “hard problems” of consciousness: why does seeing something feel different from touching it? The answer certainly doesn’t seem to lie in the electrical activity of the brain. Whatever sensory stimulus triggers the activity, be it touch, taste, sight or sound, the information is translated into electrical pulses. And no one has ever been able to find anything unique about these pulses, or where they are sent to in the brain, to explain why they produce different sensations.
That spongy feeling
O’Regan believes that his “sensorimotor” theory might provide an answer. Maybe, he says, touch, taste, sight and sound feel different because we have to perform different actions to collect the information.
Take, for example, the softness of a sponge. Where does the feeling of softness come from? No one has ever found a neural mechanism or specific part of the brain that exclusively lights up when you feel something soft. That, says O’Regan, is because there isn’t one. Working with philosopher Erik Myin at the University of Antwerp in Belgium, he has proposed that the feeling of softness comes from how you go about seeking information about the sponge. When you press the surface, it gives way. This is a different action from touching a sharp or hard surface, or a liquid.
While the theory seems to make sense for touch, or the difference between seeing and touching, what about the hardest “hard problem” of all, the sensation of different colours? How can we explain “redness” or “greenness” in terms of different actions? To complete his theory, O’Regan needed to find unique actions or activities that are associated with perceiving different colours.
It seemed an impossible task, but O’Regan and colleague David Philipona from the Sony Computer Science Laboratory in Paris were in for a surprise. When they looked at the physical properties of coloured surfaces they found fundamental differences in the way the different colours interact with light. In classical models, reflections from surfaces are the sum of two sources: one that behaves like the reflection from a matt surface, and another that behaves like the reflection from a sheet of glass laid over the matt surface. As we move our eyes, both types of reflection change their spectral composition, and the relationship differs according to colour you are looking at.
O’Regan suggests that as we move our eyes over a coloured surface, we detect something of this change in relationship. And by that we experience colour. The key point as far as perception goes is what happens when we probe the environment: it’s not the brain activation itself that gives the colour. The researchers have found that primary colours produce particularly distinctive changes, which may explain why they are universally recognised as special.
“Why does seeing something feel different from touching it? It doesn’t seem to be down to the electrical activity of the brain”
Already, O’Regan’s ideas have doubters. Bach-y-Rita thinks the explanation for sensory substitution lies with the remarkable flexibility of the brain. There are multiple pathways from all the senses to all the different sensory areas in the brain, he says. If you lose the main input from the eyes to the visual cortex, say, weaker pathways from the skin, ears, tongue and so on take over. This is what happens in the brains of Braille readers, who recruit their visual cortex when feeling the forms of the letters.
It may not be long before we know who is right, as O’Regan and colleagues are busy thinking up testable predictions of their sensory substitution theory. One such prediction is that it should be possible to make the substitution feel more convincing by making the information-gathering action “mimic” the original as closely as possible in the new medium.
To that end, O’Regan and colleague Malika Auvray have rigged up a video camera that represents the visual world in sound. Brighter objects become louder sounds, objects high up in the visual field are represented by high pitches and object low down by low pitches, while lateral position is represented with stereo sound. It is a little hard to imagine, but say the camera was looking at a light bulb in the centre of the field of view, you would hear powerful noise made up of a limited range of pitches centred in space. With a horizontal strip light you’d hear a smaller range of pitches over a wider space. As you move the camera, the sound changes.
In preliminary tests with a similar device designed by engineer Peter Meijer from Eindhoven in the Netherlands, the signals took a little getting used to. But after a couple of hours of feedback either from touching or being told what they were viewing, people were able to recognise objects by their sound. They could tell plants from statues and crosses from circles. But they weren’t fooled into thinking they were seeing. O’Regan’s prediction is that the more they can make the sound information follow the rules of visual images, the more like seeing it will feel. For example, Meijer’s system has a delay between moving the camera and hearing the sound. Another simple tweak would be to cut off the sound each time the subjects blink, which is exactly what happens to our visual world, though we scarcely notice it.
Perhaps one day blind people will play rock, paper, scissors in stereo surround sound. And if O’Regan is right, they could feel almost as if they are seeing. Now that really would be a sensation.
