YOU HAVE had a long day toiling in the virtual hospital. Your cranial wetware is numb, but you need one last look at your patient’s whole body scans. You point at the high-resolution wallscreen and quietly ask your number-crunching intelligent agent to run the data again. The wall fills with gigapixel 3D X-ray images. You reach in, grasp the internal organs, and rotate the image to find new perspectives. The sensors built into the wallscreen detect your every gesture, while the software recognises each twist and jab. There in the data forest, you find the peculiar tissue structure you’ve been looking for – your chance to make a small advance against a big disease. You grab the image, put it in your virtual briefcase, and wave the lights off.
How wonderful it would be if this scenario could be realised. With no keyboard or mouse to distance you from the data held on the computer, your mind would be free to look for new patterns and correlations. Just a few spoken phrases and gestures is all you would need to manipulate the image, see it from new angles and zoom in on the details.
In comparison with this dream, the computers of today make wretched partners. Although they can process millions of instructions per second, we are still forced to communicate with them via the QWERTY keyboard – a thin disguise for the old Teletype terminal. That wastes most of our abilities: humans are natural multimedia communicators and voice, gestures and eye contact are all used in any conversation.
Advertisement
Although we are still long way from multimedia communication with machines, many researchers believe there is hope of developing a computer that can detect and interpret a full range of human signals, including speech, hand movement, eye gaze, and facial appearance? There is good reason for their optimism: over the past few years signficant progress has been made in two areas that will both be essential for developing natural communication. First, some sophisticated software is now being developed which brings together gesture and speech so a computer can understand what you mean by integrating what you are doing with what you are saying. Secondly, new hardware that can detect gestures patterns is being rapidly developed. Several research teams are finding ways to throw out the “data gloves” – the clumsy devices that are still necessary for anyone who wants a computer to understand their gestures – and replace them with unobtrusive “electronic eyes”.
Attempts to program a computer to integrate gesture with speech began back in the late 1970s at the Massachusetts Institute of Technology Architecture Machine Lab (which later became the Media Lab). The “Put That There” project tracked a point on the user’s wrist with a magnetic field detection system. The user, wearing a glove embedded with light sensors, would point at an object on a computer screen to select it, then speak into a voice recognition system to specify the action was to be taken.
MIT is still leading the field in this area. Richard Bolt, along with David Koons and co-workers at the Media Lab, are now building systems that take speech, gesture, and eye gaze as inputs, drawing on the many developments of the intervening twenty years.
One of the landmarks was the data glove, pioneered by a team at the University of Illinois in 1977 and further developed by Thomas Zimmerman of VPL Research in California. This permitted a computer to sense the bending position of the wearer’s fingers and thumb. The amount of finger flexion was measured using a simple piece of flexible tubing running down the side of each finger with a light source at one end and a detector at the other. As the wearer bent a finger, the amount of light that passed through the tube decreased, indicating the degree of flexion. Zimmerman later improved this basic design by using more sensitive optical fibres, instead of plastic tubing.
The human element
Bolt and his team at MIT have taken advantage of this increased accuracy in their latest project – finding methods of integrating gesture, eye gaze and speech to control computers in a way that is much closer to our natural patterns of communication.
If someone gives you directions in the street, for example, he or she may point and look towards a building while saying “over there”, for example. It is easy for humans to understand, but not for a computer. One problem is that the information from these different modes of communication is organised very differently. The pattern of speech is governed by the rules of grammar, while gestures fall into classes of pointing, pantomime, and conversational emphasis. Eye gaze has many of the features of pointing, as the speaker looks at an object, at his or her hands, or at the listener.
Koons and his colleague Carlton Sparrell at MIT have tried to introduce hand motions into their communication system, including complex gestures that describe shape or movement. This raises some difficult problems. Take a spoken sentence such as “The fish I caught with that rod and reel was this big”, acccompanied with a gesture to show how big the fish was. This demands that the computer system measures the separation of the hands just as the words “this big” are being spoken. To solve this kind of problem requires a new kind of software program that can extract clues from speech and gesture simultaneously.
Sparrell and Koons attacked the problem by developing ICONIC, which comprises separate software modules designed to work together. The speech-recognition module passes information about a particular word as it is being spoken to a program known as a parser. It is then up to the parsing software to encode the content and structure of the speech. Another module extracts information on the position of a person’s hands, the shapes the hands make and their basic movements by monitoring data picked up by sensors in a pair of data gloves. This information is then sent to an interpreter module, which contains a library of hand positions and sounds, so the data arriving from the other software modules can be identified.
Naturally, the rules underpinning the “smart” ICONIC software derive from research into the importance of gesture in human communication. Such research has shown that for English speakers, the main hand movement that accompanies any sentence will occur just before the peak syllable in that sentence. This knowledge helps to trigger the interpreter program at the right time. When processing a command like “rotate the cup like this”, for example, where the cup referred to is a graphic on a computer screen, the gesture-recognition module would recognise the turning actions of the hands in the data glove while the speech-recognition software would recognise the words “rotate” and “cup”. The interpreter program would kick into action on the words “like this”, and then extract the gesture information and tell the computer in what way to rotate the graphic image of the cup.
Using ICONIC, humans can begin to interact with a computer using a combination of spoken phrases and free-hand gestures. Since people use this method of communication every day, the researchers are optimistic that the system brings us one step closer to the human-humanoid interface.
But most also agree that being able to recognise a combination of gestures and spoken words is only part of the problem. To have truly natural communication between a computer and a human, those cumbersome data gloves just have to go. Several research teams are working on gesture recognition without the need for specialised accessories. Most of this work uses video cameras to capture hand and body movements.
Virtual hands
Some early systems required the user to have a target on their hands. The target would be picked up by a static video camera. A more advanced system called Digiteyes, developed at Carnegie Mellon University in Pittsburgh, dispenses with all this by building a sophisticated model of the human hand into the software. The computer models the geometric and kinematic behaviour of the finger segments, finger joints, and how hands move. When someone places his or her hand in a special prearranged pose, the video system locks onto it and starts to track. The video compares each frame with the one before so it can estimate where the hand has moved over time. The real difference in hand movement is measured by researchers and then compared with the system’s estimate, and a slight correction brings the estimate in line with reality, and so on for each frame in the sequence.
Christoph Maggioni, a researcher at the German electronics giant Siemens in Munich, has found that hands can be distinguished from other parts of the body by their skin colour. A colour video image can compare each pixel with information stored in a table of skin colours to find the hands. Because the colour table can be refined for each user, the system responds to a wide range of skin colours.
Maggioni’s recognition system has been designed for use in a railway yard where the trains are shuttled and switched by touching their images, and for controlling surveillance cameras in an underground station, where the security officer can point to select a particular view.
Neil Gershenfeld at the Media Lab has come up with a different approach that does not require video cameras. His technique relies on the fact that humans are just big bags of saltwater, containing an aqueous broth of ionic conductivity.
He found that it was possible to detect body movements using the oscillating fields generated at small metal antennas. Pass your hand through the field and your body acts like a short circuit, signalling that something is nearby. Another method is to put the body in contact with the transmitting electrode, thus making it into a transmitter. The receiving electrode then records varying field strengths as the body moves around.
Gershenfeld believes that this approach could lead to “smart furniture” which would respond to humans. Possible developments are a desktop that senses page-turning motions to turn over pages on a virtual newspaper on a computer screen, or a computer that knows where its user is. Other possibilities are smart rooms that track body position and smart floors that sense the number of people present.
While all these technologies hold out hope that one day computers may be able to communicate on more human terms, the dream of a natural human-computer interface remains elusive. Present systems are crude: the vocabulary of gestures they recognise is limited and there is always some awkward technology that gets in the way of making free movements. And the goal of truly “deviceless” or unobtrusive gesture recognition will not be overcome soon.
As Edward Altman, a researcher at the Advanced Telecommunications Research Laboratories in Kyoto, Japan, points out, another difficulty is that human hand movements flow continuously from one configuration to the next, and the same gestures are very personal. Ask six people to wave someone goodbye and you will have six different kinds of waving.
Altman has started to address this problem by representing complex hand movements by a set of nonlinear differential equations, which can analyse the data whether it comes from a data glove or a video image. Altman’s system is rather like an array of strange attractors, to use the mathematical jargon. When the input “approaches” one of the attractors, the circuit associated with it wakes up and begins to follow. By watching the activity in the array, and sensing which circuit locks onto the data, patterns of gesture can be determined.
Going through the motions
Altman says his is a general method for detecting patterns in complex changing data streams, and is applicable to speech recognition and computer vision as well. His current efforts are directed at the learning problem: how to train the array to learn a large repertoire of gestures. For this, he has been exploring the use of genetic algorithms – self-optimising processes that would arrange the parameters in each nonlinear subsystem of the array to store a series of gestural motions.
Work like this holds promise but is still at the laboratory stage. In the real world, no one has yet come close to analysing the complexities of human gesture, where hands may overlap and move at different speeds against cluttered backgrounds, often in ways that are either culturally or personally specific.
Even when we get to the stage where computers can interpret your hand movements, integrate them with what you’re saying and respond to where you’re focusing your gaze, there’ll still be one more problem left. To create a truly natural interface, some sort of force feedback, where the hand actually senses a solid body when it grabs the virtual one, will ultimately be needed. When the computer can reach out and give you the sensation that you’re shaking hands with it, the final goal will have been reached.