“THE problems we identified were noise, emotions, accents and daleks,” says David Nahamoo. Did the leader of IBM’s “superhuman” speech-recognition project just say he had a problem with daleks? The arch-enemies of the TV character Doctor Who certainly had unusual speech patterns but isn’t it far more likely Nahamoo said “dialects”?
Identifying the correct word would only take a moment’s thought for most people, despite Nahamoo’s Iranian accent and the noisy telephone line. But his best speech-recognition machines would probably never have got it at all. Which just goes to show how much work his team has left to do.
Speech-recognition programs have been in development for around 40 years. Their success rate varies from application to application, but where humans get 1 word in 200 wrong, machines still make a mistake on about 1 word in 20. Nahamoo aims to close this gap and then have the machines overtake us – IBM hopes to have achieved superhuman speech recognition by 2010. “There’s been a 25 per cent reduction in word error rate per year over 4 to 5 years,” he points out. If this exponential improvement continues for another 8 years, he says, computers will be better than humans at recognising words.
Advertisement
But that’s a very big “if”. While you can now buy dictation software packages such as IBM’s ViaVoice and Dragon’s NaturallySpeaking, they only start to approach the skills of a human secretary after they have been trained to recognise the way you talk, and they have to be used in quiet environments. You also have to speak using well-separated words and little emotion – there’s no point talking to them as you might to your close friends.
Passengers can check flight details by talking to a computer over the phone, but try to communicate anything other than the essential information the airline computer is listening out for and you’ll soon find you are asked – terribly politely – to repeat yourself. And although many mobile phones now have a form of voice recognition, it is pitifully limited, simply matching patterns in audio files rather than recognising words. You must first record a spoken phrase such as “call my wife” in the phone’s memory, and the phone must recognise the sound of the recorded phrase before it dials the number. Computers have a long way to go before they can cope with everyday chatter.
That’s hardly surprising. Consider the physical complexities of human speech. Vowels are formed when we force air from our lungs over the vocal cords and into the tube formed by the jaw, tongue and lips. Here, some frequencies are amplified, which dictates the vowel that a listener hears. Then there are nasals, formed when we force air through the nose rather than the mouth. Another set of sounds, the fricatives, are created by forcing air through a constriction in the mouth. And then there are the stops – sounds formed by stopping the airflow. Each of these mechanisms produces a sound wave containing a certain set of frequencies. To recognise a word, a computer has to analyse the energy and frequency of each wave produced in each sound.
But because everybody produces speech sounds in a different way, there isn’t a single, characteristic frequency and energy for each one. To overcome the problem, researchers have used what are called hidden Markov models (HMMs). These are mathematical representations of a typical speech sound – “ee” for example – that incorporate some leeway. To trigger recognition, the sound must lie within a certain range of frequencies and a certain range of energies. HMMs are produced using huge databases of sounds that encompass all the typical variations in the way people talk.
The models also incorporate statistical data on how a sound usually combines with other sounds. When the computer hears a string of speech sounds, it breaks it up into the chunks it recognises and looks to see whether the neighbouring sounds make the kind of sequence it has been trained to expect. On that evidence, it outputs what it thinks is the most likely word it has been fed. Once it has guessed a couple of words, it will use these to improve its chances of guessing the next word. It does this by referring back to the “language training” of its software, in which the word order of millions of sentences was analysed.
But this may not be enough if we want to go much further. “I don’t think the HMM we are currently using is going to take us all the way to bridging the gap between machine recognition and human recognition,” says Alex Acero, head of Microsoft Research’s speech-recognition project in Redmond, Washington. Humans, for instance, use a knowledge of grammar, syntax and semantics to narrow down the range of possible words they are listening to, and linguists have spent decades studying how this works. The best that most computer models can do is choose a word based on the previous two words, says Acero. “When they hear about this, most linguists just fall about laughing.”
To narrow down the range of possibilities that their machines have to search through, Microsoft, IBM and just about every other speech-recognition research group are now trying to teach them more of the subtleties of language. When Acero’s 3-year-old son saw a dove and asked his father what it was, the boy was immediately able to use this new-found word correctly. Acero’s machines are not so gifted. “They generalise very poorly. You can add ‘dove’ to the lexicon, but you need to see the word surrounded by all possible words,” he says. To put that in context, Acero says it takes about 100 hours of speech data to train acoustic models to recognise the various combinations of speech sounds – phonemes – that occur in a language. But to learn what combinations of whole words are possible (and not possible), a machine would need to hear billions of words in real sentences – equivalent to 100,000 hours of speech, more than 10 years of continuous chatter.
The Microsoft researchers think they can go further with another approach: taking into account the limited range of sounds it’s possible for the human anatomy to produce. Acero points out that standard computer models will try to identify some sounds that could never have been made by a human being. “There is a lot of redundancy in the system, and we are not really exploiting it well,” he says. His Microsoft colleague Li Deng is building a speech recogniser that examines the resonant frequencies of speech. The idea is to find out which resonant frequencies correspond to which positions of the tongue, and hence which speech sound is being made.
“If we had a structure like that, we could use a lot fewer parameters in the system, and we’d be able to adapt more quickly,” says Deng. “If we go from a male to a female, all we’d need to do is move all the resonant frequencies higher by a certain amount. Give me all that information and I don’t need to get another 100 hours of speech from that person – which is what we need today.”
The biological approach should also help machines get to grips with fast speech. Normally, the tongue moves to certain positions, known as targets, to form certain vowels. But when we speak quickly, the tongue doesn’t have time to reach one target before it has to move to another. “With long words, speaking fast, you might drop one or two phonemes. Humans don’t even notice,” Acero says. “Machines are expecting that phoneme to be there, and so you get a higher error rate.” But if the model knows about the dynamics of the tongue, lips and the other biological “articulators”, it can hear if a person is speaking rapidly and compensate. With an understanding of how the mouth works, the machine can also rule out any sound combinations that are physically implausible.
IBM’s machines are now learning to cope with background noise by lip-reading, just as humans do, to tell both whether someone is speaking and what they are saying. “In a very noisy environment we start using much more than our ears,” Nahamoo says. “If we can see if a mouth is moving or not, that’s a good indication of whether it’s worth the effort doing signal processing – if it’s not moving, then why bother listening?”
So the IBM researchers have been creating speech recognisers that will take in both acoustic and visual information. “We get a lot of improvement in performance,” Nahamoo says. For many tasks, they can improve the signal-to-noise ratio by a factor of six. For tasks that involve a small vocabulary – such as transcribing digits – visual input can cut the word error rate fourfold. So in a noisy environment, where the error rate might have been 1 in 20, adding a camera to the microphone could bring it down to a much more acceptable 1 in 80.
Working on noise problems isn’t enough, though. If it is to match human capabilities, machine speech recognition will have to work for every language. After all, every baby has the same basic equipment in place, and it works whether it needs to learn the clicks of Xhosa, the tonal pitch patterns of Mandarin, or the confusing “liaisons” of French. Only the “training data” dictates what language the child will end up learning. So in theory, it should be possible to create one basic program that could learn to recognise any language.
Although English has been the primary focus of the developers, everyone in the field recognises the importance of globalising the technology – not least because there are a lot of untapped markets out there. Where western users are largely computer-literate and can always resort to a keyboard interface, speech recognition could revolutionise technology use in other cultures.
In India, for example, there is no standard for keyboard input across the country’s various languages. Speech recognition could eliminate this problem. IBM’s India Research Laboratory is attempting to adapt ViaVoice software to recognise Hindi. Then the researchers plan to extend the
software’s capability to other Indian languages: the ultimate goal is a multilingual system that could also be used for telephone banking, directory enquiries and so on, giving every Indian access to 21st-century communications without the hurdle of becoming computer literate.
Allied Business Intelligence, a technology research think tank based in Oyster Bay, New York, has predicted that the global speech-recognition industry could be worth $5 billion by 2008. Will this lure be enough to make superhuman speech recognition a reality? It’s certainly not a question of computing power – for a long time, researchers have had more than enough. But will they be clever enough to use it to match human capabilities?
Quite possibly: superhuman doesn’t have to mean perfect. “Human performance isn’t a fixed point,” says Philip Woodland of the University of Cambridge. “For one thing, we get bored – there are some applications where machines are as good as people, or you would just never want to get people to do them.”
In tedious tasks, people make errors. Transcribing long strings of telephone numbers would be an extreme example. Many medical personnel, such as doctors and radiologists, currently rely on dictation software to automate the process of taking down lists of symptoms and observations. A number of firms have developed machines that recognise a limited range of common words used by doctors, says Woodland. “The first stage is done automatically and then people can clean it up. Machines can be pretty accurate at doing an initial pass.”
Even in less tedious tasks, machines are starting to prove their worth. Much of Woodland’s research focuses on making accurate transcriptions of telephone conversations or news broadcasts. On a news broadcast, the best systems have a 10 per cent word error rate. Humans might manage 2 or 3 per cent, says Woodland, but the contest isn’t a fair one. “With people you have to let them have multiple passes over the data, access to dictionaries and all that sort of stuff. If there’s a time constraint they do a lot worse.”
Acero is certainly optimistic that superhuman voice recognition will eventually be possible, but he remains cautious about exactly when it will happen. “Most people think we will reach human performance somewhere between 10 and 30 years [from now]. But it depends on who you talk to, and what task you’re talking about.” Dealing with daleks – or should that be dialects? – is still a high priority.
What’s in a name?
WHEN Adoram Erell speaks to his cellphone, it’s a multilingual moment. Most of the contacts in the address book have Israeli names, but his phone doesn’t support Hebrew speech recognition. To get over the problem, Erell typed in the names using the western alphabet and has switched to German speech recognition – he reckons that, acoustically at least, it’s the closest match. “It works very well,” he says.
Erell is a researcher at the Intel-DSPC labs near Tel Aviv, Israel. The software on his phone is not yet on the market – Intel only demonstrated it publicly for the first time last month to phone manufacturers and software developers at the International Telecommunication Union conference in Geneva. But despite the lack of Hebrew, he says it takes speech recognition on mobile phones to a new level.
With any of the phones currently on the market, you have to record the sound of a name you want it to recognise. The phone doesn’t convert your speech to text, it simply matches the audio file it makes when you speak a name to the one it has in its memory. But the Intel software doesn’t need pre-recordings. It recognises the sound of any name entered as text in its address book. And it works whoever is speaking.
The system uses the spelling of a word to work out the phonemes it contains when spoken. The process of speech recognition is simply one of matching the combination of spoken phonemes with the ones it has worked out from the spelling. That’s why the language is so important, because pronunciations of the same spelling can vary widely between languages – for example the “ch” in “chat” is pronounced entirely differently in English and French. The computer has to know in advance which language you are using.
The bottom line, however, is that the software has to work on a mobile phone more efficiently than hunting through the list of contacts using the keypads. “Browsing with key navigation is not terribly efficient, but it works. So you need an accuracy that is significantly more efficient than key pushing,” Erell says.
And is the Hebrew version coming soon? Not exactly. It’s definitely do-able, says Erell, but for now, he will have to work on his German accent.
