THE SONY Computer Science Laboratory in Paris is a cosmopolitan sort of place. Here an international team of researchers converse in English, French and Japanese. But the air is also full of more exotic voices babbling, or uttering strange words such as 鈥渨abaku鈥� and half-recognisable phrases like 鈥減ush red wa blue ko鈥�. These are Luc Steels鈥檚 talking robots. Even the most accomplished linguist will have problems making polite conversation with them, because they don鈥檛 speak any language we know. Instead they invent their own.
For decades, linguists, anthropologists and biologists have argued about what allowed our ancestors to evolve something as complex and elegant as language. What sort of brain would they have needed? Followers of linguist Noam Chomsky believe that some kind of linguistic rules must be encoded in our genes and brains. So to get language off the ground in the first place, specialised linguistic structures must have evolved in our brains. His opponents argue that picking up language is simply a matter of learning, and that given enough examples we can extract meaning, rules and order from what we hear, through a sort of subconscious statistical analysis. So language evolution is more about developing the right learning and rule-extracting skills (快猫短视频, 21 August 1999, p 36). But years spent listening to electronic voices has convinced Steels and his colleagues that there is a third way.
Many times over they have heard new languages evolve in computers that are not programmed either with the equivalent of an innate linguistic sense or statistical powers. Instead, for each new language, rules are gradually invented, negotiated, built upon and spread by pairs of robots talking to and learning from one another. 鈥淟anguage is a complex adaptive system,鈥� says Steels. 鈥淚t鈥檚 like a living thing.鈥� It self-organises and spreads like a virus.
Advertisement
Over the past five years Steels and his colleagues, both in Paris and at the Free University in Brussels where he鈥檚 head of the artificial intelligence lab, have witnessed the emergence of language-like communication between robots. His machines have created basic speech sounds and put these together to form words that have meaning. Other researchers are impressed by the flashy presentation and the neat technology, but few have been persuaded that Steels has got anywhere near the complexity of human language. But that view looks set to change after Steels鈥檚 presentation at this week鈥檚 Evolution of Language conference at Harvard University. There he revealed that his robots have done what some people thought impossible: they have evolved a form of grammar. 鈥淚t鈥檚 a bit of a bombshell,鈥� says Chris Knight, an anthropologist at the University of East London, who is one of the conference organisers. 鈥淲ith minimal programming, just goals, agendas and the desire to form relationships, you end with something a little bit like language.鈥�
Steels鈥檚 work stems from an idea he had back in 1995. His starting point is that language, particularly in its earliest phases, develops out of shared experiences, and so has to be grounded in the real world. In other words, there has to be something to talk about. So he began designing ways for robots with 鈥渂odies鈥� and 鈥渟enses鈥� to have joint experiences, and programmed them with the desire to communicate these. 鈥淗e has done something no one else has done,鈥� says James Hurford, head of the Language Evolution and Computation Research Unit at the University of Edinburgh and another conference organiser. 鈥淗e鈥檚 actually got real robots to have some communicative interaction about real things.鈥� And as the robots have been finding ways to speak to one another, Steels has been able to investigate what kind of brains, memories and other abilities are needed for language to evolve.
Most of the experiments so far have focused on developing words to express objects the robots can see. This is harder than it sounds, because the robots not only have to invent new words and store the meanings, they also have to work out concepts upon which to build the words in the first place. And they have very little to work with. A camera allows them to see the world around them鈥攚hich consists of a white board covered with colourful geometric shapes鈥攁nd they can point. Their 鈥渂rains鈥� include a standard text-to-speech module, some memory and image-analysis software that can categorise what the camera 鈥渟ees鈥� in terms of colour, shape, size, texture, position and other sensory dimensions.
Steels programs pairs of robots to play a cooperative guessing game. One takes the role of 鈥渟peaker鈥�, chooses an object it can see and then tries to express which one it is 鈥渢hinking鈥� about. It can use any concept, but it has to devise ways to convert the information from the sensory channels into useful categories such as 鈥渓eft鈥� or 鈥渞ed鈥�. Sometimes a combination is needed. If there鈥檚 a red square and a red circle, 鈥渢he red one鈥� obviously won鈥檛 be enough. If the robot has no word already in its memory鈥攁nd they don鈥檛 have any to begin with鈥攊t will invent something by combining strings of sounds from a basic repertoire of syllables.
The other robot, the 鈥渓istener鈥�, must then try to figure out what the speaker is talking about. It signals by pointing, and will guess if it has no memory of the word it hears. If it鈥檚 right, the speaker confirms with an 鈥淥K鈥�, and the word and meaning are reinforced in both their memories (see Diagram). In this way each robot develops a lexicon鈥攁 tree-like mental look-up table of words and meanings. New words and concepts grow like new branches on the tree, and those that are wrong or not used wither away. In one of the largest experiments, a population of robots managed to develop around 8000 words (see 鈥淰ox pop鈥�).
Clearly it鈥檚 not exactly how humans began using words. For a start, the robots鈥� speech sounds are all given, while our ancestors must have evolved the ability to produce different sounds. And the robots are programmed to interact in a far more regimented way than any ancient humans. But, says Steels, it does say important things about how language can evolve. For a start, the robots don鈥檛 develop concepts, rules and hypotheses statistically from vast numbers of examples. They can make an educated guess. We also have a rich capacity to guess what other people might mean, says Steels. 鈥淚n my view this is the essence of language learning and an absolute prerequisite for the origins of language.鈥�
But do the robots need the sorts of specialised innate language skills that Chomsky spoke of? When it comes down to it, says Hurford, the programs that drive the robots and play the games are quite specific. 鈥淵ou could say that corresponds to something that you鈥檝e built in, so in some sense it鈥檚 innate,鈥� he says. Steels agrees, although he dislikes calling his programs an innate predisposition for language. He accepts that robots have abilities that could be called innate, such as a visual perception system that allows them to pick out one object from another and recognise shapes. They also have the desire to interact and be understood, and the ability to engage the attention of others. 鈥淭here are obviously a lot of things, but they are very general,鈥� he says. Our ancestors would not have had to evolve these abilities specifically for language.
Perhaps the most radical claim Steels is making on the basis of these experiments is that the robots are developing words and concepts together. This sets his ideas apart from those who argue that statistical learning is all-important. While these researchers believe there鈥檚 enough statistical regularity in the world to extract a set of basic 鈥渘atural鈥� concepts, the robots clearly do something different. They invent their own categories, which can only come from the sensory information they gain. By experience and guessing, those concepts develop word labels, such as 鈥渨abaku鈥� and are passed on to other agents. 鈥淢y idea is that language and meaning co-evolve,鈥� says Steels. 鈥淵ou cannot just have a concept and then develop a language label鈥攖he two are intimately linked.鈥�
This is a fascinating assertion, says Hurford. 鈥淭he robots do in some sense develop their own conceptual framework. I鈥檓 sure he鈥檚 right.鈥� It鈥檚 a totally different idea to a lot of traditional theory. As far back as Plato there have been philosophers who argued that concepts are simply waiting to be discovered. And many linguists believe that language cannot emerge until we have an understanding of these universal concepts. Steels鈥檚 picture is different. 鈥淭here are many ways to view the world,鈥� he says. 鈥淏ut I think that language helps society see things in a similar way. It鈥檚 a coordinating force.鈥� Knight agrees. 鈥淟anguage is clearly social, there鈥檚 no question about it,鈥� he says. 鈥淪ignals, sounds and concepts are socially generated.鈥�
Steels points to the way we name colours as a good example. Because they fall on a spectrum there is no reason why we should split them in any particular way. Our visual system constrains us to some extent, but it doesn鈥檛 explain entirely the categories we use. Culturally we came to a consensus, says Steels. He points to studies by Jules Davidoff of Goldsmiths College, University of London, and his colleagues, who describe how different cultures use entirely different colour categories. For example, English speakers recognise eight鈥攔ed, orange, yellow, pink, green, blue, purple and brown. But in the language of the Dani people of Irian Jaya in Indonesia there are only two words used to express colours. And the Berinmo of Papua New Guinea have five categories (Nature, vol 398, p 203).
How words acquire meaning is fascinating stuff, but it鈥檚 only a small part of language. It鈥檚 grammar that has caused the most controversy. While there鈥檚 general agreement that words and sounds are simple enough to be copied and memorised, many linguists, not just those who side with Chomsky, believe that grammar is just too complex and messy and that the basic rules must be innate. Steels disagrees. His latest experiments show that by watching and describing things happening, and so guessing what the descriptions mean, his agents can evolve at least one form of grammar, without any pre-set rules.
The game is similar to the word-creation experiment, but this time the robots watch dynamic situations, such as a hand grasping a ball. This has allowed Steels to explore the emergence of case grammar, which is what we use in describing the relationship between objects. If you just have two objects, 鈥渂all鈥� and 鈥渉and鈥�, with an action 鈥済rasp鈥�, without rules, you quickly run into problems in communicating unambiguously. Case grammar in English is largely done with word order鈥攕o 鈥渄og bites man鈥� has a totally different meaning from 鈥渕an bites dog鈥�. Other languages use word endings or additional words, known as case markers.
In trying to solve such ambiguities, Steels鈥檚 robots have the ability to invent their own rules. 鈥淧ush red wa blue ko鈥� was a construction used to communicate that 鈥渟omeone pushes a red object against a blue object鈥�. By seeing the same action as the speaker, the listener can work out what role the unknown words 鈥渨a鈥� and 鈥渒o鈥� are playing. Because the robots are programmed to be economical with their memories, eventually they develop general rules for using these sorts of case markers, that might, for example, label who is 鈥渄oing鈥�, and who is 鈥渂eing done to鈥�.
Rewriting the rules
鈥淚t鈥檚 the first time with computer programs and real computer vision we鈥檝e seen a real grammatical system emerging,鈥� says Steels. More importantly it shows that the basic categories usually associated with case grammar are not innate, he adds. Now he plans to test whether his talking heads can evolve the use of different tenses鈥攆or which they鈥檒l have to develop concepts about the temporal relationships between things they see.
This work has made Steels think about grammar in a different way to many linguists. 鈥淚t鈥檚 no frustrating set of rules to be used rigidly and to be grammatically correct. It鈥檚 purely to help us to understand,鈥� he says. This has convinced him that language is constantly evolving鈥攏ot just the words we use but also, over time, the whole rule basis upon which our unique form of communication is constructed. 鈥淐homsky has a static view of language,鈥� says Steels. Chomsky insists that some fundamental rules of grammar are common to all languages, because they are in some sense hard-wired into our brains. Steels just doesn鈥檛 buy that.
But if he鈥檚 hoping to convince supporters of an innate language ability to think again, he鈥檚 in for a tough time. 鈥淕iven what we know about real human languages, and the way real human children acquire them, only hypotheses that attribute some innate specialisation for language to children can account for the data,鈥� says Steven Pinker from MIT. He also points out that genes linked with language are now being discovered. One study, for example, showed that identical twins have greater similarities in some of their grammatical habits than ordinary siblings or fraternal twins. Another revealed a gene linked to a specific language defect (Nature, vol 413, p 465). 鈥淚t is incoherent to say that there are no genetic factors, because evolution, by definition, is a change in genetic composition over time,鈥� says Pinker.
And Jeff Elman of the University of California at San Diego points out that Steels鈥檚 work might not rule out the statistical learning ideas at all. He might be exactly right about how language evolved in the first place, says Elman. But that doesn鈥檛 mean that children don鈥檛 then learn language by induction and example. The two ideas are not mutually exclusive.
But so far, the grammar skills of Steels鈥檚 robots are not sufficiently human-like and the conditions not realistic enough to say he鈥檚 found the roots of language. And though he acknowledges that simulations like his will never be able to prove how language came about, only to test under what conditions it can, Steels is already addressing the problems of realism. In the first experiments his robots were programmed with basic speech sounds, but now they can evolve their own. The researchers have found that if they give the computers a realistic virtual human vocal tract, and a virtual ear, they develop sounds remarkably like the ones we use.
And instead of playing games where the interaction follows a strict pattern, Pierre-Yves Oudeyer, a colleague of Steels, has developed a system of emotions, to give the robots a desire to seek out interactions and a sense of 鈥渇ulfilment鈥� after communicating successfully. It鈥檚 much more realistic than the ritualised games. 鈥淗ere they are truly autonomous,鈥� says Oudeyer. The emotion-led games are interactions rather like mother and baby, learning how to engage each other鈥檚 attention, create sounds, imitate and reinforce successful copying. It makes them chatter and babble almost continuously.
But perhaps the only real proof will come when all the pieces of the language-evolution puzzle have been put together, to see whether it鈥檚 possible to simulate the whole process, progressing through a stage rather like babbling, on to simple language, then a lexicon, and finally grammar. Even then, we won鈥檛 know if that鈥檚 what really happened in our own evolutionary history鈥攐nly that it might have.
One thing鈥檚 certain. Robots and computers can talk to each other, whether we like it or not. How long before they are teaching us to speak a new language? It鈥檚 an interesting by-product of this academic curiosity. No wonder Steels says: 鈥淟inguists find this work all very odd.鈥�
Vox pop
TO TEST how language develops and spreads in a population, Luc Steels from the Sony Research Laboratories in Paris needed to find a way to do a large-scale social experiment鈥攚ith robots. He wanted the robots to be able to meet and travel, while still allowing the researchers to control their surroundings. Steels, along with Angus McIntyre and Frederic Kaplan, hit on the idea of teleportation.
They realised that they could build just a few robot bodies but many robot brains. The brains were in the form of software 鈥渁gents鈥� that could be teleported, or downloaded into a body over the Internet. In this way hundreds, even thousands, of different robotic agents could meet and interact in pairs in a few controlled locations all over the world.
The robotic hardware鈥攁 camera to see the surroundings, and the ability to point, speak and hear鈥攚as stationed in various public places in Brussels, Paris, London, Tokyo and elsewhere. At each location a series of agents would use the bodies to play a language game in which they had to devise words to describe geometric shapes on a white background. Agents would then teleport off to another location with a different array of shapes, so that in a fraction of a second words learned in Tokyo might be used in Paris.
The experiment became a public one. People were invited to create an agent through a website and choose where it went to play its games. They could follow its progress and even act as a speaker or listener themselves, teaching agents words, or trying to learn from them.
Soon these globetrotting robots were using English, German, French and Japanese words, as well as ones they had invented themselves. Briefly, some of the more isolated groups would develop different dialects and languages, but with more interaction, words and meanings mingled and spread. At its height, the population was about 3000 strong, using maybe 8000 words, 300 of which were stable and almost universally understood.
Then the inevitable happened. Hackers began to teach agents to swear and the foul language spread like a virus through the population. You can鈥檛 have robots swearing in public, so the experiment had to be stopped. Steels admits they may have been naive. But, he adds, it showed us one thing: the more robots interact with people, the more they talk like us.