OCEAN, castle, train. It鈥檚 not exactly Shakespeare, but it鈥檚 a start. A neural network has cobbled together a rudimentary vocabulary in a way similar to how a child learns to speak, by learning to associate images with spoken descriptions.
and at the Massachusetts Institute of Technology wanted to see if a machine could learn words without ever seeing them written down. They presented a neural net with more than 200,000 images and corresponding audio captions, and then tested it on a fresh set of 1000 images.
It learned to pair sounds from the captions with objects in the images. For example, it learned to associate images that featured lighthouses with the word 鈥渓ighthouse鈥�.
Advertisement
This method is closer to how we attain speech than standard learning algorithms, says at Tilburg University in the Netherlands. 鈥淸Children] listen to what people say and at the same time perceive the world and the situation in which these references are made, and correlate those things together,鈥� he says.
The neural net learned to name hundreds of things that recurred across the set of images, including the sky, trucks and trees. It picked up some adjectives too, for instance learning that the word 鈥渨ooden鈥� could be applied to photos of both attic rafters and desks ().
The technology could make it easier to build speech recognition programs for less common languages.
Conventional speech recognition algorithms, like those behind Apple鈥檚 Siri and Amazon鈥檚 Alexa personal assistants, are taught by comparing huge data sets of spoken and written words. 鈥淵ou have to train these systems with thousands or even tens of thousands of hours of people speaking, and you have to have someone manually go in and transcribe that data,鈥� says Harwath.
鈥淭he system could make it easier to build speech recognition programs for less common languages鈥�
As it is expensive and time-consuming to create these data sets, voice assistants usually only work in a limited number of languages. Siri offers 21 so far, while Alexa has only got to grips with English and German.
But Harwath and Glass鈥檚 neural net only needs audio captions and images, so it could help develop speech recognition for languages that lack large bodies of transcribed spoken data. It鈥檚 even possible, says Hawarth, that a neural net could use images as a reference point to translate from one language to another.
This article appeared in print under the headline 鈥淣eural net learns words like a child鈥�