THERE鈥橲 nothing quite like downloading your digital photos to your PC and showing them off to your friends. But how much more useful it would be if your PC could listen to the banter and use it to caption and index the pictures. That鈥檚 exactly what researchers at Hewlett-Packard in California are trying to do.
Digital photography is booming, and people are storing ever greater volumes of photos on their hard drives. The trouble is that people rarely label their photos. 鈥淭his is the weak link for digital photo collections,鈥� says Margaret Fleck at HP鈥檚 lab in Palo Alto. 鈥淚n 10 years鈥� time, finding something amongst them will be very difficult.鈥�
Fleck鈥檚 answer is to tap into the wealth of information in the conversations we have when we talk about our photos with friends. She says the stories we tell don鈥檛 merely describe the photo, but also talk about the events that happened before and after the picture was taken.
Advertisement
To harness this information, Fleck has developed software that records these conversations to hard disc, converts the speech to text using a speech-recognition program, and then extracts keywords with which the photos are captioned and indexed.
Her current prototype runs on a PC equipped with a microphone, and automatically starts recording when you open a digital photo album and begin a commentary on the pictures. It stops after 30 seconds if you are not talking. The system converts the soundtrack into text in real time, and identifies keywords such as 鈥淰enice鈥�, 鈥渉oneymoon鈥� or 鈥淐hristmas鈥� that can be used to index the photograph. To find an image later, you simply type keywords into a search box.
As speech-recognition software becomes more accurate, the system should be able to generate lengthy captions describing each scene. Today鈥檚 commercially available speech recognition systems are nearly 99 per cent accurate, but they have to be trained for a particular user, and the user has to speak directly into a microphone.
However, Fleck wanted to build a system that wasn鈥檛 limited to one individual, and could catch 鈥渙pen-air鈥� conversations 鈥� not spoken directly into a mike 鈥� between people crowded around a monitor looking at photos. Although such systems are far less accurate, the conversations they capture are richer. She used some of HP鈥檚 own speech-recognition software, originally intended for transcribing webcasts of TV and radio news bulletins spoken by multiple presenters. It gathered enough keywords to index the photographs.
鈥淚t鈥檚 a really clever way of annotating pictures,鈥� says Mor Naaman of Stanford University, also in Palo Alto, who is working on using GPS devices built into digital cameras to annotate photographs with details about the picture鈥檚 location (see 鈥淲ish you were here鈥�).
Fleck believes her method will be just one of many that we will rely on to help us organise digital files, as hard drives approach terabyte levels over the next few years. 鈥淧robably any good solution is going to use several different approaches,鈥� she says, pointing to work at the University of California in Berkeley. Researchers there have developed software that can identify key elements in photos, such as types of animal, flowers, geographic features like rivers and mountains, and use them to index pictures.
Wish you were here
We can already add location information to digital pictures. For instance, some cameraphones can tell where they are simply by quizzing the nearest base-station antenna, and that information can be added to any pictures the phone snaps.
And now that GPS chips are being built into high-resolution digital cameras, researchers at Stanford University in California have developed ways to use this positioning data not just to assign a location but also to annotate digital photos with a wealth of extra detail.
Software developed by Andreas Paepcke, Mor Naaman and colleagues can turn the location and time of a GPS-labelled photo into details such as the city, country, weather, season and temperature, whenever and wherever the picture was taken. It finds the data by simply searching the web with the photo鈥檚 time and positioning coordinates.
Naaman says people often relate better to their photographs when they have such extra information. People may recall it was absolutely freezing cold, but not what month a snap was taken in, he says.
