Read more: Engines of the future: What’s next in internet search?
Standard searches can’t keep up with the latest news and tweets, so engines have to get smarter about where they look – and maybe ask your friends too
Few of us scan much beyond the first 10 results turned up by a search engine, so it is a huge task to rank and present them in a way we can cope with. “It’s easy to overwhelm people with too much information,” says Danny Sullivan, a web search expert at Calafia Consulting based in the UK.
Advertisement
What’s more, billions of new pages are added to the web every day, and surfers expect search engines to be up to speed with 24-hour newsfeeds as well as the latest posts and status updates from sites such as Twitter and Facebook. Yet web crawlers can take hours to work through the changes, meaning juicy scraps of gossip may be old news by the time a search engine tells you about them.
So search engines have had to up their game by delivering real-time results. This takes more than just powerful computers and faster software. Engineers are exploring strategies such as indexing the content of links inside tweets and spotting newsworthy events through the sudden prevalence of posts transmitted from smartphones at specific locations or even the occurrence of certain terms (for example, “earthquake”) in new content. Then there is the need for a reputation system that gives priority to posts from bloggers with the most followers or from Facebook users with the most friends.
Others hope to crack the problem by turning search on its head. They think surfers should be part of the solution. Start-ups and , for example, are betting on the idea of decentralising the search engine. The idea is to convince users to download file-sharing software which tracks their browsing habits. The index this creates is stored on the user’s computers rather than in a data centre and can be accessed by other users. This should make searching the web far quicker, they suggest, and could serve as an alternative to conventional page ranking, as how often a page is visited should provide a good indicator of how relevant its contents are. Keeping track of someone’s surfing habits in this way creates a “democratised, user-centric ranking” for search results, according to Faroo’s founder, Wolf Garbe.
There is another way to harness the collective wisdom of online networks: using what Sullivan terms “help engines”, such as . Suppose you are headed to New York and want a good place to stay. Why not ask people you know and trust?
Pose the question on Aardvark, and the site asks those of your Facebook friends or friends of friends who are most likely to know the answer. To decide who to choose, Aardvark looks at past blog posts, online profiles and tweets to identify users with . According to Aardvark’s co-founder Damon Horowitz, 60 per cent of questions submitted on the site are answered within 10 minutes.
Using humans to improve web search isn’t new: in the 1990s, search engines used armies of editors to categorise sites. Now the idea could be back to stay. In February, Aardvark was bought by Google, and in July Facebook launched a trial of , called Questions, which allows people to put queries to the site’s 500 million users.
What everyone would like, though, is a semantic search engine – one which emulates the way humans understand the intent of a query and so can deliver highly accurate results. Semantic search is an ongoing development in the underlying structure of the web. Progress has been slow, however, in part because it relies on programmers and users to attach extra, computer-readable information to every page. But in July Google bought Metaweb, owner of a large database with entries that are tagged using machine-readable code, so it seems Google hopes to make its own searches smarter. That, and recent changes to Twitter and Facebook which allow users to annotate their own posts (żěè¶ĚĘÓƵ, 31 July, p 20), suggests the day of semantic search is drawing closer.
“Eventually a search engine will be able to decide for itself what information it gives you. You won’t even need to type in a query”
Getting Personal
We are used to seeing sponsored links, paid for by advertisers, alongside our search results. These are targeted: search engines keep track of the most popular links associated with every search term and display these accordingly. Google can also keep track of your web search history to provide results it thinks you will like. Now Bing has gone one step further by results based on the likes of your Facebook friends. Eventually, , given your web search history, your location plus a few other details, a search engine will be able to decide for itself what information to give you. You won’t even need to type in a query.
