BACK in the early days of the World Wide Web, users could consult a directory of servers personally compiled by its creator, Tim Berners-Lee, to help them find their way around. A snapshot of this directory from 1992, provided by CERN, shows people used it to link to everything from physics journal abstracts to science fiction reviews and song lyrics.
As the web exploded in popularity, personal recommendations could not keep up with the growth in new sites. To help people navigate this expanding resource, commercial search engines like AltaVista, Yahoo and ultimately Google sprang up. These used software such as Google鈥檚 famous PageRank algorithms to estimate the popularity of websites based on the number of times they were linked to from other sites, consigning personal recommendations to history.
Software has limitations, though. While it can determine how popular a site is, it cannot match a human鈥檚 ability to judge the quality of information it contains. So inspired by the success of 鈥渃rowd-sourced鈥� websites such as Wikipedia, Digg and Delicious, search engines are again looking to the human touch to improve their results.
Advertisement
Wikia Search, launched last year, for example, allows any user to annotate a set of search results and adjust their ranking, while Anoox allows its users to vote for the most interesting web page from the results.
Critics of these search tools argue that cunning marketers could use them to raise the ranking of their own sites and edit out their competitors. What鈥檚 more, such search engines still only produce generic results that may not be relevant to an individual user, says Kevin Ryan from Search Engine Watch in New York. 鈥淵our ideal results for a query might be very different to what I鈥檓 interested in,鈥� he says.
Their results could also be adversely affected by the expertise of their users, he says, or lack of it. 鈥淵ou really need millions of people adding accurate information to the sites,鈥� he says. 鈥淏ut if lots of stupid people contribute, it鈥檚 really the antithesis to the wisdom of crowds.鈥�
To solve these problems, a new generation of collaborative search engines is being developed. They will use previous searches carried out by the user鈥檚 trusted circle of friends on networks such as Facebook, Windows Live Messenger or Skype to tailor their results. The idea is that since we share interests with our friends and colleagues, the sites they visit should be more relevant to us than those visited by strangers. This should provide a more personal response that is also free from the influence of spammers and marketers.
Some of these collaborative search engines are designed to help groups carry out web searches more effectively, by pointing friends or colleagues towards the websites most relevant to their group. Others, dubbed social search engines, aim to alter the ranking of your search results based on what your friends are interested in.
Collaborative tools would be particularly useful for students or colleagues working in a research group, says Meredith Morris from Microsoft Research in Redmond, Washington. 鈥淎t the moment it鈥檚 very difficult for research groups to divide their work in an efficient manner,鈥� she says.
To address this problem, Morris has developed a Windows Live Messenger tool called SearchTogether, which helps users to share and delegate internet research projects. When embarking on a project, users invite their colleagues to trawl the web together. Each user鈥檚 search history is viewable to the other members of the group, preventing people wasting time on duplicate searches, and encouraging them to modify the search terms to produce different results. If a team member finds a website that is so useful they want everyone to see it, they can leave a rating or comment on a link.
SearchTogether, which Morris will present in November at the Computer Supported Cooperative Work conference in San Diego, California, also automatically divides the results of a search and delegates different links to the various members of the group to follow up. The software analyses each user鈥檚 web history and allocates those sites with the most similar content, helping to direct people to pages that best match their area of expertise. So for example, a research team developing insect-like flapping-wing aircraft is likely to contain both aeronautical engineers and biologists. The websites relevant to them should be clear from their previous searches, and the software divides the research work accordingly.
Future versions of the software may use the group鈥檚 previous search activity to suggest other relevant sites to view. Morris is developing a feature that analyses recent activity, then alters the ranking of new searches so that websites with the most similar content are placed at the top.
The PeerSpective tool, developed by Alan Mislove at the Max Planck Institute for Software Systems in Saarbr眉cken, Germany, also tailors search results by analysing your network of friends (快猫短视频, 13 December 2006, p 29). Instead of searching the entire internet, the software trawls the web histories of network members to find relevant sites that other users have visited.
That still means trawling through everyone鈥檚 search history, however, as the software cannot differentiate between the various groups within its network that its members belong to, such as families or sports teams. 鈥淭he results become diluted so they just approximate Google,鈥� says Mislove. 鈥淒epending on the search, results from my football team, family members or colleagues might be more relevant.鈥�
To tackle this, the team has developed software to identify the social groups that lie within a network. By analysing each member鈥檚 list of contacts on the internet telephony service Skype, for example, the software quickly develops a map of connections between them. Groups such as football teams or colleagues tend to be clustered tightly together with a large number of links between each of the members and just a few people linking different groups. By cutting the links between clusters, the software is able to break down the network into its constituent social groups. The team tested the software on a group of Facebook members who all belonged to the same university and it successfully separated students into their year groups and halls of residence.
So when a user searches for a term, the software scans through their friends鈥� web histories and returns a separate list of results for each social group. The results are then ranked according to how many of the user鈥檚 friends visited the page, as well as by standard search algorithms that analyse the extent to which the content of a page matches the query.
The team are testing the software on a group of Skype contacts from the institute. 鈥淭here鈥檚 no reason why it wouldn鈥檛 work with any other social network,鈥� says Mislove. A test version of the updated PeerSpective software should be available in November.
So could these social and collaborative tools one day challenge existing search giants? Mislove admits that PeerSpective will never index as many pages as Google, but he says it could reach some areas of the web that Google cannot. For example, if a group consists of people who all work for the same company, the search engine could access pages from its intranet site that would be hidden to Google behind a firewall. It could also highlight pages that are only relevant to a small community and therefore might be drowned out in Google鈥檚 15-billion-document index, or recently created pages that Google鈥檚 crawlers haven鈥檛 yet picked up.
鈥淧eerSpective could highlight pages drowned out in Google鈥檚 15-billion-document index鈥�
Ryan remains sceptical that social tools will pose a serious challenge to the existing search engines. 鈥淚 don鈥檛 think any of the large search engines will be unseated,鈥� he says. But he believes sites like Yahoo and Google might add some social tools to future versions of their search engines to help improve their results.
This is certain to happen, says Morris. 鈥淪ocialising and searching are the two most common activities on the internet. It makes sense that the two worlds are now blurring.鈥�
