Read more: Engines of the future: What鈥檚 next in internet search?
The web鈥檚 top search engine has got rich from cleverly picking the right few pages out the trillions online. What kind of upstart could usurp its crown?
Google is synonymous with web search. Some 60 per cent of surfers use it, performing roughly . All these clicks combined to earn the company a cool $6.5 billion profit last year. But if you dream of muscling in on some of this fortune you should be aware of what you are letting yourself in for.
Advertisement
Building a search engine is like compiling a searchable index for a . The index has to be contained on a limited number of servers, and each of its entries must be stored in such a way that the whole thing can be searched in milliseconds. Then you must face the fact that the web鈥檚 contents constantly change and that most searches are frustratingly vague 鈥� just three words long, on average.
We might be vague but we are also demanding: we want our first 10 results to include the information or site we are looking for. What鈥檚 more, we want these results to appear near-instantaneously and, of course, for free. Still want to take on Google?
To understand why Google is so dominant, you need to grasp the basics of web indexing. Search engines use 鈥渨eb crawler鈥� software to trawl through online content, remembering not just the words but also how often they crop up. They also make use of something a book doesn鈥檛 have: hyperlinks. The degree to which web pages cross-reference each other with hyperlinks provides a measure of each page鈥檚 importance and relevance. This provides a way to structure the sprawl of the web and rank the pages, and was that set Google on its path to World Wide Web domination.
Another trick lies in exploiting anchor text 鈥� the text associated with a hyperlink. Anchor text gives an indication of the content of the page the hyperlink points to, so it can be hoovered up by a search engine as being likely to contain keywords that a surfer might use to find that page.
These innovations, along with many small refinements, have advanced web search to the point where it works perhaps as well as we could hope for. In fact, search engines are now so good it feels as if progress has tailed off, says Monika Henzinger, a Google research director: 鈥淭he low-hanging fruit has certainly been picked.鈥� Yet there is plenty left to do, she says.
New kinds of search engines can, for example, handle queries posed in ordinary language, so you can ask them questions as you might another human, or perform sophisticated image searching and matching. Others improve results by amalgamating information from different online resources, including databases, multimedia documents and social networks. Right now these novel search engines account for just 2 per cent or so of queries. Google needn鈥檛 worry just yet.
