THE second wave is about to break. Several infant companies that owe their
existence to the Internet are readying themselves to go public. Their aim is to
repeat the success of Netscape, which put itself on the market last year and is
now estimated to be worth $4 billion. The launch left 24-year-old Marc
Andreessen, one of the company鈥檚 founders, worth more than $100 million.
As an undergraduate he had developed the program that forms the basis of
Netscape鈥檚 software for browsing the World Wide Web.
Financial pundits believe the next Netscape will come from the ranks of a new
class of Internet company. Where Netscape and its rivals such as Microsoft gave
people the basic vehicles for travelling the Internet, the new operations create
personalised guides so that people can visit sites of their own choosing. The
idea is simple: just type in a word or phrase and out comes a list of sites
around the world that mention it. But the reality is far more complex. The dozen
or so companies that offer search services, such as Lycos, Infoseek and Open
Text, employ search engines鈥攑rograms that scour the Net, record everything
they find, and salt it away in gigantic databases.
That the new companies are hungry for commercial success is beyond question
says Jim Bair, a California-based technology analyst with the research company
Gartner Group. Some, such as Lycos; Excite and Yahoo, have already taken the
first steps towards public ownership. But which of them will be first to take
the plunge is still unclear, Bair says. He likens the situation to a coachload
of executives at a casino, all holding back to see who will make straight for
the roulette wheel and who will have dinner first.
Advertisement
The pages that these companies post on the Web are fast becoming the main
gateways to the Internet. This assures them of huge numbers of visitors
and鈥攁t least for the present鈥攈igh incomes from selling advertising
space on their sites. But the Net is not going to be the biggest source of
revenue in future. Business analysts reckon that the juiciest profits by far
will come from 鈥渋ntranets鈥� 鈥攕mall versions of the Internet set up by
companies to improve the way they share information internally. Searching the
Net may be free, but companies that want to delve into their own archives will
have to pay for their search engine software.
Today鈥檚 search engines are not the first attempts at indexing the Net.
Gopher, for example, creates hierarchical menus that impose a fairly primitive
structure on the data stored at individual sites, while Veronica lets you search
Gopher menus around the world. Another early tool called Wide Area Information
Servers, known as WAIS, allows you to search a small number of specialised
databases for words and phrases. The problem with WAIS is that somebody has to
index the information by hand.
The new search engines are much more ambitious. They aim to allow users to
explore the graphically displayed information on the Web. Initially
organisations such as Yahoo (http://www. yahoo.com/) attempted to compile
conventional catalogues of sites on the Web by hand. While such lists are useful
as a way to find starting points for further searches of the Net, they do suffer
problems. First, the Web does not consist of neat chunks of linear text and the
information it contains is not arranged in any sort of system or hierarchy.
Embedded within its pages are the addresses of other sites that hold text,
images, audio and video. This creates a haphazard tapestry of links that makes
creating a simple list very difficult.
Tireless agents
Worse still, the Web is growing in an erratic way and at a rate that is
certain to doom any attempt to catalogue it using conventional procedures. The
solution is to employ programs that index the Web according to predetermined
rules. These simple 鈥渟oftware agents鈥� never tire of their tasks or forget where
they are (鈥淪oftware鈥檚 special agents鈥�, 快猫短视频, 9 April 1994, p
19).
The first search engines such as Lycos (http://www.lycos.com/), WebCrawler
(http://www.webcrawler.com/) and the WWW Worm (http://wwww.cs.
colorado.edu/wwww.html) began as academic research projects. Their creators were
interested as much in understanding how to build software agents that could
search the Net as in the results they produced. But the indexes of information
that they created as a by-product proved so useful that sites built round them
became among the most popular places online. It is the indexes created by search
engines that people search when they type in their key words. Alongside this
generation of search engines, many new ones have been set up as commercial
ventures from the start, including Open Text (http:// www. opentext.com/),
Excite (http:// www . excite.com/) and Infoseek (http://www. infoseek.com/).
In principle, the task these search engines face is not difficult. As an
analogy, imagine that telephone directories did not exist and that you wanted to
create one. Your starting point would probably be the numbers of all your
friends and colleagues. You would ring round those people and ask for their
lists, adding any new numbers to your own. Then you would phone those numbers
and repeat the cycle.
Spiders, crawlers and worms
The search engines鈥攌nown as crawlers, worms, spiders and
robots鈥攇o about their work in similar fashion. First, they access the Web
addresses their creators know. From these sites they find links to other pages
via embedded addresses. Indexing software then compiles a database of all the
new addresses and links, and also keeps a note of text references. As time goes
by, the engines are sent on new foraging missions to find new sites, or updates
of existing sites.
Searching large areas of the Net is a mammoth undertaking. But it is not
inconceivable with spiders such as the one used by Digital鈥檚 Alta Vista
(http://altavista.digital.com/), which can scan 1000 sites at the same time.
While the first search engines stored only subject titles and perhaps a few
keywords from documents they visited, recent entrants into the field, such as
Open Text, Excite, Inktomi (http://www.inktomi.com/) and Alta Vista index the
full text of every page they visit.
鈥淲e index every word, plus some information such as whether the word appears
in a title [or] a link,鈥� says Louis Monier, lead scientist at Alta Vista, which
has the largest store of pages at present. Its database holds around 22 million
pages containing 10 billion words in its 33-gigabyte index. All this information
sits on one of Digital鈥檚 most powerful computers, which has 10 main processors,
6 gigabytes of memory and 210 gigabytes of hard disc storage. At present, Alta
Vista can index around 2.5 million pages a day, while a new incarnation of the
Inktomi search engine will be able to index 10 million a day.
Valuable though these search engines are, the quality of information they
hand out leaves much to be desired. At the time of writing, for example, a
search for 鈥淥lympics鈥� on Open Text produced a list of 3235 hits. One of the most
pressing tasks facing the designers of search engines is to give some shape to
these long lists of results, by placing them in some order of relevance for the
user. 鈥淲hen users search for something such as golf, getting hundreds or
thousands of hits is not useful,鈥� says Zara Haimo of Infoseek. 鈥淢ost users don鈥檛
look past the first 10 or 20 hits and will get very frustrated if the good golf
pages are buried behind thousands of less interesting pages that just happen to
mention golf.鈥�
Search engines need to be smarter, says Tim Bray, a cofounder of Open Text.
They need to 鈥渋ntuit what the pages are about as opposed to what they say鈥�, he
adds. 鈥淲e pursue incremental improvements in the areas of responsiveness, search
focus and usability, [but] we see the major gains coming in the area of
context.鈥� However, these gains are still some way off. 鈥淭here is no technology
on offer or even on the horizon that does a good job with this,鈥� says Bray.
The other deficiency with search engines is that they do not search the
entire Net. The rough consensus among those running these services is that there
are 100 000 sites holding 50 million Web pages, and this number is doubling
every nine months. By these figures, even Alta Vista has less than half the
available pages indexed.
A number of obstacles stand in the way of achieving the goal of complete
coverage. Some pages from locations that have already been searched are excluded
because they have been listed in something called the robot exclusion file held
at each site. This simple document tells well-mannered search engines where not
to go. This might be because the site owner does not want certain pages indexed,
or because there are good technical reasons for not trying to add them to an
index. Why search for pages that will not exist for very long, for example?
Many Web pages are created on the fly, in response to users鈥� requests, and so
exist only transiently. These 鈥渄ynamic鈥� pages present the search companies with
a real headache. 鈥淚f you are asking if anyone can ever index every page on the
Web, it should be obvious that this would both make no sense and be impossible
since they could never get all the dynamic pages,鈥� says Haimo.
Then there is the problem of how to index multimedia objects, such as images,
audio or video, which are becoming important elements of Web pages. Monier sees
no immediate prospect of indexing graphics, and before anyone tackles audio or
video, he reckons that most work will go into the increasing number of Web pages
not written in the Roman character set. Search companies will probably first
begin indexing pages written in Chinese, Japanese and Arabic characters.
Finally, there are new kinds of Internet object that cannot be captured by
existing search engines. One type is files written in VRML, the virtual reality
modelling language used to define three-dimensional worlds. Another is Java
applets鈥攕mall programs that are sent with a Web page to add extra features
that, for example, make graphics flash or run short video clips. Digital
announced last month that it now has a method for finding and retrieving these
programs. The other search companies will probably not be far behind.
The competition between search engines is fierce, not least because Net users
are quick to switch when a better information mousetrap comes along. Whoever
supplies the most popular approach will soon establish themselves as the top
search site. Moreover, big money is at stake. Infoseek claims that with 25
million visits a day it is the second most popular site after Netscape
Communications, which receives more than 45 million visits a day. Such heavy
traffic means that the potential for attracting advertising鈥攁nd becoming
the next Netscape鈥攊s high.
But advertising may not always be so lucrative, says Clay Ryder, an industry
analyst with Zona Research in Redwood City, California. At present search
companies can earn in the region of $20 000 a month for a single
advertising slot. But there are no demographic data on who is accessing these
sites, says Ryder, and one person visiting a site 20 times looks the same as 20
people logging on once. 鈥淭here鈥檚 no proof yet that people read the ads or take
any notice of them,鈥� he says.
Research by Zona suggests that this year the market for companies
specialising in information retrieval from the Net is worth just over $80
million, while intranets will probably yield around $67 million. But by
1998, says Ryder, information retrieval companies will stand to make nearly
$1.2 billion from intranets鈥攏early four times as much as they could
make from the Net.
Fast-moving train
Intranets give staff access to company information in an alternative form to
conventional e-mail, company circulars, phone directories and so on. They
exploit the same standards and software that are used on the Net: server
software and a browser such as Netscape Navigator or Microsoft鈥檚 Internet
Explorer, and a search engine.
Ford Motor Company鈥檚 intranet grew from the needs of technical and
engineering staff to shuffle documents鈥攃omplete with graphics鈥攁round
the world. It now stretches to about 550 sites in more than 30 countries and is
growing fast. Only about 50 of these sites are official; the rest are the home
pages of offices or individuals. Ford puts out news bulletins and glossy
marketing programmes on its intranet and, though its search facilities are
relatively primitive so far, the phone number of any one of Ford鈥檚 60 000
employees is immediately available at the click of a mouse button. Bob Kramek,
who helped to set up the system, says its main benefit has been the removal of
geographical boundaries. 鈥淭his is a train that is moving very fast,鈥� he
says.
For others, the real advantage of intranets is opening up company information
held in documents such as minutes of meetings and departmental reports. Just as
search engines on the Net allow you to winkle out information from obscure
areas, so too can search engines on intranets.
Zona Research does not just keep an eye on the market for intranets鈥� it
also has one of its own. Zona is a small company with only 10 staff, based at a
single location. But it still has several Web sites on its intranet. 鈥淚 have
access to the collaborative efforts of my colleagues and can easily share
information,鈥� says Ryder. Every night, the search engine sifts through the files
on Zona鈥檚 network looking for additions, and remakes the company鈥檚 database. 鈥淚f
we add any new information on one day, by the next day you can search for it,鈥�
he says.
Expensive tastes
Search engines are essential for locating internal information, says Ryder
with his analyst鈥檚 hat back on. Although there will be a market for search
engines as add-on products for companies with intranets, Zona expects software
companies to form alliances that will offer a suite of intranet tools including
a search engine. In the long term, Zona predicts, the search and retrieval
companies may find themselves swallowed up by bigger fish.
But, in the short term, this is unlikely to happen. All the signs are that
the companies waiting to go public, such as Excite, Yahoo and Lycos, will follow
the pattern laid down by Netscape: they will be worth too much for even the
biggest fish to afford.
* * *
Little and Large
YOUNG is an apt word for Lycos. Michael Mauldin only developed the Lycos
spider two years ago, while carrying out research into information retrieval at
Carnegie Mellon University in Pittsburgh. In 1995, his idea caught the eye of
CMG@Ventures, a venture capital company that focuses on the Internet. It put up
the money to found Lycos as a company and bought an exclusive licence for the
spider technology.
One of its first Internet products is a2z, which lists the Web鈥檚 most popular
sites under specific subject headings. Within these headings it is possible to
search for more specific subject areas using the spider. Technology analyst Jim
Bair of the Gartner Group puts Lycos in the group of companies waiting to launch
themselves on the open market. Four months ago, Lycos took the first tentative
steps towards issuing shares. Because of this, nobody in the company is allowed
to talk about its future plans.
If Lycos is new, Digital is one of the oldest names in the computer business.
It was founded in 1957 and has more than 61 000 employees around the world. But
Digital鈥檚 search engine, Alta Vista, is one of the most recent to hit the Net.
It went live to World Wide Web users on 15 December 1995, and is the most
powerful in existence.
Louis Monier, lead scientist at Digital鈥檚 Palo Alto research laboratory, and
a colleague who prefers anonymity, came up with the idea to build a search
engine while chatting in the corridor last summer. It had nothing to do with the
area of research they were pursuing at that time. By the autumn they had a
prototype spider ready for testing, and Digital gave Monier permission to let
the spider loose on the company鈥檚 internal networks.
When it worked, Monier and his team were granted the use of one of the
company鈥檚 fastest computers to support the system. Various colleagues
contributed code and advice to the project, while the indexing software had
already been developed by other Digital researchers. Alta Vista now handles more
than 2 million requests for information a day. 鈥淭he average query today on our
33-gigabyte index is answered in less than half a second,鈥� says Monier.
Bair is not confident that a big, established company such as Digital will be
able to move fast enough to keep pace with the start-ups. Monier disagrees. He
says that working for a large company has helped him get the project off the
ground quickly. For the future, Monier sees indexing pages written in Chinese,
Japanese and Korean as an important growth area. Five years down the line, as
voice recognition improves, it may also be possible to index audio. As for
indexing graphics and video clips or developing truly personalised search
engines that know people鈥檚 preferences鈥攖hese are still firmly in the realm
of science fiction, he says.
Claire Neesham