The Speechbot team took public domain speech recognition software and tuned
it to cope better with the low-quality audio available on the Net. Sound
files—such as those in RealAudio format or Microsoft’s Media
Player—are played into Sphinx and the resulting rough transcript is
indexed using a conventional search engine.

The results are not accurate enough for reading, but according to Compaq’s
business development manager Ron Gentile, repeated keywords allow effective
indexing of the file’s content. Speechbot can also match the words in a
transcript to the times they were said in a given clip so you don’t have to
listen to a three-hour speech to get to the bit you want. Compaq has yet to
decide how it will expand and commercialise the service, but some commentators
are already impressed. “Other attempts to search multimedia files rely on closed
caption information and other indexing material. This is the first system I have
heard of which doesn’t do that,” says Danny Sullivan, editor of the Search
Engine Watch website. “This new approach sounds very exciting.” The software has
been indexing selected sites from the US media every day since March—try
it out at speechbot. research.compaq.com.

Altogether the Terra-Server database will contain images of seven million
square kilometres, compressed to 1500 gigabytes of data, drawn from the US
Geological Survey and the Russian Space Agency. People will be able to search
through the database and view the images as long as their software supports
Java. The aim of the project is to show that Microsoft’s “Back Office” software
can handle huge amounts of data, and to provide a shop window for the USGS and
Aerial Images, the company selling the Russian images.

The highest resolution images that will be offered of the US are at a
resolution of one metre—each pixel represents a square metre—while
the Soviet data that covers cities in the rest of the world is at 1.6-metre
resolution. All of the images will initially be in black and white, and were
taken over the past ten years. Users will be able to view all images, but will
not be entitled to use them commercially or place them on Web pages without
paying.

Prices have not been set, but Terra-Server’s designers at Microsoft say: “You
should be able to buy a detailed image of your neighbourhood for a few
»ĺ´Ç±ô±ô˛ą°ů˛ő.”

These images could also have military value. Modern spy satellites can offer
considerably higher resolutions, but John Pike, an analyst with the Federation
of American żěè¶ĚĘÓƵs says: “For most military purposes this resolution is more
than adequate —you don’t get much more information about buildings at
resolutions better than a metre.”

The FAS’s Public Eye project shows non-profit organisations how to
use new information gathering tools, including Internet searches and satellite
images. Its first project is a guide to the extent of nuclear proliferation.
“There are about 20 places that I want to take a look at in Iran
alone—several hundred around the world,” says Pike. But the people
peddling satellite images sell mainly to large businesses and governments, so
they tend to charge thousands of dollars for data covering thousands of square
kilometres. This makes it difficult for the FAS to afford the images.
Terra-Server should bring this kind of information into the price range of such
organisations, which could use it to locate everything from toxic waste dumps to
refugee camps.

Jim Gray, a senior researcher at Microsoft, says there will
initially be some gaps in the database. For example, the Russians have provided
very few images of Australia, and none of Russia itself.

To address security concerns, Microsoft has also put a system in place that
will allow them to fence off certain areas on request from national governments.
“Microsoft is not out to pick a fight with governments,” says Gray. Ordinary
citizens or corporations that object to having pictures of their homes or
offices available for all to see may not find it as easy to keep prying eyes
away. “They will be able to ask, but they might have to pay to be
removed—here in the US, you have to pay to have an unlisted telephone
number,” says Gray.

Viewed through a magnifying lens from a distance of 3 centimetres, the
display appears as large as a conventional 25-centimetre computer display. The
lens and screen unit fits on the end of a mobile phone, making it possible to
surf while you stroll.

The CyberDisplay was designed by the Kopin Corporation of Taunton,
Massachusetts. The black-and-white screen, announced in May this year, has a
resolution of 320 by 200 pixels.

The version which Motorola and Siemens are planning to use has a colour
display, and by the end of the year may have a slightly larger 9-millimetre
screen with 640 by 480 pixels—the same resolution as the Video Graphics
Array standard widely used in PCs for many years.

The tiny screen is based on the same active matrix technology found in
conventional laptop screens. But because it is so much smaller, it uses only 20
milliwatts—a mere 1 per cent of the power consumed by a typical laptop
screen—one of the most power-hungry parts of larger computers.

Of the 200 000 or so commercial-as opposed to academic or personal-Web sites that are accessible via the Internet, up to 6000 are sex-related. Hundreds more advocate drug use, deny that the Holocaust occurred or provide information on how to build bombs. These sites are every parent’s nightmare, for short of monitoring their children as they browse, they are virtually powerless to block them out.

Some countries, including Germany and the US, have tried to control access to objectionable sites by removing them entirely from the Net. But they have had little success. The Communications Decency Act in the US, which would have made it illegal to publish “indecent” material online, was ruled unconstitutional in June. And last year, when the German government pressured Net providers in Germany to block access to a site holding Radikal, a banned left-wing magazine, it was copied by activists to more than 50 sites across the world.

Governments have now stopped demanding outright bans on controversial sites. When ministers of 29 European countries met in Bonn in July at the Global Information Networks conference, they agreed to push for the adoption of filtering mechanisms and rating systems that would enable parents, libraries and others to bar access to certain sites. “The computer industry is developing a whole toolbox full of technologies that can do for the Internet what the V-chip will do for television,” President Clinton told a Net summit in Washington DC in July.

But Clinton may have been wildly optimistic. Many Web technicians doubt whether any technology is capable of providing even a cursory system of censorship for the whole Net. “Blocking software is a huge step forward in solving the dilemma of sexually explicit speech, but it does come at a cost,” says Jonathan Weinberg, who studies communications law at Wayne State University Law School in Detroit, Michigan. A typical problem is that “people will miss out on worthwhile speech, through deliberate exclusion, through inaccuracies in labelling inherent to the filtering process and through the restriction of unrated sites”.

The most widely used filtering technology is the Platform for Internet Content Selection (PICS), run by the World Wide Web Consortium (W3C), which coordinates the Net’s technology advances and policy. PICS can block a user’s access to Web pages according to certain criteria, such as the degree of nudity or the amount of bad language. These criteria can be defined by any organisation and individual sites can be rated either by third parties or, more commonly, by the sites themselves. The ratings are embedded in the pages. So far, however, only Microsoft’s Internet Explorer Web browser-used by less than one third of Web users-uses PICS.

PICS is more effective than the various censorship software programmes that are available (Review, 6 September, p 46). It works with a ratings system produced by the Recreational Software Advisory Council on the Internet, or RSACi, a nonprofit censorship group in the US. The key to its success is the number of sites it rates. If most are unrated, parents must either limit their children to that small section of the Net that has been rated, or leave them free to browse at will and risk seeing something they should not.

Steve Balkam, head of RSACi, says that over 42 000 sites have already rated themselves using its system. He claims that 5000 sites are being rated each month, and that the council’s objective is to have 100 000 rated by September next year. But even this is a small proportion of the half a million sites accessible on the Web, a number that is forecast to double within the year.

RSACi’s rating system concentrates on four main areas-nudity, sexual content, violence and language, including racist “hate speech”. The organisation concedes, however, that its system does not cover all contentious subjects. “We’ve been offered at least 13 different categories, and there is no end to the number you could come up with,” says Balkam. RSACi’s decision to include hate speech came in response to European concerns, and it is aware that Australian authorities are particularly concerned about material that encourages or enables suicide.

Sites can rate themselves by answering questions about their content on RSACi’s Web site, which then issues them with a rating between 0 and 4 for the four categories. The ratings for language, for example, range from no profane language (0) and “mild expletives” (1) to “crude, vulgar language or extreme hate speech” (4). Anyone using the RSACi system can set their browser to the maximum rating they consider acceptable for each category.

A major problem is that RSACi’s ratings do not take into account the context in which pictures and words appear-a clearly pornographic photograph of a naked woman, Botticelli’s Birth of Venus and pictures of naked aboriginals from a documentary programme are rated in the same way. To satisfy news organisations, RSACi recently proposed a “news” rating that would override other ratings, but the Internet Content Coalition, a group of leading news organisations in the US, last month refused to back the plan (This Week, 13 September, p 5).

Because of the sheer number of sites and the speed with which they change, the effectiveness of RASCi depends on the willingness of site owners to rate themselves accurately. Balkam says that in the one and a half years since RSACi started, only two sites have wrongly rated themselves. However, the organisation checks only 20 per cent of ratings. Otherwise, it checks when it receives a complaint.

Most of the controversial self-rated sites are sex-related, and they are generally happy to keep away under-18s in order to forestall government regulation. But when it comes to restricting access to drugs information or hate speech, sites are much less willing to regulate themselves. żěè¶ĚĘÓƵ contacted one site, Hyperreal, which offers information about ecstasy and other drugs, and asked about self-regulation. Brian Behlendorf from Hyperreal replied: “We don’t plan to rate our site in the future. Part of our mission is based on the notion of freedom of information, and we regularly receive feedback in great praise of the potentially controversial content in certain parts of our site.”

At the end of this month organisations from around the world, including RSACi, the Australian Broadcasting Authority and a coalition of European groups, will meet to try to hammer out a new global standard for PICS ratings. Meanwhile, the W3C is working on a new censorship tool called RDF that would provide more detailed descriptions of the contents of sites and pages.

The Web is not the only place on the Net where a child could encounter unsuitable material. Usenet is a set of over 43 000 discussion groups where Net users can publish pictures or messages anonymously. In Britain, Turnpike Software has produced an extension to PICS that allows Internet service providers to rate Usenet groups based on the typical content of their discussions. But nobody is using it. This situation is typical of the problem facing those who wish to censor material on the Net: it is one thing to develop the technology, quite another to apply it effectively.

If a call comes while your line is engaged, the exchange automatically checks
to see if you are calling a recognised Internet provider. If so, it will notify
your computer that there is an incoming call and give you the opportunity to
answer via the Internet. Alternatively, you can stop surfing and take the call.
Telstra plans to offer the service to the public from November.

So what are parents, teachers and librarians to do? In the best of all
possible worlds, they would surf with the child, discussing anything disturbing
or controversial. This takes time. Instead, governments and the Internet
industry encourage parents to rely on censorship software.

The most restrictive of these programs monitor what users type or try to
access, and block specified sites. There are two problems with this approach:
they don’t cover enough, and they cover too much. Net Nanny’s “banned lists”,
for example, claim 4500 Web addresses (some are duplicates). There are more than
a million Web sites and more than 100 million pages.

You can set up Net Nanny (http://www.netnanny.com/, $39.95) to
lock users out of a computer if they attempt to visit a site on its banned list,
or if they type certain keywords (“porn” or “sex”). This may deter a curious
teenager trying obvious routes, but it wouldn’t take much searching (or talking
to mates, online and off) to find pornography that hasn’t been blocked.
Moreover, Net Nanny has concentrated largely on blocking sexual content. Plenty
of hate literature, for example, slips through its net.

Cybersitter (http://www.pow-dist.co.uk/, ÂŁ34.95) is not very
good at blocking access, but it can be set up to record what sites were visited
and whether keywords were found on the browsed pages. CyberPatrol
(http://www.microsys.com/, £24.95) claims to have 4 million “Internet
resources” in its filtering database. It blocked 19 of the 20 sites I tried, a
better success rate than the other two. It offers more control—you could
choose to block hate literature but not pornography, for example. However, once
a child has discovered an unblocked site, they can often follow links to more of
a similar character, and no one can tell afterwards whether they spent the
evening reading żěè¶ĚĘÓƵ online or trying to get at porn.

Net Nanny and some similar programs block access to a number of gay resources
and, notoriously, to the US National Organization for Women’s web site at
http://www.now.org. Cybersitter blocked access to a gay magazine,
http://www.planetout.com/, even though I had set it not to block “gay and
lesbian activities”. As far as I could determine, CyberPatrol didn’t block any
more sites than it was told to. It won’t allow you to search for “Essex” or
“asexual”, though.

By far the majority of sites on any of these lists are clearly unsuitable for
children. You should still be cautious, though, about handing over control over
what your child or student can see to an unaccountable commercial organisation.
There is a Web site about this issue at
http://cgi. pathfinder.com/netly/spoofcentral/censored/.

The coming thing is definitely PICS: Microsoft and Netscape are building this
“ratings” system into their Web browser programs. Designed by the World Wide Web
Consortium, in principle PICS offers a flexible and open way of labelling pages
with a variety of “attributes” and controlling access. It won’t be something you
can rely on, however, until a very large numbers of pages has been “rated”.

Commercial porn merchants are happy to rate themselves (already excluding
kids, by demanding credit-card payments for access). Those with political,
religious or plain strange axes to grind are another matter. This leaves private
companies or governments to rate their sites, again begging the question: “Who
watches the watchers?”

Last but not least is a simple, cheap and effective—but
partial—solution. Surf Spy
(http://ourworld.compuserve.com/homepages/esmsoftware/sspy.htm)
and Net Snitch (http://www. netsnitch.com/) are shareware that keep logs of the titles and
addresses of all Web pages viewed. They should at least give teenagers pause for
thought.

And I wouldn’t worry too much about younger children stumbling on this stuff
accidentally: it’s out there, but you do have to look for it. In the end, the
only way to get 100 per cent protection from controversial material on the Net
is to unplug your computer.

A Solihull-based company, Scorpion, is sending unsolicited e-mails offering
50 000 British e-mail addresses, along with the software needed to send mass
mailings, for ÂŁ299. It has not registered with the Office of Data
Protection (ODP). Companies that hold computer data about people have to
register with the ODP, and assistant registrar Philip Jones says it could be
prosecuted for not doing so. The maximum fine for failing to register is
ÂŁ5000. Scorpion says the addresses are already publicly available and are
gathered from Web pages, trade directories and newsgroups. There is nothing
illegal about bulk e-mailing, it says.

The ODP says it is not clear whether people who post e-mails to newsgroups
have any presumed right of privacy. “It’s trivially easy to collect UK addresses
from news groups, and the spam that results is rising exponentially,” says Steve
Harris, the British author of Spam Hater and other anti-bulk e-mail software.
“Checking your e-mail is getting to be a chore, not a pleasure.”

On the surface, the Web is doing pretty well, thanks to remarkable search
engines such as AltaVista, Infoseek and Lycos. You can search millions of Web
pages simply by typing in a key word or phrase. Within a few seconds a list of
pages that meet your criteria appears in front of you.

But the many users of these search engines may be surprised to learn that
they cover fewer than half the pages available on the Web. And with the
phenomenal growth of the Web and new methods of presenting information on its
pages, the search engines are falling further and further behind. “Nobody can
afford enough hardware to index the whole Web and serve it back to the entire
planet,” observes Louis Monier, chief technical officer at AltaVista
Software.

Just how much information is out there is impossible to say, not least
because anyone with a computer attached to the Internet can publish Web pages.
Monier estimates that when AltaVista’s search engine was launched in 1995, the
Web contained around 50 million pages on 100 000 sites. Now, he says, there are
between 100 million and 150 million pages on around 650 000 different sites.
Meanwhile, most Web indexes have hardly grown at all over the past year.

The companies that run the search engines say that rather than making their
databases bigger, they are putting their efforts into making it easier for users
to find what they want on the Web. Adding more pages to a database often makes
little difference to the user, they argue. “If the answer people need is in the
first 50 000 pages searched, do you need the other 50 000?” asks John Nauman,
vice-president of engineering at Infoseek. And as Monier points out: “There is
so much duplication of information on the Web. There may be ten more versions of
the answer I am looking for elsewhere unindexed, but I won’t mind as long as I
get my answer.” He says a study has shown that, if users perform a search on
more than one database, they will often believe that a small one is more
effective than a larger one, because it offers them fewer “hits” and they assume
that it is giving them more precisely targeted answers.

The people who complain most about the shortcomings of search engines are
those who publish Web pages. In many cases, search engines are the only
mechanism they have to announce themselves to the world. John Pike, webmaster
for the Federation of American żěè¶ĚĘÓƵs, was horrified when he discovered that
only 600 of the 6000 pages of his site were indexed by AltaVista. “This is like
buying a phone book that only has even-numbered phone numbers,” he says.

Danny Sullivan, a London-based consultant studying search engines, recognises
the problem that search companies are up against. “They all face an uphill
battle in keeping up with the growing Web.” But he feels they should be more
open about their limitations. “If they are just sampling the Web, users should
understand that.”

AltaVista is concentrating on providing at least a sample from every Web
site, says Monier. Only on the most frequently visited sites does it try to
index the majority of the content. Although this works well for those favoured
sites, and also for very small sites where a few pages might give a good
indication of their content, it presents big problems for other larger sites
such as Pike’s. But Monier defends the approach. “Our current policy is based on
fairness,” he says. “Every site gets a chance to be represented somehow.”

Infoseek adopts a similar approach. According to Nauman, its database now
contains information on 25 million to 30 million pages of text. But about 90 per
cent of queries are answered using the most frequently accessed 1 million pages.
And more than 90 per cent of the pages in the database are never accessed as the
result of a search. The company is therefore concentrating on making sure that
the information it has already indexed is as up-to-date as possible, although
Nauman admits that there are those inside Infoseek who think the company should
be aiming to cover the whole of the known Web.

Maintaining the quality of a search engine’s index is made especially
difficult by the Web’s volatility. A random sampling of pages on 2000 sites over
three months in 1995, carried out by two US universities as part of research
into a new indexing system, indicated that the average time a page of text
remained unchanged on the Web was just 75 days. A substantial percentage changed
every 10 days or less. Sometimes pages disappeared entirely, but more often the
information they contained was simply updated, or the page was moved to a
different address.

Nauman says that around 10 per cent of the pages indexed in Infoseek’s
database no longer exist. This, he believes, is one of the biggest frustrations
for users. The company now plans to visit all its indexed pages repeatedly over
a period to determine how often they change. Those that change infrequently will
then be checked every two months, while those that change more regularly or are
looked at more often will be checked every day or two.

All search engines work in a broadly similar way. They send out programs
called “spiders” to scan and catalogue the Web automatically by following the
links between documents. The pages the spiders collect are indexed by key word
and stored in huge databases that can then be accessed by the public. A site
will generally not be picked up by the spider unless another site links to it,
or its owner registers it manually with the search engine.

In the early days of the Web, pages carried only simple text. What is more,
they were open to anyone who cared to view them. Now, as technology advances and
different organisations pursue varied commercial goals, the Web is beginning to
fragment. Some information cannot be read with an ordinary Web browser such as
Netscape or Internet Explorer. For example, the documents in the vast collection
of scientific papers on a variety of subjects held by Los Alamos National
Laboratory in New Mexico and known as the “xxx” e-print archive—contain
numerous complex diagrams and formulae. These can only be properly displayed and
printed if the user has installed additional software that can cope with the
software company Adobe’s PostScript or PDF file formats.

Sites that contain mainly text can be difficult to index, too. The New
York Times, for example, is one of a number of sites that deliberately bar
users who have not paid an entrance fee or filled out a registration form. The
spider programs cannot easily register to view protected sites, although some do
now index many of the more important ones.

Some sites that are officially open to all may cause other difficulties for
the spiders. Paul Holbrook, director of Internet technologies at CNN, says the
company used to exclude all search spiders from its site because they distort
the figures that the company records on the numbers who visit it. Many Web
publishers use these data to sell advertising space. The British Library’s newly
unveiled online catalogue is an example of another kind of site that will defeat
a search engine’s spider: it contains information about millions of books and
periodicals, but it is visible only to users who have made a specific search
through the library’s online query form.

Sangam Pant, vice-president of engineering at Lycos, believes that faster
software and more powerful computers may keep search engines abreast of the
growth of the Web for a while. But in the long run “trying to keep up using
brute force is not the answer,” he says.

Instead, search engines may come to rely increasingly on
“meta-data”—brief descriptions of the contents of a page or site that are
embedded in the Web pages but read only by searchers.

Search engines already use meta-data to a certain extent to index pages. In
the future, instead of striving to maintain a database containing the text of
every page on every site, search engines may use meta-data to index entire
sites. But the system will only work if Web publishers can be relied on to
describe their pages accurately; no computerised catalogue would have the
resources to check the validity of every description. “We could provide Web
browsers with a lot more value if we could rely on the key words we are given,”
says Pant.

But all too often this is not the case. Already, the meta-data fields of many
Web pages are stuffed with a small number of key words repeated over and over
again in an attempt to ensure that search engines place the page high up in the
list of hits it displays to the user. The designers of search engines are aware
of this trick and try to compensate for it.

Monier is not optimistic about using meta-data to solve the accessibility
problem. Instead, he foresees an increasing number of specialist search engines
and indexes providing in-depth coverage of the Web by language or by subject. He
suggests that the difficulties facing search engines may be overstated. And he
believes that the exponential growth of the Web is bound to stop at some point.
There is already evidence that the growth is slowing, he says. At the end of the
day he points out, “if users felt they had serious difficulty finding what they
wanted, they would stop using us”.

But Sullivan thinks there is still plenty of work for the search engine
companies to do. “At some point, if you are going to run these services, you
have to take responsibility for maintaining standards,” he says. If you know it
is not good enough, you should do something about it whether your customers are
complaining or not.

If a telephone, television or personal organiser is going to be connected to
the Net today, it needs to contain all the basic components of a
computer—a processor, memory, and operating software.

But this can add hundreds of dollars to a manufacturer’s costs, says Ryo
Koyama, the president of iReady. The Internet Tuner chip that is being
developed by iReady should allow manufacturers to put browsing and mail
features in pagers, mobile phones and personal organisers for just “a few
additional dollars”, he says. The Internet Tuner chip will still require an
input-output system however.

The Internet Tuner will take raw data from a modem or other network
connection, recognise Net protocols, translate the incoming information into
e-mail or Web pages, and then display them on the device they are built
into.

The first versions of the chip Toshiba is making only work with plain text.
Future versions should be able to handle still images, fax, Java applets, video
and audio. This should mean savings: TV-based Web browsers might come down in
price, and future telephones or faxes might avoid high charges by using the Net
for long-distance calls.

The first products to use the chip are expected to hit the shops by the end
of the year. Toshiba will manufacture the chip and products based on it, but
other manufacturers will be able either to purchase the chip from Toshiba or
license their own designs from iReady.

The virtual animals first made their appearance in a computer game called
Creatures, released in November last year by Cambridge-based Cyberlife
technologies. Owners of the game could teach their pet to talk or perform simple
tricks. The creatures have rudimentary brains made up of a network of 1000
neurons, grouped in five lobes and linked by 5000 synapses. Now a customised
version of the same nervous system is being used by Cyberlife to make virtual
bank customers for the Knowledge Lab in London, part of the US-based computer
company NCR.

The cyber customers will be set free inside computer mock-ups of banks.
Although the environment is different from the game, the concepts being tested
are the same, says Anil Malhotra of Cyberlife. “In the game, creatures need to
eat or breed. In a bank, they might need to get cash or find information.”

The virtual bank branches will have a variety of layouts, and different
numbers of cashiers and cash machines. How the virtual customers react will
depend on what they are in the bank for and what they have learnt—for
example that chairs or pot plants don’t hand out cash.

Early trials have proved the simulation works well says Malhotra. Instead of
guessing customers’ likes and dislikes, designers will be able to watch the
reaction of the virtual customers.

Cyberlife is now working on more complex computer brains for future
simulation projects. At the moment, agents can learn to avoid fire by
associating it with heat, and heat with pain. But they cannot yet understand
longer chains of reasoning.