EACH day, millions of people rely on search engines to keep them afloat on the sea of information that underlies the web. But if you think search engines such as Google, Yahoo, MSN Search and Ask Jeeves provide an independent guide to what is online, think again. Search results are increasingly influenced by a growing industry of 鈥渟earch engine optimisers鈥�, companies whose job it is to ensure search engines place their clients at the top of the results list for the most relevant matches.
Businesses have wised up to the fact that ranking high in search engine results makes money. Because of this, the search engine optimiser industry of North America alone is now worth around half a billion dollars, according to the Search Engine Marketing Professional Organization (SEMPO) based in San Francisco, California. Optimisers Oneupweb in Suttons Bay, Michigan, estimate that a US company can increase sales by 42 per cent in the first month it moves into the top 10 on a Google search and by about 80 per cent in the second month.
So how does the work of optimisers affect what you get when you search? Some, called 鈥渨hitehats鈥�, are marketing companies that advise their clients on how to make sure that search engines interpret their websites accurately. Their work probably benefits users, says Greg Jarboe of SEMPO, because it means that sites will pop up in response to relevant search queries, making it easier for people to find what they want. After all, search engines can鈥檛 do their job if websites are poorly designed. 鈥淣o search engine is perfect or omniscient,鈥� says Jarboe.
Advertisement
But the search optimisation industry has a darker side, too. Some optimisers employ dirty tricks to place websites at the top of search results even if they are irrelevant to the search terms. They hope that by driving people towards their clients鈥� websites, irrespective of whether they intended to go there or not, a few might spend money when they get there.
These 鈥渂lackhats鈥� are also called 鈥渟earch engine spammers鈥�, analogous to email spammers. If the spammers ever get the upper hand, they will transform a simple search for a restaurant into a hellish trawl through a litter of unofficial or irrelevant sites hawking porn, Viagra and online casinos, says Seth Finkelstein, an independent researcher into spam on Google who is based in Boston. 鈥淭hink of your unfiltered bulk email,鈥� he says, 鈥渢hat鈥檚 what search engine results pages would look like if the spammers ever seized control.鈥�
Search engines have been locked in a battle with spammers since engines emerged in the mid 1990s, a battle that shows no sign of abating. As fast as search engines tweak their algorithms to detect spam websites, spammers are updating their methods for playing the system. 鈥淚t鈥檚 an arms race, a cat-and-mouse game,鈥� says Chris Sherman, editor of the daily newsletter published by in Boulder, Colorado. In an online survey of 300 professional optimisers in January 2005, 17 per cent said Google was 鈥渇looded with spam鈥� while 43 per cent said they saw it 鈥渘ow and again鈥� (see Diagram).
Search spamming is possible because the major search engines rank websites using software that can be tricked, says Marc Najork of Microsoft Research in Mountain View, California. The search engines send software 鈥渃rawlers鈥� around the internet. The crawlers download pages and follow hyperlinks, just like a person surfing with a browser. The difference is that the crawler examines everything on the site, rifling through the html code behind the pages, extracting content and feeding it back to a central indexing system run by the search engine company headquarters. There, a complex concoction of algorithms decides which sites should be served up in response to which queries.
Exactly how these algorithms work is a closely guarded secret, and varies between search engines. But industry experts point to a few things that all the engines have in common. The algorithms look for words in the site鈥檚 text to determine what it is about. They try to determine the quality of the text to see if it is a real site written by a human or gibberish generated by a computer program. They also count the number of external links pointing to the site to assess its importance, and weigh these by the quality of the linked sites.
Unfortunately, spammers can fake all these things. The most common trick is 鈥渃loaking鈥�, creating a site that shows one set of content to a search engine crawler and another set to you. Spammers can cloak in several ways. Each computer or network of computers on the internet is identified by a unique number, called an IP address. Spammers might program a site to serve up the contents of a respectable news site to an IP address owned by a major search engine company, thereby qualifying for a high ranking, while displaying porn or shopping to everyone else. Alternatively, spammers can hide reams of high-quality text and links in the same colour as a website鈥檚 background, so that a person won鈥檛 see it but a web crawler will. In the most extreme examples, the hidden text might include the entire contents of a site with a high search engine ranking 鈥� for example, www.newscientist.com 鈥� which would boost the search ranking of a site selling online gambling or pornography.
The companies behind the search engines take a dim view of cloaking. After all, their business model depends on them consistently serving up relevant, high-quality results. In the registration statement Google filed in April 2004 before going public, it listed spam as one of the 鈥渞isks to business鈥�. 鈥淚 wouldn鈥檛 classify it as a serious threat. It鈥檚 one of the things we are looking at that we think we can improve,鈥� says Peter Norvig of Google in Mountain View, California.
Search engine companies deal with spammers either by moving spammed sites back down the search results to a lower rank or dropping their websites from the search results entirely. Both Google and Yahoo publish rules of acceptable behaviour online. Serving up content to the search engine which is different from that for human users is a no-no. 鈥淐loaking is the really nasty stuff that would definitely get you banned from Google,鈥� says Sherman. 鈥淚t鈥檚 a heavy weapon.鈥�
鈥淭he search engines will probably win but in the meantime it鈥檚 a guerrilla insurgency鈥�
Cloaking isn鈥檛 the only trick spammers have up their sleeves. Search engines also look at the number of links pointing to a page. So some in the search optimiser industry develop websites called 鈥渓ink farms鈥� that have the sole purpose of linking to other spam sites, charging their clients per link. Such a website would come high up the ranking of a search engine because of the number of links to it.
Among search optimisers, link farms are considered to be not quite as bad as cloaking. In a tiny office in North Yorkshire, UK, 鈥淒aveN鈥� and four programmers say they make tens of thousands of dollars a week by guaranteeing websites a place in Google鈥檚 top 10 results for a given search. DaveN calls himself a 鈥済reyhat鈥�. He doesn鈥檛 see anything wrong in bending the rules set up by Google or Yahoo by arranging for high-quality sites to link to that of his clients, but he says he would stop short of 鈥渓ying鈥� to the search engine crawler.
As crawlers patrol the net, search company algorithms eventually notice link farms, although the details of how their algorithms do this are company secrets. But it takes time for the crawlers to pick them up. Najork says MSN recently noticed static link farms being abandoned in favour of cascades of pages that are created in real time as fast as crawlers can index them. They each contain a link back to the client whose rating will benefit as a result, and link to each other. 鈥淭his is one of the most aggressive tactics used by spammers today,鈥� says Najork. 鈥淚t blows my mind.鈥� What is more, spammers have even been known to write programs that instantly insert sentences containing the same keywords that users are typing into the search engines, making their site appear highly relevant to popular queries.
But innovation is not unique to spammers. When Google recently noticed that spammers were littering the comment section of online journals, or blogs, with irrelevant links to spam websites, the company adapted its search algorithms to discount links in blog comments. Yahoo and MSN Search swiftly adopted the same tactic.
There is a fine line between taking decisive action against spammers and knocking genuine companies. In an example often cited by search optimisers, Google altered its algorithm in November 2003 to crack down on spammers. While industry experts agree the change got rid of many spam sites, some complain it also punished many clean sites. John Goslin of the Goslin Group in Cambridge, Massachusetts, says that strollers.com, a client of his at the time, 鈥渘ose-dived鈥� overnight from being ranked in the top 10 for a search on the term 鈥渟troller鈥� to being ranked at around 1000, even though the site was actually selling strollers. It was unclear why the site had been targeted, but the effect on sales was massive because the change came just before the Christmas shopping period.
Google responds that it was a routine update. 鈥淭hings are always changing. People were upset, but there are always just as many people who are happy,鈥� the company told 快猫短视频. US courts have ruled that Google鈥檚 search results are 鈥渙pinions鈥�, meaning that Google is entitled to serve up whatever results it likes under the First Amendment. 鈥淚t鈥檚 up to us to decide what our opinion is,鈥� says Norvig.
Nevertheless, the spammers continue. In some industries, says Sherman, it is widely believed to be impossible to stay in business without spamming to some extent, because the competition is doing it. While the companies most often associated with spam are porn and betting sites, he says he thinks there may be 鈥渟everal well-known Fortune 500 companies that are using questionable tactics鈥�. Even some whitehat optimisers tread a fine line between keeping their clients visible on search engine results and crossing the boundary into spamming-like techniques. 鈥淪ome are really afraid,鈥� says Goslin.
Greyhat or whitehat, the more competition there is, the more money there is to be made by the optimiser industry, and the more incentive for spammers to outwit the search engines. 鈥淭he search engines will probably win the war on spam eventually but in the meantime it鈥檚 an ongoing guerrilla insurgency and spammers do a lot of harm,鈥� says Finkelstein.
