鈥淭HERE are a lot of very bright people here. And what are you doing? Blocking penis-enlarger ads. That鈥檚 not why your mothers paid for your education. It鈥檚 depressing.鈥�
Barry Shein of internet service provider The World is talking to an audience of top computer programmers, or hackers, as they swashbucklingly call themselves. They are all jammed into the Massachusetts Institute of Technology鈥檚 largest lecture hall with one purpose in mind 鈥� finding new ways to combat spam, the unwanted emails that cascade into every inbox on the Internet, pushing everything from pills to porn.
Almost nobody likes spam, but hackers hate it with a special passion. A hacker will give dozens of plausible reasons for hating spam. But according to Paul Graham, a programmer based in Cambridge, Massachusetts, and organiser of this, the First Spam Conference, the real reason is ego: a hacker鈥檚 computer is his world, and he is its ultimate ruler.
Advertisement
鈥淗ackers are able to perfect their computer experience,鈥� Graham says. They can unravel the code of the operating system and make the computer do anything they want. Anything but ignore spam. The result is that hackers spend a huge amount of their time reading, thinking and writing programs to eliminate spam. And their efforts are finally paying off.
Last August, Graham published a paper entitled A Plan for Spam, in which he described the world鈥檚 first truly effective spam filter. Even more importantly, it was written in clear, strong prose, and doubled as a manifesto, a rallying cry to hackers everywhere.
About time, too. So far, nobody has had the least success in solving the spam problem or even in checking its exponential growth. According to Brightmail, a company that provides anti-spam services, in September 2001 just 8 per cent of all email was spam. In December 2002 that proportion had risen to 40 per cent, and it is still climbing.
The problem goes far deeper than hackers鈥� bruised egos. William Yerazunis of the Mitsubishi Electrical Research Laboratory in Cambridge, Massachusetts, estimates that it takes two seconds to identify and delete a spam message, which adds up to a cost of about $2860 per million spams, at the US minimum wage. But the spams cost the spammer almost nothing to send, and just a few positive responses will keep them in business.
The burden of deleting this much spam is crippling. Yerazunis describes a kind of psychological fugue state that he calls a 鈥渟pam-spasm鈥�, in which a glassy-eyed user accidentally deletes a piece of ham 鈥� the hacker term for non-spam. And the consequences of losing a piece of ham can be enormous.
The flood of spam messages is so intense that it can overwhelm ISPs like The World, which is based in Brookline, Massachusetts. 鈥淭he spam comes in so hard, so fast, flooding your pipes, flooding your mail servers,鈥� Shein says. 鈥淵ou know you鈥檙e not doing a good job until you get this under control.鈥�
Service providers have not been idle. One approach to reducing the volume of spam that they have to handle is blacklisting. To protect their identity and cut costs, most spammers rely on 鈥渙pen relays鈥� to send their spam. An open relay is a mail server that allows all comers to use its facilities. Spammers soon figured out that they could hijack open relays and use them to send millions of emails. All they needed was a throw-away account, such as a two-week free trial from AOL, say, and they were in business. The expense of sending all those emails was carried by the open relay, and as a bonus the spammer鈥檚 identity was hidden.
To combat this problem, lists of open relays were published on the internet. Many ISPs and company servers began blocking all email coming from them, even though many of these servers also handled email from legitimate users. Most blacklisters do not care about this 鈥渃ollateral damage鈥� 鈥� the fight against spam is, after all, a war, and sometimes the innocent must be harmed. But civil libertarians abhor blacklisters, and Graham even calls them terrorists.
To a programmer, the most obvious way to filter spam is to write a program that weeds it out. After all, this is what most people do successfully every day, at least until they succumb to a spam-spasm. 鈥淢ost hackers鈥� first instinct is to try to write software that recognises individual properties of spam,鈥� Graham wrote in his paper. 鈥淵ou look at spams and you think, the gall of these guys to try sending me mail that begins 鈥楧ear Friend鈥� or has a subject line that鈥檚 all uppercase and ends in eight exclamation points. I can filter out that stuff with about one line of code.鈥�
Existing spam filters work in this way, by looking for the giveaways in the messages. Graham notes that almost 80 per cent of the spam in his email box contains the word 鈥渃lick鈥�. But what about the other 20 per cent? You can add words 鈥淰iagra鈥� and 鈥渦nsubscribe鈥� as well, but the spammers soon catch on and begin to use other words instead. And what if the words are inadvertently used in your regular email? Being forced to read a few pieces of spam is bad, but missing an important bit of ham is intolerable.
Like many other programmers, Graham was fighting a losing battle. 鈥淚 got addicted to trying to identify spam features myself, as if I were playing some kind of competitive game with the spammers,鈥� Graham says in his paper. But having to read thousands of pieces of spam was just too demoralising.
That was when Graham decided to write a program that analyses his emails for him. All he has to do is feed the program sample spams and hams, and it then rates the spamminess of each word. For example, if 90 per cent of emails containing the word 鈥淰iagra鈥� are spam, then the word 鈥淰iagra鈥� gets a negative rating of 90 per cent. To prevent false positives, the program rates words that turn up in a ham message twice as positively as those in spams score negatively.
When the program is subsequently fed a new email, it runs a statistical analysis to calculate how likely it is to be spam, based on the words in it. It does this using a bit of mathematics discovered among the papers of the English mathematician Reverend Thomas Bayes after his death in 1761. Bayes鈥檚 theorem provides a way of calculating conditional probabilities 鈥� how the likelihood of an event changes when new information is introduced (see Graphic). Emails that have more than a 90 per cent chance of being spam are filtered out.
The result of all this is a spam filter that, once trained, can spot over 99 per cent of all spam. A few borderline messages slipped through, such as one from a person looking for 鈥渁n alien or a time traveller鈥�. But more importantly, the program almost never labels a piece of ham as spam. And as long as the filter is kept up to date with samples of current spam, spammers鈥� efforts to change their vocabulary will not get round the filter.
Others have tried Bayesian filtering before, but with only moderate success. Graham鈥檚 extra tinkering seems to have made the difference. Following his paper, other hackers have written their own Bayesian spam filters, adding layers of protection and ever more complicated mathematics. Mozilla Mail already uses Bayesian filtering. The program is open-source, meaning its guts are open to inspection and tinkering.
The big guns are piling in too. Joshua Goodman from Microsoft told the spam conference that the firm was shipping its latest Microsoft Network software with a 鈥渟tatistical鈥� spam filter. True to Microsoft鈥檚 tradition of secrecy, Goodman would not reveal the exact nature of the statistical tests the filter uses, but like Bayesian filters, it learns from experience.
Inspired by Graham鈥檚 work, programmer John Graham-Cumming, based in New York City, wrote an open-source mail filter called POPFile in his spare time. POPfile is freely available on the Internet and does a wonderful job of recognising spam after only a brief period of training. But Graham-Cumming warns that this happy state cannot last: spammers are just too clever.
There are dozens of ways spammers can beat these filters. Many exploit properties of HTML, the language that tells browsers or email software how to display text. Spammers can incorporate hidden text in the HTML 鈥� a serious discussion of nuclear disarmament, say 鈥� that will not be visible to the reader but can skew the message鈥檚 statistical profile so that it slips through a spam filter. Another technique that Graham-Cumming is particularly impressed by is one that 鈥渃uts鈥� a message into long, thin, parallel strips. To a spam filter these strips look like confetti from a document shredder, but the HTML code in the receiver鈥檚 email software automatically reassembles the strips into a message.
鈥淭here are some really clever people working to make spam better,鈥� Graham-Cumming tells the audience, with genuine admiration. 鈥淪pammers鈥� technical ability should be respected.鈥� Looking around, there is no obvious blush of pride to be spotted. But here among this group of over 500 hackers must be at least a few poker-faced spammers.
Once spammers begin to outwit the Bayesian filters, one possible response will be to make anti-spam programs even more intelligent. Jason Rennie, a graduate student at MIT鈥檚 Artificial Intelligence Laboratory, suggests training filter programs almost the way children learn language. Rennie鈥檚 prototype program considers each email as a long string of characters rather than words. It performs a statistical analysis to identify the basic units of the 鈥渓anguage鈥�, that is, the meaning-carrying parts of messages. The hope is that these will be different in spam and ham.
Although Rennie鈥檚 technique is still far too slow to filter heavy email traffic, it does give a hint of where all this is leading. From primitive blacklists through Bayesian filters to the first glimmerings of intelligence, an arms race is afoot. Give it a few more years of hacker vs spammer, and the smartest thing on any computer will be the email filter.