快猫短视频

SOMEWHERE among the two billion blobs of light captured in the Palomar
Observatory鈥檚 Digital Sky Survey are quasars鈥攄istant galaxies that are
among the brightest objects in the Universe. Astronomers would dearly like to
know more about them and their mysterious power source, but which of those
myriad blobs of light should they be looking at? It is like finding the
proverbial needle in a haystack, and this haystack is the entire cosmos.

The astronomers鈥� predicament is shared by some unlikely bedfellows:
supermarket executives, stock market analysts and detectives. All are faced with
a surfeit of data, but a dearth of information.

Now a growing band of computer scientists say they can dig out nuggets of
24-carat knowledge from huge mountains of database dross. They call themselves
鈥渄ata miners鈥�, and they are wielding some pretty impressive tools, drawn from
esoteric fields such as artificial intelligence and statistical inference
theory. But the impact of their efforts is anything but esoteric.

By identifying potential new customers鈥攐r ways of hanging on to
existing ones鈥攖his information is worth millions in extra revenue. And
this is just the start, according to Usama Fayyad of Microsoft Research and
co-editor of a new book on data mining.

鈥淏ig corporations can obviously benefit most, as a small improvement in, say,
prediction or modelling can easily add up to millions of dollars through sheer
numbers,鈥� he says. 鈥淏ut data mining can be quite as powerful for small
businesses too鈥攍ike a restaurant owner who serves 500 meals a week and
wants to know what new dishes to recommend to customers, based on their past
肠丑辞颈肠别蝉.鈥�

At first sight, data mining sounds like little more than an exercise in
graph-plotting: just rummage through your customer data and find out who chose
prawn cocktail and steak and chips, but eschewed the Black Forest gateau in
favour of apricot tart. But what if there are 5 starters, 10 main courses and 8
desserts? That鈥檚 400 combinations for a start. Then there are the different
permutations of age groups, social classes and income levels. It鈥檚 called the
鈥渃urse of dimensionality鈥�: the way in which just a handful of variables can
produce a colossal number of permutations. Multiply it by the size of the
customer base鈥攚hich can easily be hundreds of thousands, even
millions鈥攁nd finding a trend starts to look impossible.

Yet data miners are now happy to tackle such daunting tasks. Dealing with
such large volumes of data might seem to demand heavy-duty computer power and
certainly some data mining companies, such as Bracknell-based White Cross
Systems, wield big-hitting parallel processing computers to blast through huge
corporate databases in seconds.

Other data miners try to take a leaf out of the old prospector鈥檚 book, and
look for 鈥減romising ground鈥� before they begin major excavations. 鈥淚t鈥檚 not
unusual for the first part of a data mining project to be concerned with how to
get a suitable sample of the database to use,鈥� says Dave Shuttleworth, senior
consultant with White Cross. 鈥淚鈥檝e personally experienced projects where people
take months to decide which 98 per cent of their data to ignore.鈥�

Data miners certainly have to be prepared for a fair bit of drudgery before
they can get to work. The raw data often has to be 鈥渃leaned up鈥� so that all the
information is in a uniform state for mining. Shuttleworth recalls a case where
a retailer鈥檚 database included 500 different ways of describing which American
state the information came from.

Once the ground is prepared, data mining can begin in earnest. The challenge
is not simply to dig out valuable information from huge amounts of data, it is
also to pull out trends, groupings and connections that depend on many
variables, linked in complex ways. Such patterns cannot be found using simple
textbook methods such as linear regression, which finds the best straight line
linking one variable to another.

Instead, data miners are turning to more powerful methods such as rule tree
induction. This uses ideas taken from information theory and the laws of
probability to extract鈥斺�漣nduce鈥濃�攔ules that can best account for the
data. For example, by looking at the numbers of customers who choose different
starters, main courses and desserts, tree induction can reveal the most likely
rules linking customers together: 鈥淚F prawn cocktail AND cheap plonk THEN steak
and chips.鈥�

Pattern matching

Patterns can also be dug out using neural networks, computers that crudely
mimic the brain鈥檚 ability to find relationships in data by being shown many
examples. Such networks are first trained on data samples showing, say, the
relative proportions of customers who order particular starters, main courses,
drinks and desserts. The network then tries to classify each type of customer
according to their preferences. At first, the classification is inaccurate, but
the neural network鈥檚 algorithms allow it to learn from its mistakes, revealing
relationships between, say, orders for liqueurs after roast beef dinners.

Techniques such as tree induction and neural computing have been around for
years. But data miners are discovering that they must do more than merely apply
these old methods to huge databases. 鈥淒ecisions based on data mining results may
involve very large amounts of money,鈥� says Beatriz de la Iglesia of the
University of East Anglia. 鈥淎nd management is not enthusiastic about embracing
ideas they cannot understand or analyse for themselves.鈥�

This demand for lucidity is proving a challenge for data miners. For example,
tree induction has a nasty habit of throwing up appallingly complex rules even
with relatively simple databases鈥攍ogical nightmares such as 鈥淚F chips AND
(NOT steak AND peas AND (NOT ice-cream AND fruit cocktail)) AND. . .鈥� on and on.
In a recent analysis of customer behaviour for a British financial institution,
de la Iglesia and her colleagues Justin Debuse and Vic Rayward-Smith found that
some induction programs produced huge decision trees with dozens of
branches.

The situation with neural networks is even worse. Famed for their ability to
find useful rules from complex and messy data, they are also notoriously opaque
in their reasoning. Konrad Feldman of the London-based data analysis consultancy
SearchSpace recalls developing a neural network for an Italian credit reference
company that predicted which companies were most likely to file for bankruptcy
with around 75 per cent accuracy鈥攁 much higher score than traditional
methods. 鈥淭he problem was that the company then had to justify its predictions
to clients, and they kept wanting to know exactly why a particular company was a
bad risk鈥攁nd what was a neural network anyway?鈥�

Feldman and his colleagues started again, this time using 鈥済enetic
algorithms鈥�. They began with a set of guesses about which rules might apply, in
the form of combinations of traditional financial measures such as share price
to earnings ratios. Each guessed rule was then tested to see how well it
performed on predicting bankruptcy. By weeding out the less successful ones and
letting the better ones combine with each other, a Darwinian process of
鈥渟urvival of the fittest鈥� led to increasingly reliable prediction rules.

Overall, the accuracy of the predictions was slightly worse than using the
neural network. But Feldman鈥檚 customers still preferred the data mining process
that used genetic algorithms, mainly because they could understand the rules it
was using to make predictions. Understanding the origin of mined nuggets is
about more than making clients feel comfortable, however. Rob Milne of
Intelligent Applications, an artificial intelligence applications company in
Livingston, West Lothian, points out that it can also help to protect data
miners from unearthing fool鈥檚 gold.

He cites his own experiences analysing the database of a leading financial
services company offering pensions, insurance and investment policies. The
company wanted to identify which customers were most likely to be poached by
aggressive rivals. Milne and his colleagues began by using rule induction
methods, but the results that were being produced did not match reality.
鈥淪uddenly from one combination of inputs, the accuracies jumped to over 95 per
cent accurate in predicting both the customers most likely to stay and the
customers with a propensity to leave,鈥� recalls Milne.

Had the data mining unearthed some amazing seam of marketing gold? 鈥淥ur
experience made us very suspicious鈥�, says Milne, and he and his colleagues set
about looking for an explanation. It turned out that the accuracy was due to
quirks of both data collection and market behaviour that appeared during one
short period. The bad news was that the rules applied only to that part of the
dataset, and had no predictive power at all.

The moral of the story is clear, says Milne: the use of clear, rule-based
methods let them trace the source of the spurious accuracy, and spot the fool鈥檚
gold. 鈥淚f we had a black box approach to data mining鈥攍ike neural
networks鈥攚e would have no way to check on what basis the decisions were
being made.鈥�

Best bet

At the Thomas J. Watson Research Center in New York State, Chidanand Apte and
Se June Hong have attacked the problem of intelligibility by using logic methods
to find the simplest rules capable of spotting trends in data.

Their target was a familiar one: predicting which companies will do best on
the American Securities Market. Apte and Hong used the same 40 financial
indicators as those monitored by stockbrokers鈥攕uch as average monthly
earnings and investment opportunities. They then tried to find the simplest
rules for spotting the best investment bet each month, using a technique known
as disjunctive normal form logic, a way of connecting descriptions of data
together so that any contradictions can be rapidly found.

The resulting simple investment rules worked very well, turning in a 270 per
cent return over five years, compared to a market average of just 110 per cent.
Not surprising, Apte and Hong would like their technique to be taken up by an
investment house in the business of buying and selling stock.

One of the biggest delights of the data-miners鈥� work is finding a technique
that uncovers information no one would have expected to find. Shuttleworth and
his colleagues at White Cross, for example, unearthed a surprise for one
customer in the telecommunications business that was looking for a good way to
identify users who were unlikely to pay their bills on time.

Before carrying out the exercise, Shuttleworth and the telecommunications
company expected that the people most likely to have trouble paying their bills
would be those on low incomes. 鈥淲e discovered that the `urban
achievers鈥欌�攚hite collar, good salary, college educated鈥攖urned out to
be among the worst offenders.鈥�

In another project for the same telecommunications company, the White Cross
team expected to show that the company could improve profits most by trying to
encourage low-usage customers to make better use of the services. 鈥淚n fact, the
data mining showed that the highest growth sector was high-usage customers
moving to even higher usage.鈥�

While such successes look set to trigger demand for data mining far beyond
the financial sector, there is a problem holding up its progress. Most of the
world鈥檚 data is still stored on paper, microfiche, or word processed documents
in some obscure format. Simply reading such data at all is a major challenge
facing data miners.

Epsom-based Software Scientific have recently made a major breakthrough in
this problem using natural-language processing鈥攖echniques for controlling
computers that use normal words rather than programming language. The result is
a data-mining software package that hunts for information in ordinary text. For
example, given text files of hundreds of statements taken from criminal
suspects, the program uses set theory and linguistic analysis to find key facts
and relationships in the data. Detectives can simply ask the computer 鈥淲ho is
the most likely culprit?鈥�, and the relevant extracts from the statements appear
on the screen in a few moments.

High interest

With so many organisations having the bulk of their data locked up in
ordinary text, such systems are attracting interest from many quarters,
including police forces, says Lea. 鈥淎lthough it is obviously no replacement for
police officers, it can be used to make better use of the resources.鈥� While
impressive, such techniques raise the spectre of data-mining falling prey to
hype by those who seize on every new technology like a child with a new toy. 鈥淚t
is true that some simple data mining work can often result in great successes,
but this by no means justifies people thinking the fundamental problems are
solved,鈥� warns Fayyad.

One of most pressing, he believes, is the fool鈥檚 gold problem. 鈥淢any patterns
and trends are extractable from data鈥攁nd most of these are likely to be
junk, simply because the data is finite and computation is limited,鈥� says
Fayyad.

Statistical inference鈥攖echniques for drawing reliable conclusions from
complex data鈥攃an help. One key idea is that the reliability of a finding
is proportional to its plausibility. For example, Feldman and colleagues at
SearchSpace found a connection between sales of dog food and fizzy drinks
lurking in a supermarket鈥檚 database. The sheer implausibility of this connection
led them to write it off as a quirk. But other connections may not be so easily
dismissed.

Data mining is under threat from another bugbear of new technology: the
reluctance of commercial users to trumpet any successes they have using the
technique. Instead, the best adverts for data mining are likely to come from
those working in more open fields.

Last December, Fayyad and colleagues at the Jet Propulsion Laboratory in
Pasadena, announced the discovery of a slew of new quasars among those myriad
blobs of light on the Palomar sky survey鈥攃ourtesy of data mining. Using
decision tree and rule-based methods, the team trained algorithms to classify
light sources as stars, galaxies or distant quasars. They were then able to tell
astronomers searching for quasars which objects might reward closer study. The
result: 16 previously undiscovered ancient quasars bagged in a fraction of the
telescope time usually needed.

With funding for science under so much pressure, making discoveries by mining
existing data could well prove to be an idea whose time has come. As Fayyad
says: 鈥淚t is a truly new way of doing science.鈥�