THE scene is Jerez Grand Prix circuit, the last race of the 1997 Formula One
season. Michael Schumacher is one championship point ahead of arch-rival Jacques
Villeneuve—thanks in part to brilliant tactical driving by his Ferrari
team-mate Eddie Irvine in the previous race. Villeneuve’s Williams team-mate
Heinz-Harold Frentzen may well play the same game this time, so qualifying in
pole position is even more critical than usual . . .
So what happens? Villeneuve, Schumacher and Frentzen all lap in exactly 1
minute 21.072 seconds. The astonished commentators hailed it as an amazing
coincidence. Well, “coincidence” it surely was—the lap times coincided.
But was it truly amazing?
Questions like this do not just apply to sport. They turn up everywhere,
trivial and important. Just how surprising was it to run into Great Aunt Lottie
from Sweden in that San Francisco strip bar? Is it really unexpected that three
different people at the Christmas party are wearing the same dress? And in
science, how significant is a leukaemia cluster? Does a strong correlation
between lung cancer and having a smoker in the family really prove that passive
smoking is dangerous?
Advertisement
One of us, Jack Cohen, is a reproductive biologist. Last year he was asked to
explain two very curious statistics. While in Israel, he was told that 84 per
cent of the children of Israeli fighter pilots are girls. “What is it about the
life of a fighter pilot,” he was asked, “that produces such a predominance of
daughters?” The second statistic arose in connection with in vitro
fertilisation. Nowadays, IVF clinics use ultrasound to monitor ovulation, and so
can determine whether an egg—and the resulting baby—comes from the
left or right ovary. One clinic discovered that most of the girl babies came
from the left ovary, and most of the boys from the right. A breakthrough in
choosing the sex of your children? Or just a statistical freak?
It’s not easy to decide. Gut feelings are worse than useless, because human
intuition is poor when it comes to random events. Many people believe that
lottery numbers that have so far been neglected must be more likely to come up
in future. In justification they plead the “law of averages”—everything
ought to even out in the long run. But the truth is different, and not at all
intuitive. Yes, in the long run, each lottery number is indeed just as likely to
turn up as any other. But the lottery machine has no memory. The proportions do
even out, in the long run, but you can’t say in advance how long that run will
be. In fact, if you choose any specific number of attempts, however large, then
the best prediction is that any initial imbalance will remain unchanged.
Our intuition goes even further astray when we think about coincidences. You
go to the local swimming pool, and the guy behind the counter pulls a key at
random from a drawer full of keys. You arrive in the changing room and are
relieved to find that very few lockers are in use . . . and then it turns out
that three people have been given lockers next to yours, and it’s all “Sorry!”
as you bang locker doors together. Or you are in Hawaii, for the only time in
your life . . . and you bump into the Hungarian you worked with at Harvard. Or
you’re on honeymoon camping in a remote part of Ireland . . . and you and your
new wife meet your department head and his new wife, walking the other way along
an otherwise deserted beach. All of which happened to Jack.
These coincidences all seem striking because we expect random events to be
evenly distributed, so statistical clumps surprise us. We think that a “typical”
draw in Britain’s national lottery is something like 5, 14, 27, 36, 39, 45, but
that 1, 2, 3, 19, 20, 21 is far less likely. In fact, these two sets of numbers
have the same probability: 1 in 13 983 816. Sequences of six random numbers are
actually more likely to be clumpy than not.
Lapping it up
How do we know this? Probability theorists tackle such questions using
“sample spaces”. A sample space contains not just the event that concerns us,
but all possible alternatives. If we are rolling a die, for instance, then the
sample space is 1, 2, 3, 4, 5, 6. For the lottery, the sample space is the set
of all sequences of six different numbers between 1 and 49. A numerical value is
assigned to each event in the sample space, called its “probability”, and this
corresponds to how likely that event is to happen. For fair dice each value is
equally likely, with a probability of 1 in 6. Ditto for the lottery, but now
with a probability of 1 in 13 983 816.
Thinking about the size of the sample space is a good way to assess how
amazing an apparent coincidence really is. Take the Formula One lap times. Top
drivers all lap at roughly the same speed, so it’s reasonable to assume that the
three fastest times would fall inside the same tenth-of-a-second period. At
intervals of a thousandth of a second, there are 100 possible lap times for each
to choose from: this list determines the sample space. Assume for simplicity
that each time in that range is equally likely. Then there is a 1 in 100 chance
that the second driver laps in the same time as the first, and a 1 in 100 chance
that the third laps in the same time as the other two—which leads to an
estimate of 1 in 10 000 as the probability of the coincidence. Low enough to be
striking, but not so low that we ought to feel truly amazed. It’s roughly as
likely as a hole-in-one in golf
Estimates like this help to explain astounding coincidences reported in
newspapers, such as bridge players getting a “perfect hand”, each having the
thirteen cards in a suit. In any one game, the chances of this happening are
staggeringly small. But the number of games of bridge played every week world
wide is huge. So huge that every few weeks the actual events explore the entire
possible sample space. In other words, you should expect perfect hands to turn
up somewhere—as often as their small but non-zero probability
predicts.
The use of sample spaces, however, is not entirely straightforward.
Statisticians tend to work with the “obvious” sample space. For that question
about Israeli fighter pilots, for instance, they would naturally take the sample
space to be all children of Israeli fighter pilots. But that might well be the
wrong choice. Why? We often tend to underestimate the size of the sample
space—and so assume that coincidences are more surprising than they really
are. It is all down to a crucial factor which we call “selective
reporting”— which tends to be ignored in most conventional statistics.
That perfect hand at bridge, for instance, is far more likely to make it to
the local or even national press than an imperfect one. How often do you see the
headline “Nottingham Bridge Players Get Entirely Ordinary Hand”, for instance?
The human brain just can’t resist looking for patterns, and seizes on certain
events that it considers significant, whether or not they really are. And in so
doing, it ignores all the “neighbouring” events that would help it judge how
likely or unlikely the perceived coincidence actually is.
Selective reporting affects the significance of those Formula One times. If
it hadn’t been them, maybe the tennis scores in the US Open would have contained
some unusual pattern, or the snooker breaks in that week’s tournament, or the
golf . . . Any one of those would have been reported, too. But none of the
failed coincidences, the ones that didn’t quite happen, would have hit the
headlines. If you include just ten major sporting events in the list of
would-be’s that weren’t, that 1 in 10 000 chance comes down to only 1 in 1000.
It’s like tossing a coin and turning up heads 10 times in a row.
So going back to the Israeli fighter pilots—is it just random chance or
is something else happening? To answer this, conventional statistics would set
up the obvious sample space (children of fighter pilots), assign probabilities
to boy and girl children, and calculate the chance of getting 84 per cent girls
in a purely random trial. But this analysis ignores selective reporting. Why did
anyone look at the sexes of Israeli fighter pilots’ children in the first place?
Presumably because a clump has already caught their attention. If it had been
the heights of the children of Israeli aircraft manufacturers, or the musical
abilities of the wives of Israeli air traffic controllers that showed up as a
clump, their clump-seeking brains would have pounced on those strange
circumstances instead. The conventional statistical approach tacitly excludes
many other factors that didn’t clump—it ignores part of the sample
space.
The human brain filters vast quantities of data, seeking things that appear
unusual, and only then does it send out a conscious signal: will you look at
that! The wider you cast your pattern-seeking net, the more likely it is to
catch a clump. There’s nothing wrong with that. But if you want to know how
significant the clump is, you can’t include the data that drew your attention to
the clump in the first place. In a room with 20 people there will typically be
one—the tallest—whose height puts them in the top few per cent in
the country. But you can’t then remove that person from the room, re-measure
their height, and then deduce that it is surprising that you have found somebody
with such an unusual height. That’s what you chose them for.
Extraordinary powers
This is exactly the error made in early experiments on extrasensory
perception. Thousands of subjects were asked to guess cards from a special pack
of five symbols. After several weeks of selection, anyone whose success rate had
been above average was invited back and tested some more. At first, these “good
guessers” seemed to have extraordinary powers. But as time went on, their
success rate slowly dropped back towards the average, as if their powers were
“running down”. This happened because their initial high scores—the
clumping for which they were chosen—were included in the running total. If
these fortuitous scores had been excluded from the second set of tests, then the
scoring rate would have dropped, immediately, to near average.
And so it is with the fighter pilots, and with the left/right ovaries. The
curious figures that drew researchers’ attention to these particular effects may
well have been the result of selective reporting—or, much the same,
selective attention. If so, then you can make a simple prediction: “From now on,
the figures will revert to fifty-fifty’. If this prediction fails, and if the
results instead confirm the bias that revealed the clump, then the new data can
be considered significant. But the smart money is on the prediction
succeeding.
The same fallacy can arise in conventional experimental studies, say to find
out whether certain foodstuffs give you cancer. To save time, the usual way is
to look at many different foods at once—fibre, fat, red meat, vegetables
and so on—and see how they all compare with cancer rates. So far so good.
But now, you pick out the biggest entry—the food that is unusually closely
correlated with cancer rates. Unless you’re careful, you now forget all the
other factors, and publish a paper saying that eating red meat significantly
increases your chances of cancer. However, you chose the most significant from
the hundred different foods you tried. Of course at least one comes out to be
significant. You’d be surprised if it didn’t, on statistical grounds alone, even
if all the foods were chosen at random.
Highly sensitive
A study last year by a team at London’s Institute of Child Health that
claimed to detect a “gene for sensitivity” fell into this trap. The researchers
administered a questionnaire, correlated all of the answers with likely genetic
or cultural influences, and claimed that the sensitivity of human
beings—how kind they are to others, how “feminine” their approach to the
world is—is genetically determined. The sensible prediction is that these
correlations will not be repeated in future studies. If that’s wrong, then the
alleged gene starts to look more interesting—but not yet.
The alleged decline in the human sperm count may be another example of
selective reporting. Not by Niels Skakkebaek’s team at Copenhagen University
which published the first widely accepted evidence for a decline in 1997. The
selective reporting was done by researchers who had contrary evidence but didn’t
publish it because they thought it must be wrong, by journal referees who
accepted papers that confirmed a decline more often than they accepted those
that didn’t, and by the press—who strung together a whole pile of
sex-related defects in various parts of the animal kingdom into a single
seamless story, unaware that each individual instance had an entirely reasonable
explanation that had nothing to do with falling sperm counts and often nothing
to do with sex. Sexual abnormalities in fish kept in water from sewage treatment
plants, for instance, are probably due to excess nitrites—which all
fish-breeders know cause abnormalities of all kinds—and not to
oestrogen-like compounds in the water, which would bolster the “sperm count”
story.
The message here is that when you are estimating statistical significance,
you must tailor your choice of sample space to the experiment as it was actually
performed, not a selectively reported version. The safest way to do this is to
discard the data that led to your particular result, and repeat the experiment
to get new data. But even then you must not let the coincidence, the clump,
choose the sample space for you—if you do, you’re ignoring the surrounding
space of near-coincidences.
We decided to test this theory on a recent trip to Sweden. On the plane, Jack
predicted that a coincidence would happen at Stockholm airport. His reason:
selective reporting. If we looked hard enough, we’d be sure to find one. We got
to the bus stop outside the terminal, and no coincidences had occurred. But we
couldn’t find the right bus, so Jack went back to the enquiries desk. As he
waited, someone came up next to him—Stefano, a mathematician who normally
lived in the office next door to Jack’s at Warwick University.
Prediction confirmed. But what we really wanted was to lay hands on a
near-coincidence—one that hadn’t happened, but could have been selectively
reported if it had. For instance, if some other person we knew had shown up at
exactly the same time, but on the wrong day, or at the wrong airport, we’d never
have noticed. So for all we knew, Sweden could be overrun by acquaintances of
ours, making it virtually impossible for us not to bump into one of them at
almost any moment. Near-coincidences, by definition, are hard to observe. But we
happened to mention all of the above to Ian’s friend Ted, who visited soon after
we got back to England. “Stockholm?” said Ted: “When?” And we told him. “Which
hotel?” “The Birger Jarl.” “Funny, I was staying at the Birger Jarl one day
later than you!” Had we travelled one day later then, we wouldn’t have met
Stefano—but we would have met Ted. Selective reporting would have ensured
that we only told our friends about the one that actually happened.
Probability theory assesses how likely an event is in comparison to others
that could have happened, but somehow or another didn’t. Our intuition for
probabilities is poor because the feature-detection system in our brains notices
only the things that happen. In the world around us, every event is unique.
Every meeting, every sex ratio, every bridge hand. “What, your telephone number
is nearly the same as your car registration? How amazing!” But when you realise
that a typical citizen has several dozen significant numbers (address, postcode,
fax, mobile, credit cards . . .) a chance resemblance between some two of them
is surely mundane, rather than remarkable.
Wonderful rain
What we must not do, then, is to look back at past events and find
significance in the inevitable few that look odd. That is the way of the
pyramidologists and the tea-leaf readers. Every pattern of raindrops on the
pavement is unique. We’re not saying that if one such pattern happens to spell
your name, this is not to be wondered at—but if your name had been written
on the pavement in Beijing during the Ming dynasty, at midnight, nobody would
have noticed. It’s no use looking at past history when assessing significance:
you need to look at all the other things that might have happened instead.
Every actual event is unique. Until you place that event in a category, you
can’t work out which background to view it against. Until you choose a
background, you can’t estimate the event’s probability. On the other hand, if
something that seems spooky to you really does turn out to have a small sample
space, that’s when you should be really amazed.