YOU ARE about to set off into town to do some shopping. You will only be out for an hour or so, but rain has been forecast, so what are you going to do? You know the forecasts are pretty good-around 80 per cent accurate, in fact. So the chances that you will need an umbrella are 80 per cent, right? Wrong, they鈥檙e actually more like 30 per cent.
This strange result has nothing to do with forecasters making exaggerated claims, or with the innate cussedness of the climate. It鈥檚 an example of a curious mathematical effect that can trip up any attempt to make sense of uncertain data, from forecasts of relatively trivial matters such as rainfall to predictions of earthquakes or testimony of witnesses in a murder trial.
Lurking behind all of these cases is the 鈥渂ase-rate effect鈥�. This, put simply, is the effect that the chances of an event occurring at all-its base-rate-have on our ability to predict it. It comes into its own whenever you are trying to predict something that is rare. Then, even when you think you are making very accurate predictions, your correct forecasts of the rare event can be swamped by a huge number of failures.
Advertisement
Forecasts of rain are a classic example. Since forecasts of rain by the British Meteorological Office are currently around 80 per cent accurate, it does seem obvious that 8 times out of 10 you would expect that the rain they forecast will fall. But the reason that鈥檚 wrong is that it ignores the overall chances that rain falls at all-the base-rate of rain.
A moment鈥檚 thought shows how crucial this is. Even the most dimwitted forecaster could predict rainfall with staggering accuracy over Chile鈥檚 Atacama desert: the base-rate of rain is so low-one squall every few decades-that all you need do to get a virtually 100 per cent forecasting record is to say: 鈥淣o rain tomorrow, folks.鈥�
For the fickle British climate, things are a bit more complicated, but the base-rate still has a dramatic effect on the reliability of forecasts. On the hourly timescales relevant to shopping trips, Britain鈥檚 base-rate of rain is about 0.1-that is, there is only a 1 in 10 chance of rain falling in any particular hour, and thus a 9 in 10 chance of rain not falling. And this has a significant impact on how much trust we can put in even an 80 per cent reliable forecast.
To see why, suppose you make a hundred 1-hour trips during a year. The base-rate of rain means that on 90 (100 x 0.9) of the trips, the weather will be fine, while rain will fall on the other 10. And of these 10, a forecast accuracy of 80 per cent means the forecasters will correctly predict rain on 8 trips.
But 80 per cent accuracy also means that the forecasters make mistakes on 20 per cent of occasions-so that they鈥檒l wrongly forecast rain on 20 per cent (18 trips) of the 90 dry trips. That makes a grand total of 26 trips when they forecast rain, of which just 8 will prove correct. So in spite of the apparently high rate of accuracy, rain will only fall on 30 per cent of the trips for which they forecast rain.
Growing errors
In effect, the large number of dry hours has magnified the relatively small errors in the forecast accuracy until they swamp the correct forecasts (see 鈥淕etting to grips with rare events鈥�). All this has clear implications for the vexed question of when to take weather forecasts seriously. In a recent note in Nature, I showed that if you鈥檙e only going out for an hour or two and you鈥檙e not totally paranoid about getting wet, the optimal strategy is never to carry an umbrella-even if the Met Office predicts a downpour. You just won鈥檛 be out long enough to give decent odds of getting drenched.
As well as helping to make better umbrella-carrying decisions, the base-rate effect may also resolve the paradox of why even today鈥檚 sophisticated weather forecasts are still regarded with suspicion. In a paper just out in Mathematics Today, I suggest that this is because people are rarely out and about long enough to notice when forecasts of rain do prove correct. Certainly anyone planning to spend the whole day outside should take Met Office forecasts very seriously: the daily base-rate of rain-0.4 or a 4 in 10 chance of rain falling-is so high that forecasts of rain falling during your outing are almost three times more likely to be right than wrong. It鈥檚 simply that the more frequent an event, the easier it is to forecast accurately.
Put like that, the base-rate effect seems pretty intuitive and obvious-which has prompted psychologists to ask why we are so bad at dealing with it. Pioneering studies of human ability to deal with uncertainty by the late Amos Tversky at Stanford University and Daniel Kahneman at Princeton University have long been held to prove that we鈥檙e all rotten at dealing with probability in general, and base-rate effects in particular. A classic demonstration often wheeled out is the so-called Cab Problem.
Colour confusion
This centres on a night-time 鈥渉it-and-run鈥� accident involving a cab in a town where there are just two cab operators-one with green cars, the other with blue. A witness who saw the accident said that the cab was blue, and in scientific tests to judge how well she could pick out the colour of a cab, the witness proved correct 80 per cent of the time. So, if 15 per cent of the cabs are blue, what are the chances that the witness is right about the colour of the cab?
Knowing nothing about the base-rate effect, it鈥檚 tempting to give an answer around 80 per cent-which is just what people tend to do. But the correct answer-easily found using a contingency table-is just 41 per cent. The relatively large number of green cabs means that the witness鈥檚 mistaken identifications of these cabs as being blue swamp her successes at spotting cabs that really are blue. Result: the police would do better tossing a coin to discover the culprit.
It is easy to dismiss such problems as mere mind games, but doctors making life and death decisions also seem to have trouble with base-rates. In a study of Harvard Medical School published in 1978, 60 staff and students were asked to estimate the chances that a patient who tests positive for a disease known to affect 1 in 1000 people actually does have it, given that the test gives false positive results in only 5 per cent of cases.
About half the medics seemed totally unaware of the base-rate effect, and said that they were 95 per cent certain that the patient had the disease. Less than 20 per cent gave the correct result: the chances that the patient has the disease are just 1 in 50.
Worryingly, other studies of medics have found similar muddle-headedness-suggesting that ignorance of the base-rate effect could be leading to massive levels of over-treatment following screening programmes. But some psychologists are now beginning to ask if such studies can be taken at face value. In a major review of the base-rate issue published recently in the American journal Behavioral and Brain Sciences, Jonathan Koehler of the University of Texas, Austin, argued that many of the supposed demonstrations of incompetence say more about how the questions were posed than about our ability to handle probability.
He points out that academic riddles such as the Cab Problem are ambiguous, For example, you might argue that the important base-rate is not that of blue cabs in town, but of cabs involved in night-time accidents. After all, there may be fewer blue cabs, but they may have a worse safety record. But this base-rate isn鈥檛 given-which may prompt those given the cab problem simply to guess. If you don鈥檛 give people information they think is relevant, says Koehler, it is hardly surprising if they make blunders.
Ask the right question
He adds that recent studies suggest that people are much better at dealing with base-rates if the questions are re-cast in 鈥渇requentist鈥� terms. So instead of asking people to estimate the probabilities of, say, a particular patient having a disease in the light of a test result, but instead how many patients out of 100 are expected to have the disease.
This subtle change in wording can make all the difference, it seems. In research published last year by Leda Cosmides and John Tooby of the University of California, Santa Barbara, students were given a disease diagnosis problem similar to that posed in the Harvard study. It turned out that the students were as bad as the Harvard medics when the problem was stated in terms of probabilities. But when the question was re-cast in the frequentist form of how many patients out of 100 would really have the disease, most of the students came up with perfectly sensible answers.
Such findings have obvious-and serious-implications for the way doctors are trained to interpret test results. The old adage that one should 鈥渄iagnose rare diseases rarely鈥� isn鈥檛 a bad one, but it hardly makes the most of today鈥檚 sophisticated screening methods-and would not impress a jury in a medical negligence case.
Juries and judges, themselves often confronted with probabilistic evidence, would also benefit from a better approach to the base-rate effect. One form of evidence that is particularly vulnerable to the base-rate effect is DNA profiling. Many legal experts are worried about the way such evidence is presented in court, with judges, juries and forensic experts misunderstanding the true meaning of the DNA match probabilities (鈥淚mproving the odds on justice?鈥�,快猫短视频, 16 April 1994, p 12-13).
Yet even where the evidence is presented impeccably, ignoring the base-rate effect can lead to miscarriages of justice. If there is very little other evidence against the accused-that is, the base-rate of guilt is very low-then even apparently impressive DNA match probabilities may not totally eliminate 鈥渞easonable doubt鈥�.
Return to reality
Our long-suffering Met Office might also benefit from making more of the base-rate effect. Considering the intrinsic difficulty of weather forecasting, the Met Office does a pretty good job of predicting the great British weather. It might get more credit for its hard work by incorporating the base-rate effect into its forecasts if forecasters used phrases such as: 鈥淵ou鈥檒l probably escape the rain if you are only out for a short while.鈥�
While a better understanding of the effect would obviously improve weather forecasts, court evidence and hospitals鈥� diagnoses, its central lesson-that rare events are hard to predict-could also save us from spending millions on what may well be scientific wild goose chases.
Take the case of earthquake forecasting. Over the past century, seismologists have invested a vast amount of money and effort in trying to predict earthquakes-without a single unequivocal success. Even so, researchers press on doggedly with the search for tell-tale 鈥減recursors鈥� which could warn us of impending 鈥淏ig Ones鈥�.
The base-rate effect strongly suggests that such persistence is completely misguided. Predicting major, Kobe-style, quakes would indeed be marvellous, but such quakes are very rare-about one every 50 to 100 years. This very low base-rate means that any precursors of a quake would have to be incredibly reliable if decision-makers are to feel safe in using them to order mass evacuations.
Rough estimates suggest that any useful quake forecast method demands precursors that will prove correct 98 per cent of the time. To date, no so-called quake precursor has achieved anything like that degree of reliability. And, as evidence mounts that quakes, like avalanches, are intrinsically unstable 鈥渃ritical鈥� phenomena, the prospects that such reliable precursors will ever be found become ever smaller.
Long trapped in the psychological research literature, the base-rate effect is more than just a subtle logical trap to delight academics and smart-alecks. Understanding how it works can help us all make better decisions-and show where we might be overreaching ourselves.
* * *
Getting to grips with rare events
THE chances that a prediction of an event-such as a shower of rain, a disease, or an earthquake-will prove correct can be calculated algebraically using probability theory, but it is tedious and not very enlightening. 鈥淐ontingency tables鈥� are a faster and clearer way. In their simplest form, these are simply boxes with two possible states as the columns-for example, rain or no rain-and the corresponding predictions-rain forecast and no rain forecast-as the rows. After a couple of simple calculations, you can fill the whole box in, and read off whatever probabilities you鈥檙e interested in.
Take the case of rain forecasts. There are two key figures: the hourly base-rate of rain-0.1 or 1 in 10-and the forecast accuracy of 80 per cent. This means that out of 100 one-hour trips, it will rain on 10 of them, and stay dry on the other 90-giving the whole of the bottom line of the contingency table.
Now, going down the first column headed 鈥渞ain鈥�, we鈥檙e told that when it does rain, the forecast will have predicted the rain correctly 80 per cent of the time. So, of the 10 rainy trips, 8 (10 x 0.8) of them will have been correctly forecast, while the forecast will have predicted no rain on the remaining 2 trips. The first column is now filled.
Similarly, of the 90 trips with no rain, the forecast will have predicted no rain on 72 (90 x 0.8) of them, while on the remaining 18 the forecast will be wrong. Once these figures are entered in the second column, the whole table can be completed, and a range of information read off.
For example, reading across the top row, you can see immediately that rain will be forecast to fall in 26 out of the 100 hours, but of these forecasts only 8 will prove correct-a hit-rate of just 30 per cent. In contrast, of the 74 forecasts of no rain, 72 will prove correct-a 97 per cent hit-rate. Any forecasting system, from quake prediction to cancer screening, can be analysed in the same way once the base-rate and accuracy figures are given.
