NOBODY likes to be ridiculed, but for some people it can become a matter of life and death. Take Robert FitzRoy, the founding father of the UK鈥檚 Meteorological Office and captain of the Beagle during Charles Darwin鈥檚 five-year voyage. A keen amateur forecaster, he enthusiastically applied the science of his day to weather prediction. Much good did it do him. Instead of hailing his tentative prognostications as a useful first step, politicians, newspapers and other scientists harangued and mocked FitzRoy whenever he got it wrong. Depression quickly set in, to fatal effect. One Sunday morning in 1865, FitzRoy cut his throat in despair.
These days, most meteorologists have got used to being the target of jokes. The criticisms are the same, though. People expect weather forecasts to be accurate. In this age of weather-forecasting supercomputers and 24-hour satellite surveillance, what鈥檚 more, those expectations of accuracy have risen to new heights. The forecasters, of course, say their predictions are more accurate than ever before. Can we tell if they鈥檙e right?
Though theories abound, no one 鈥� not even the meteorologists 鈥� can agree on the best way to measure a forecast鈥檚 accuracy. Forecasters produce a whole slew of predictions every day, and those predictions can be checked against hard drives full of weather data. The problem is how best to put the two side by side to see how good the forecasts are. 鈥淎 lot of the standards we use were developed more than 100 years ago,鈥� says Barbara Brown, an expert on forecast verification based at the US National Center for Atmospheric Research in Boulder, Colorado. 鈥淚t鈥檚 really sad.鈥� It鈥檚 time, she says, for meteorologists to put their house in order 鈥� something she hopes to work towards this month when meteorologists from around the world will gather in Reading, UK, to talk about methods of forecast verification. At the moment, odd as it may seem, the best way forward may be to go back to the 鈥渨arts and all鈥� way forecasts were presented 100 years ago.
Advertisement
鈥淭he best way forward may be to go back to the 100-year-old 鈥榳arts and all鈥� approach鈥�
Forecasting the weather is a huge undertaking. The computer programs that provide the raw materials for modern weather forecasts employ complex arrays of equations that take a set of initial conditions 鈥� variables such as air pressure, temperature, ocean current speeds, humidity and so on 鈥� and use theories such as fluid dynamics and thermodynamics to work out what sort of weather those initial conditions will evolve into after a given time. To make accurate predictions the information has to be gathered from as many points on, above and below the surface of the Earth as possible.
However many weather stations you have they will never be enough. The famous butterfly effect of chaos theory, which can turn a tiny air movement in one part of the globe into a storm thousands of miles away, is all too real: minuscule errors or omissions in the initial conditions can blow up into hugely erroneous forecasts. That chaotic behaviour is why two and three-day forecasts are so much more reliable than five-day forecasts: those extra few days are enough for the weather at a given location to develop in a completely different way.
If two or three days doesn鈥檛 sound like much, consider that in FitzRoy鈥檚 day, when little more than personal experience was available to turn weather data into forecasts, a one-day forecast pushed the limits of what was possible; it was an attempt at a two-day forecast that was FitzRoy鈥檚 undoing.
Forecasting has moved on a very long way since then. Ever more powerful supercomputers are ingesting ever more detailed data from satellites and weather stations as the initial conditions for their calculations. Over time, our understanding of the processes behind the weather 鈥� the way oceans and the atmosphere interact, for example 鈥� has improved, making computerised climate models more realistic. FitzRoy鈥檚 brainchild, now called the Met Office, claims its three-day forecasts are now as accurate as its one-day forecasts were 20 years ago. That doesn鈥檛 mean, however, that all the forecasters鈥� predictions are useful.
Indeed, when it comes to longer-range predictions looking months ahead, some experts doubt there is any value to them at all. In December, Pascal Mailier, a meteorologist at the University of Reading, told an audience of European energy-industry executives that when they buy certain long-range predictions to assess future energy demand, they might be wasting their money. The Met Office, for instance, sells long-range forecasts as well as providing public information. Last July it issued its early predictions for the British winter, based on temperature measurements in the North Atlantic Ocean. This 鈥淣AO signal鈥� is supposed to give an indication of whether the winter will be wet and mild, or dry and cold. Mailier鈥檚 analysis says those who bought this information might have been better off saving their cash and just casting their minds back a few months: it is just as instructive to assume that this winter will be the same as last year鈥檚, Mailier says.
Keith Fenwick, a forecaster at the Met Office, disputes Mailier鈥檚 analysis. The NAO signal is more complex than Mailier makes out, he says, and provides more information than a simple persistence forecast can. It can, for instance, make good predictions of year-to-year changes in winter surface temperatures.
Not all assessments of forecasting skill are so contentious, but they are still largely subjective, as forecasters assess themselves using whatever criteria they choose. The Met Office, for example, uses something called a numerical weather prediction (NWP) index. This starts off with a set of 鈥渟kill scores鈥�, which gauge how well a forecast compares with weather-station measurements of rainfall, surface temperature, visibility, total cloud cover and wind. The assessors then combine the skill scores for each of these parameters into one result for each month鈥檚 forecasts for the climate model they are using.
The monthly score does not provide a good measure of success by itself. Local effects at the weather stations create huge variations, says the Met Office鈥檚 Simon Fuller. That鈥檚 because, in order to get forecasts out in a reasonable time, the weather computers operate on a limited resolution: they give single predictions for the rainfall, wind speed, temperature and air pressure for a region of, say, 4 square kilometres, because it would take too long to calculate more detailed forecasts. The forecasters know that the weather will vary in microclimates across that area: a farm in the shelter of a hill, for instance, might get less rain than is forecast for the whole region on a particular day. Only by averaging out the observations over a long period can you get a true reflection of the accuracy of the predictions, and if you smooth the anomalies out across 36 months, Fuller says, you can see that the overall forecast skill is improving steadily (see Graph).
Not all meteorologists are convinced that the NWP index is a good measure of short-range forecasting skill. Though it gives some indication of how things are going, the NWP index is 鈥渁 bit simplistic鈥�, says Beth Ebert of the Bureau of Meteorology Research Centre in Brisbane, Australia. She thinks measuring the accuracy of a forecast requires taking its end users into account: airports need accurate fog measures, for example, rather than accurate measures of sunshine; hurricane forecasters need to know wind speeds and the location and timing of landfall, rather than cloud height.
The truth is, she says, there isn鈥檛 a straightforward way to assess how forecasters are doing 鈥� but try telling that to a forecaster鈥檚 boss. 鈥淧eople prefer a short answer,鈥� Ebert says, 鈥渆ven when the short answer is clearly inadequate.鈥� It can be worse than inadequate, too 鈥� the pressure to meet an oversimplified 鈥渁ccuracy鈥� target can change the way forecasters interpret their data, altering the prediction. 鈥淚t can warp the forecasting process,鈥� says meteorologist David Stephenson at the University of Reading. That鈥檚 because forecasters can improve their numbers by 鈥渉edging鈥�, he says: not making forecasts based on what their models actually predict, but arbitrarily reining in any perceived extremes, blurring the location of a weather front, or reducing the probability of rain from 50 per cent to 20 per cent. All these moves can improve a forecast鈥檚 verification scores 鈥� and, at the Met Office, a forecaster鈥檚 annual bonus 鈥� but they will also make it less useful.
Head in the sand
There鈥檚 another, less cynical reason why forecasters sometimes don鈥檛 pass on what the computer says: because we don鈥檛 want to hear it. Television forecasts, for example, rely on phrases such as 鈥渟cattered showers in the west鈥� and 鈥渨indy in the north-west later鈥�. But think about it: what do these phrases mean? 鈥淲indy in the north-west later鈥� might be a simplification of the computer鈥檚 prediction that 鈥渁t some time in a 12-hour period mean wind speed at station A in the north-west will exceed a specified threshold鈥�; 鈥渇rost is likely鈥� may mean 鈥渢he probability of frost exceeds 70 per cent鈥�. The trouble is, no one wants to digest those details over breakfast, so they are boiled down to a set of vague predictions. Unfortunately, the media-issued forecasts most of us run our lives by are so vague they are 鈥渋mpossible to verify with objectivity鈥�, according to a report published last year by the UK鈥檚 Royal Meteorological Society. 鈥淎ny claim for skill of descriptive forecasts should be treated with scepticism,鈥� the authors say.
鈥淭he media-issued forecasts most of us run our lives by are so vague they are impossible to verify with any objectivity鈥�
To make that scepticism melt away, TV forecasters could start by doing a very simple thing: embrace uncertainty and start giving us a precise indication of their confidence. 鈥淯ncertainty is鈥� a fundamental characteristic of weather,鈥� says a report issued by the US National Research Council late last year. 鈥淣o forecast is complete without a description of its uncertainty.鈥�
The report recognises that things have got worse, not better, in this respect. In FitzRoy鈥檚 time, some forecasts were called 鈥減robabilities鈥�, and included specific information on uncertainty. Not any more. 鈥淒ecisions by users at all levels, but perhaps most critically those associated directly with protection of life and property, are being made without the benefit of knowing the uncertainties of the forecasts upon which they rely,鈥� the report says. The US National Weather Service now recognises that being clear about uncertainty has to become a fundamental part of its forecasts again.
The point, after all, is to give forecast information that helps people make decisions. It鈥檚 not that TV forecasts are useless; for most people they are a valuable guide to the next day鈥檚 weather. However, that value would increase dramatically if the forecasters were willing to add an indication of probability alongside a prediction. Should you take your umbrella out with you in the morning? You get a much better idea of the answer from 鈥渢here is a 70 per cent chance of rain鈥� than 鈥渢here is a chance of some rain鈥�.
Forecasters have been reluctant to go down the probabilistic route, though. Ten years ago, a study led by J. Frank Yates of the University of Michigan, Ann Arbor, showed that the general public viewed people giving a probabilistic forecast as incompetent, ignorant or even lazy (International Journal of Forecasting, vol 12, p 41). A 50 per cent chance of rain, for instance, gets misinterpreted as 鈥淚 don鈥檛 know what the hell is going to happen tomorrow.鈥�
鈥淭here is a misconception that a 50:50 probability is complete ignorance,鈥� says Met Office meteorologist Mark Roulston. That鈥檚 only true if you know the event you鈥檙e talking about occurs exactly half the time, but if it normally occurs less frequently, then being told there鈥檚 a 50:50 chance is telling you there鈥檚 been an increased likelihood. So, a 50 per cent chance of rain is worse than, say, a 10 per cent chance; without this information, making a decision based on the chance of rain is like placing a bet at the bookies without knowing the odds.
The National Research Council report makes a similar point, but says that forecasters will just have to live with any accusations of ignorance: they have been getting away with an unjustifiable air of certainty for too long, anyway.
Roulston is now working with psychologists at the University of Exeter, UK, to establish how uncertainty information might best be presented. He has been watching people play games in which they pretend to run a road-gritting company. The grit costs money but the local authority will fine the company if it fails to grit when the temperature falls below freezing; the company profits depend on good weather forecasting. Roulston鈥檚 results show that being equipped with a temperature forecast plus its margins of error puts the user at a great advantage. Adding in the probability that the temperature will fall below freezing makes little difference: the margins of error are enough, it seems (Weather and Forecasting, vol 21, p 116).
Unscrupulous forecasters
Of course, providing explicit information about the uncertainty of a forecast is a dangerous thing in an unregulated market: what if the utilities company or supermarket is approached by a forecasting service that offers a lower error level or 鈥� something that no one can do honestly with any accuracy 鈥� predictions out to 10 days? Might unscrupulous companies be tempted to claim a higher-than-realistic accuracy? The answer, from current experience, is yes. 鈥淭here are companies here in the US that offer 10-day forecasts, updated hourly,鈥� Brown says. 鈥淚t鈥檚 nuts 鈥� but people do buy them.鈥�
Perhaps that鈥檚 why some forecast providers are averse to a big, open discussion of forecast verification and quality issues. In the course of their research Stephenson, Mailier and two of their Reading colleagues issued questionnaires to organisations providing forecasts, and asked them to forward the questionnaires to the people who used their forecasts. Many simply refused to take part. 鈥淔orecast providers and their users have rather intimate, special relationships,鈥� Stephenson says. 鈥淚 am not sure all the providers want their users to know that there are better providers or how to judge and criticise the forecasts.鈥�
Whatever the truth, all kinds of prediction may be about to get even more difficult, thanks to climate change. Though no one is sure exactly what the effects will be, it seems that extreme weather events such as storms and hurricanes are likely to become more common, in many parts of the world (快猫短视频, 20 January, p6). Such events have far-reaching effects on distant weather systems, making general forecasting much harder. Though our models are getting better and satellite data is more comprehensive and accurate, we may soon be like the Red Queen in Alice through the Looking Glass, running to stand still. For forecasters, the only answer will be to go back to the way forecasting was done in FitzRoy鈥檚 day: with doubts and disclaimers. All the weather satellites in space, it seems, can鈥檛 compensate for some good old-fashioned honesty.
