OFFICIAL 鈥渓eague tables鈥� ranking the success of Britain鈥檚 schools and
health authorities contain a fundamental flaw which makes them virtually
useless, according to two leading statisticians. The impact of the flaw is so
great that institutions rated among the best performers may in fact be among the
worst鈥攁nd vice versa.
Trumpeted by the government as a reliable source of performance data that
allows parents and patients to make informed choices, league tables have been
controversial since their introduction in 1994. Most criticism has focused on
their failure to take into account such factors as the family circumstances of
the children attending schools, or the condition of patients when they arrive at
hospital.
But according to a detailed analysis published in the current issue of the
Journal of the Royal Statistical Society, the league tables ignore a
more basic statistical effect that renders even adjusted tables almost
meaningless.
Advertisement
In ranking each institution, the league tables take a single number as a
measure of the performance of each institution. One set of tables, for instance,
is based on the A-level exam performance of 18-year-old school-leavers. Rankings
use this one figure: the higher it is, the 鈥渂etter鈥� the school.
While the resulting tables appear easy to understand, they conceal the fact
that the reliability of each quoted figure depends crucially on the size of the
sample on which it is based. Small samples are intrinsically less reliable than
large ones, because random effects have a disproportionate influence on the
overall figure. For example, a school putting just a few dozen pupils through
A-levels can have its ranking ruined by a handful who happen to be suffering
from colds on the day of the exams.
According to the study鈥檚 authors, most schools simply do not have enough
pupils taking exams to counteract the small sample effect. 鈥淔or the 16 to 19 age
range, there are a few schools in the country which have three or four hundred
pupils taking the examinations. Most have just 50 to 100鈥攊t is these that
give problems,鈥� says Harvey Goldstein of the Institute of Education of the
University of London. 鈥淭hey are simply too small to give reliable rankings.鈥�
To illustrate the impact of ignoring the effect of sample size, Goldstein
took A-level performance figures for 10 schools in one local authority and used
standard statistical theory to calculate 鈥渃onfidence intervals鈥�, a type of error
bar, for each figure. It emerged that the confidence intervals on each school鈥檚
performance were so large that they encompassed those of every other
school鈥攎aking it impossible to rank them with any confidence.
Teachers鈥� unions have pointed out that the league tables fail to take into
account the abilities of the pupils at each school. This has led to demands for
the rankings to include a 鈥渧alue-added factor鈥�, showing how successful schools
have been in improving the performance of their pupils.
Yet Goldstein showed that even when attempts are made to compensate for such
effects, the problem of small sample size still renders rankings all but
meaningless. Taking A-level data from 325 schools and colleges and adjusting for
exam performance two years earlier鈥攐ne way of capturing the 鈥渧alue added鈥�
factor鈥攊t emerged that the only clear difference was between schools in
the top 15 per cent and those in the bottom 15 per cent. None of the other 70
per cent could be reliably ranked (see
Diagram).
Health service league tables are also undermined by small sample effects,
according to the study鈥檚 second author, David Spiegelhalter of the Medical
Research Council鈥檚 Biostatistics Unit in Cambridge.
Rates for preventable medical conditions are used to gauge the success of
health authorities in meeting targets set under the government鈥檚 Health of the
Nation initiative. But the catchment areas of many health authorities are so
small that the confidence intervals are huge, again ruling out any hope of
reliable rankings. 鈥淭hese ranking exercises are frankly a waste of time,鈥�
Spiegelhalter told 快猫短视频.
The study has prompted mixed reactions from government officials. Senior
statisticians at the health and education departments insist that the 鈥渞aw鈥�
performance figures are valuable and worth publishing.
鈥淚f the information for the public is to have impact, it must be simple to
understand,鈥� says one statistician at the Department of Health. 鈥淚 agree that
information on limitations and uncertainties should be provided, but how can
this be done so that the public will understand?鈥�