YOU鈥橠 be a fool to believe everything in the papers. Made-up facts, half-baked analysis, hand-picked conclusions 鈥� yes, scientific papers can seem no more reliable than the tabloid press.
In the past month alone, concern over fraud in medical research papers has prompted the retraction of one paper in The New England Journal of Medicine and disciplinary action against the lead author of another, published in the British Medical Journal. In both cases the whistle-blowers were co-authors of the papers who became concerned about the reliability of the research and asked that the papers be retracted.
Some will see this as evidence that the scientific community can police itself. But what if the authors have no idea that what they have done is questionable or misleading? What if they routinely use unreliable methods, with top research journals cheerfully publishing results based on them?
Advertisement
That is the stark prospect highlighted by the epidemic of flawed statistical analysis plaguing today鈥檚 research literature. From medical 鈥渂reakthroughs鈥� that prove to be mirages to grand conclusions drawn from tiny samples, the journals are awash with unreliable findings based on faulty statistics.
The threat this poses to the credibility of science has been pointed out repeatedly over the past 40 years. Yet the warnings have gone unheeded. Take the case of that sine qua non of research, the 鈥渟tatistically significant鈥� result. As long ago as 1963, statisticians at the University of Michigan issued a warning that the methods used to gauge the statistical significance of findings were 鈥渟tartlingly prone鈥� to exaggerate real significance.
At the heart of these methods is the so-called P-value. By time-honoured tradition, P-values below 0.05 are deemed 鈥渟tatistically significant鈥�, as they seem to imply that there is just a 1 in 20 chance of the results being flukes. In fact, because P-values take no account of the inherent plausibility of the hypothesis being tested, it is in fact four times as likely that the result is a fluke. In other words, a substantial proportion of 鈥渟tatistically significant鈥� findings are meaningless flukes 鈥� scientific fool鈥檚 gold that lures others to waste time and money in attempts to replicate findings.
Then there鈥檚 the flip side: the rejection of potentially important ideas because a small study fails to confirm them. Such studies may be quick and cheap, but they also lack statistical power 鈥� that is, they run a high risk of failing to detect real effects.
As with statistical 鈥渟ignificance鈥�, experts have been warning about the dangers of underpowered studies for over 40 years, but you would never know it to look at the current medical literature. A recent analysis of trials of treatments ranging from antidepressants to plastic surgery has shown that many are still far too small to detect clinically worthwhile effects.
That hasn鈥檛 stopped researchers making grand pronouncements off the back of negative results. Last month, the Journal of the Royal Society of Medicine published the results of a study comparing a homeopathic arnica extract, a treatment for post-operative pain, with a placebo. Though the study involved just 62 patients, its failure to find any evidence of benefit led the researchers to describe the treatment鈥檚 supposed efficacy as a 鈥渕yth鈥�. They may well be correct, given that homeopathic treatments have no obvious scientific basis. Yet the trial was so small that it stood barely a 1 in 3 chance of confirming the efficacy of many tried and trusted conventional treatments for post-operative pain.
Small studies are not wholly useless: when taken together in a meta-analysis they can produce scientifically reliable insights. Cynics point out, however, that small studies are also perfect for researchers keen to churn out lots of papers with minimal effort and cost. Certainly, overinterpreting negative results from such trials risks killing off interest in promising new lines of research. One recent study found that of 25 clinical trials in orthopaedic medicine reporting negative results, not one had the statistical power to detect a potentially worthwhile benefit. A similar study of surgical trials found that the authors of fewer than a third of papers reporting negative results bothered to check if their trial was even big enough to detect any benefit.
The editors of some scientific journals have recognised the threat posed by faulty statistical analysis and have tried to clamp down. Many statisticians suspect the real issue is a lack of awareness among working scientists about the limitations of statistical methods. Some have called for professional bodies such as the Royal Statistical Society in Britain and the American Statistical Association to speak out about the routine abuse of statistics in scientific research.
So far, however, their silence has been deafening. Until they join with other scientists to take this issue seriously, the rest of us can surely be excused for dismissing the next headline-grabbing 鈥渂reakthrough鈥� by repeating that old saw about lies, damned lies 鈥� and statistics.