快猫短视频

LISTENING to When I鈥檓 Sixty-Four by The Beatles can make you younger. This miraculous effect, dubbed 鈥渃hronological rejuvenation鈥�, was revealed in the journal Psychological Science in 2011. It wasn鈥檛 a hoax, but you鈥檇 be right to be suspicious. The aim was to , simply by picking and choosing methods and data in ways that researchers do every day.

The paper caused a stir among psychologists, and has become the most cited in the journal鈥檚 history. The following year, Nobel prizewinning psychologist Daniel Kahneman stoked the fire with an open email to social psychologists warning of a 鈥渢rain wreck鈥� if they didn鈥檛 clean up their act. But things only came to a head last year with the . It described a major effort to replicate 100 psychology experiments published in top journals. The success rate was little more than a third. People began to talk of a 鈥渃risis鈥� in psychology.

In fact, the problem extends far beyond psychology 鈥� dubious results are alarmingly common in many fields of science. Worryingly, they seem to be especially shaky in areas that have a direct bearing on human well-being 鈥� the science underpinning everyday political, economic and healthcare decisions. No wonder the whistle-blowers are urgently trying to investigate why it鈥檚 happening, how big the problem is and what can be done to fix it. In doing so, they are highlighting flaws in the way we all think, and exposing cracks in the culture of science.

Science is often thought of as a dispassionate search for the truth. But, of course, we are all only human. And most people want to climb the professional ladder. The main way to do that if you鈥檙e a scientist is to get grants and publish lots of papers. The problem is that journals have a clear preference for research showing strong, positive relationships 鈥� between a particular medical treatment and improved health, for example. This means researchers often try to find those sorts of results. A few go as far as making things up. But a huge number tinker with their research in ways they think are harmless, but which can bias the outcome.

This tinkering can take many forms (see 鈥�To err is human鈥�). You peek at the results and stop an experiment when it shows what you were expecting. You throw out data points that don鈥檛 fit your hypothesis 鈥� something could be wrong with those results, you reason. Or you run several types of statistical analysis and end up using the one that shows the strongest effect. 鈥淚t can be very hard to even see that biases might be entering your reasoning,鈥� says psychologist Brian Nosek at the University of Virginia in Charlottesville, who led the team trying to replicate 100 psychology studies. Take the tendency to scrutinise results that don鈥檛 fit with your predictions more carefully than those that do. 鈥淭here鈥檚 no nefarious motive,鈥� says Roger Peng at Johns Hopkins University in Baltimore, Maryland. It鈥檚 just natural to assume these results are likely to be 鈥渨rong鈥�.

You might think that journals, which get peers from the same scientific field to review papers, would pick up on such practices. But, say critics, the system isn鈥檛 up to the task. For one thing, most journals don鈥檛 ask researchers to give them a tour of their statistical sausage factory. 鈥淭he vast majority don鈥檛 require that you make any data available beyond a brief description of the methods,鈥� says Peng. Peer-reviewers usually don鈥檛 see the complete data and methods either. And even if they did, they might not have the time, ability or inclination to check them. Refereeing is unpaid and anonymous 鈥� so there鈥檚 no reward and no recognition in it.

鈥�In a major effort to replicate 100 psychology experiments the success rate was little more than a third鈥�

All this helps explain why so many studies don鈥檛 hold up when others try to replicate them. But it doesn鈥檛 explain why psychology in particular is facing a 鈥渃risis鈥� right now. There鈥檚 nothing new about researchers being subconsciously committed to proving their own theories, or journals favouring headline-grabbing research. Sure, the pressure on researchers to publish is ever greater, however, what鈥檚 really new is the scrutiny being given to their published findings.

Traditionally, once results are published they tend to go unchecked. 鈥淭he current system does not reward replication 鈥� it often even penalizes people who want to rigorously replicate previous work,鈥� wrote statistician John Ioannidis of Stanford University in California entitled 鈥淗ow to make more published research true鈥�. Proponents of a new discipline called metascience (the science of science) aim to change that, and .

Psychology may have borne the brunt of the controversy so far, but Ioannidis has for a long time argued that the problem is widespread. In 2005, he claimed that sloppy methods could mean . Some fields of research are less susceptible than others, though. In astronomy, chemistry and physics, for instance, 鈥減eople have a very strong tradition of sharing data, and of using common databases like big telescopes or high energy physical experiments鈥�, Ioannidis says. 鈥淭hey are very cautious about making claims that eventually will be refuted.鈥� But in fields where such checks and balances are absent, irreproducible results are rife.

Take the case of cancer researcher Anil Potti when he was at Duke University in Durham, North Carolina. In 2006, staff at the MD Anderson Cancer Center in Houston, Texas, wanted to investigate treatments based on Potti鈥檚 published work on gene expression. Before pressing ahead, they asked their colleagues, biostatisticians Keith Baggerly and Kevin Coombes, to look over the findings. Their efforts illustrate how hard it can be for peer reviewers to pick up on mistakes. It took them almost 2000 hours to disentangle the data and reveal a catalogue of errors. It later transpired that , but in the meantime, three clinical trials had been started on the basis of his research.

Evidence is mounting that medical research is particularly prone to irreproducibility. In 2012, Glenn Begley, a biotech consultant, showed that . estimates that irreproducible preclinical research costs the US $28 billion a year and slows down the development of life-saving drugs. 鈥淭he truth is everyone knew that this was a problem,鈥� says Begley. 鈥淣o one really knew the magnitude of the problem.鈥�

Dodgy statistics

It鈥檚 the tip of the iceberg. by Megan Head of the Australian National University in Canberra and her colleagues showed that dodgy statistics are rife in the biological sciences. They scrutinised results from a wide range of scientific disciplines for evidence for 鈥減-hacking鈥� 鈥� collecting or selecting data or statistical analyses until non-significant results becomes significant. They found it to be particularly common in biological sciences. 鈥淎 lot of biologists go into biology because they don鈥檛 want to do maths, and then they get a rude shock when they learn they have to do statistics,鈥� says Head.

鈥�Sloppy methods could mean that over half of all published scientific results are flawed鈥�

But even mathematicians make errors. In 2010, economists Carmen Reinhart and Kenneth Rogoff at Harvard University published research showing that when a country鈥檚 debt reaches more than 90 per cent of GDP there is an associated plunge in economic growth. The paper, which appeared in a non-peer-reviewed edition of the American Economic Review, was seized on by politicians in the UK and US . However, three years later, when Thomas Herndon, a graduate student at the University of Massachusetts, Amherst, tried to replicate the findings, he ran into trouble. . The effect they had observed had, according to critics, been largely a mirage. Nevertheless, it had a major impact on the public policy debate.

Given how influential a flawed paper can be, it鈥檚 no wonder people are up in arms. One obvious concern is that it could undermine public faith in science itself. 鈥淚t could very quickly become a wave of mistrust of the kind we find associated with climate change,鈥� says psychologist Nicholas Humphrey at the London School of Economics. Drawing an analogy with the global financial collapse of 2008, he calls it 鈥渟ub-prime science鈥�. 鈥淎fter the disgrace of the bankers, science must not be next,鈥� .

So what can be done? There has already been a rapid response in one area of research where irreproducible results can have life-or-death consequences. Since 2005, a group of major medical journals has required researchers to , and the methods they intend to use, before recruiting patients. Ioannidis estimates that about half of all clinical trials now are pre-registered, vastly reducing the possibility of flawed work.

Psychologists have also taken matters into their own hands. In 2011, the authors of the When I鈥檓 Sixty-Four paper 鈥� Joseph Simmons and Uri Simonsohn of the University of Pennsylvania and Leif Nelson of the University of California, Berkeley 鈥� met with Eric Eich, the newly appointed editor of Psychological Science, to discuss the problems facing their discipline. 鈥淭hat was really eye-opening for me,鈥� says Eich. 鈥淭here were a lot of things that were essentially broken.鈥�

In January 2014, the journal began asking researchers more questions about their methods and giving them more space to explain them. It also introduced a 鈥�nudge鈥� to reward good practice by displaying badges on papers to recognise those who made data and methods available or pre-registered their study. The result? Submissions fell off a cliff. 鈥淚 thought I had broken the damn journal,鈥� says Eich. However, after five months, submission rates were back to normal, and now some 40 per cent of new Psychological Science papers have open data 鈥� up from 3 per cent before the badges were introduced.

Now the idea is being rolled out. Last year, Nosek and his colleagues came up with to increase transparency and reproducibility. by the US National Institutes of Health, and adopted by more than 500 journals, including Science, and 50 organisations. . Meanwhile, the Center for Open Science, co-founded by Nosek, has established a free online platform, the Open Science Framework, where researchers can register studies and display all their data and methods. More radically, to replace peer reviewers with paid experts 鈥� accredited specialists in the analysis of research.

Quality not quantity

Universities may join the movement too. Ioannidis and others are working to create a 鈥渃oalition of university leaders鈥� to address the problem. 鈥淯niversities are the gatekeepers of promotion and tenure,鈥� he says. 鈥淚 hope that we will be moving pretty soon on that front.鈥� One obvious solution is to stop rewarding scientists on the basis of how much they have published 鈥� to consider quality not quantity when making academic promotions.

Ultimately, we may need to create novel ways of determining which studies are valid. Working with Nosek鈥檚 team, asked psychologists to place bets on which studies would stand up and which wouldn鈥檛. 鈥淚t turned out that in predicting the outcome of the replications,鈥� says Anna Dreber Almenberg at the Stockholm School of Economics in Sweden, who leads the project. Such an approach could be harnessed to help identify iffy results before they are accepted for publication. It鈥檚 still early days, but Dreber Almenberg says that prediction markets 鈥渃ould be interesting to think more about鈥�.

Meanwhile, replication projects are gaining popularity. Groups are now looking at and experimental economics. One member of the economics group, Colin Camerer at the California Institute of Technology in Pasadena, says the project, which published in March, has been greeted with enthusiasm. 鈥淧eople have been emailing us saying, if you do more, we鈥檒l help you out,鈥� he says.

鈥淚t will take years to play out,鈥� says Eich. 鈥淏ut hopefully at the end of it, you get more replicable, high-quality science.鈥� Given that we fund academic research through our taxes and rely on it to improve our lives, that will be good for everybody.

This article appeared in print under the headline 鈥淭he unscientific method鈥�