Physics is in a crisis. Over the past year, a series of research papers claimed to find evidence for high-temperature superconductors – materials that conduct electricity without losses, which could revolutionise global energy use. These were retracted after all efforts to reproduce them failed, and in some cases original data was shown to . Now, physicists are sounding the alarm about a wider reproducibility crisis in their field.
“That is just the latest story, it is by far not the only one, or the story that triggered this conference,” says at the University of Pittsburgh in Pennsylvania who co-organised a meeting for researchers, scientific journal editors and funding organisations. “All these stories that we heard are just the tip of the iceberg.”
The took place from 9 to 11 May in Pennsylvania. Across the three days, tensions ran high as researchers openly voiced concerns they may have previously reserved for hallway chatter or post-conference drinks.
Advertisement
The past few years have seen the field of condensed matter physics marred in scandal. Across several talks, researchers described issues with the reproducibility of their peers’ studies. Rather directly, they called for accountability from each other – and from the journals that publish their work and bodies that fund it, such as the US National Science Foundation.
During one panel, at the University of Basel in Switzerland confronted the executive editor at the American Physical Society (APS), . In 2023, an APS journal published a paper by Microsoft’s quantum computing team about an experiment in so-called topological quantum computing alongside a note acknowledging that it did not contain enough detail to be reproduced. Thomas said she stood by that decision and said that APS is continuing to develop policies about how similar cases may be handled in the future, but acknowledged problems with reproducibility in the field more generally.
Related work from the Microsoft team was retracted in 2017 and 2018, and Legg’s research has contradicted Microsoft’s claims. Microsoft independently shared additional data supporting its experimental findings, although Legg says this did not address the issue raised by him and his collaborators. “We are committed to engagement with the community that further scientific progress,” says at Microsoft. He says the firm is “confident in our peer-reviewed research” on topological quantum computing.
at the German research institution Forschungszentrum Jülich detailed a years-long effort he and his collaborators undertook to make the alleged misconduct behind those original retractions public. “I think this is a dire situation, honestly. You start as a normal scientist with some fact-checking and end up with years of non-transparent processes, with no end in sight,” he said.
at the University of Würzburg in Germany echoed the frustration, recounting a time when he identified a blatant case of data manipulation in a paper claiming the discovery of elusive elementary particles called Majorana fermions published in the journal Science. “There were so many irregularities that I could give a 10-hour talk,” he said. “But it took Science four years to retract the paper, and I expected it would be more like four weeks.”
The aim of the conference was to find consensus on how to reckon with physics’ reproducibility crisis, a phenomenon that has been seen in other fields like psychology. But first, says Frolov, everyone had to get their frustrations off their chest. Researchers were finally saying things that they may have thought for their entire careers as they kept watching things go slowly sideways, he says.
The recommendations they arrived at were unsurprising, which raises the question of why they haven’t been instituted before. The strongest point of agreement was to make sharing raw data more common and to give incentives to do so. This could make checking each other’s work far easier and disrupt the current situation where it is up to each researcher whether they want to make their data available for replication attempts and new analyses.
There are legitimate concerns around sharing raw data – including securing intellectual property and issues of national security, which may require that some information stay private. But overall, there was little pushback on creating infrastructure for storing and sharing the vast amounts of data produced each year by scientists hunting for miracle materials or new forms of matter.
at the National Science Foundation suggested that agencies like hers could have more impact if the researchers agreed on and publicised clear norms among themselves. , the physical sciences editor at the journal Nature, suggested that journals could require some data to be submitted alongside papers, as they do in disciplines like geoscience, but that would again require community buy-in from condensed matter physicists.
It is often said that science is self-correcting, as fraudulent or incorrect work will eventually fall by the wayside, but the group that met in Pittsburgh agreed that this process is not passive, and they have to actively be a part of it.
“We have seen that there is this hunger for talking about this. And that means people also want things to change. That is a cultural shift that probably would not have been possible five or 10 years ago, and it gives you have hope that we will continue,” says Frolov. He and other conference organisers are working on keeping the conversation going beyond a few days of passionate debate.
Article amended on 22 May 2024
We clarified comments from Legg about the Microsoft study
