Fewer than 1 in 3 breast cancer cell studies are reproducible, according to an analysis that used a robot to partly automate experiments. This doesn’t necessarily mean the findings aren’t accurate, but could suggest that researchers need to report their work more carefully.
Reproducibility – or the ability for other researchers to replicate the results of experiments in their own labs – is a cornerstone of the scientific method, but it isn’t often put to the test. “There are unbelievable numbers of papers being published, but very few repeating the same experiment,” says at the University of Cambridge.
To test the reproducibility of research on breast cancer, King and his colleagues trawled scientific databases for relevant papers, finding more than 12,000 in one open-access repository. Next, they used text-mining tools to pick out phrases they felt were important to the scientific literature in the field, to identify those papers whose findings were most meaningful.
Advertisement
They selected 74 papers to try to replicate and used a “robot scientist” called Eve to carry out the experiments. Eve added chemicals to cancer cell cultures that the published studies said would affect the activity of certain genes.
Just 22 out of 74 results from the studies could be fully replicated by the robot. “Most of the literature suggests if you really try hard, you get about half of them to work,” says King. “We semi-automated it, so it’s a lower bound.”
King believes that the inability to reproduce so many published findings is an indication of two things. First, biological systems are incredibly complicated. “Slight differences in conditions become important,” he says.
The other issue is lax record-keeping in experiments. “Generally, experiments are not monitored, measured and recorded enough to make them reproducible,” he says. Because cancer is such a high-stakes issue, it is important that experiments can be reproduced by anyone.
at the University of Sheffield, UK, agrees. “A cell biologist is a flexible and adaptable operator in the lab environment,” he says. “We’re easily able to deal with complex processes and adapt to imperceptible problems using training and experience as cues.” Robots aren’t – and their failings highlight how experiments need to be better recorded and written up, he suggests.
King hopes that using robots, which cost less than humans to conduct the same experiment, could help to ensure that more cancer findings are replicable in future.
Journal of the Royal Society Interface
