Current reforms to end the ‘rat race’ between scientists can help; but are they enough?
Science is society’s best method for understanding the world. Yet many scientists are unhappy with the way it works, and there are growing concerns that there is something “broken” in current scientific practice.
Many of the rules and procedures that are meant to promote innovative research are little more than historical precedents with little reason to suppose they encourage efficient or reliable discoveries. Worse, they can have perverse side-effects that harm both science and scientists. A well-known example is the general preference for positive over negative results, which creates a “publication bias” — giving the false impression that certain effects exist, where in reality the dissenting evidence simply fails to be released.
Arizona State University researchers Thomas Morgan and Minhua Yan, working with ASU graduate Leonid Tiokhin, now at University of Technology Eindhoven in the Netherlands, have developed a new model, published this week in Nature Human Behaviour, to better understand the challenges facing the scientific process and how we can make it better. They focused on the “priority rule”: the tendency for the first scientist to document a finding to be disproportionately rewarded with prestige, prizes and career opportunities while those in second place get little to no recognition.
Winner Takes All
Many scientists have sleepless nights worrying about being “scooped” — fearing that their work won’t be considered “novel” enough for the highest-impact scientific journals because a different group working on the same topic manages to publish first. The priority rule has been around for centuries. In the 17th century, Isaac Newton and Gottfried Leibniz haggled over who invented calculus. And in the 19th century, Charles Darwin rushed to publish “On the Origin of Species” to avoid being scooped by Alfred Russel Wallace.
“Rewarding priority is understandable and has some benefits. However, it comes at a cost,” Tiokhin said. “Rewards for priority may tempt scientists to sacrifice the quality of their research and cut corners.”
“The idea is that competition encourages scientists to work hard and efficiently, such that discoveries are made quickly,” said Morgan, a research affiliate with the Institute of Human Origins and associate professor with the School of Human Evolution and Social Change. “But if everyone is working hard, and you need to come in first to be successful, then there’s a temptation to cut corners to maximize your chances, even if it means the science suffers.”
This is partly why some academic publishers, such as PLOS and eLife, now offer “scoop protection,” allowing researchers to publish findings identical to those already published within a certain timeframe. The problem is that science and publishers currently don’t have a good idea about whether these reforms make sense.
Modeling the Priority Rule
To figure out how exactly the preference for priority affects science, and whether recent reforms offer any solution for its potential drawbacks, the collaborators developed an “evolutionary agent-based model.” This computer model simulates how a group of scientists investigate or abandon research questions, depending on their own results and the behavior of other scientists they compete against.
“The benefit of an evolutionary simulation is that we don’t need to specify in advance how scientists behave. We just create a world in which success is rewarded, and we let selection figure out what kinds of behavior this favors,” Morgan said. “We can then vary what it means to be successful — for instance, whether or not it’s critical to come first — and see how selection changes the behavior of scientists in response. We can also measure the benefit to society — are scientists being efficient? Are their findings accurate? And so on.”
No Panacea
The researchers found that a culture of excessive rewards for priority can have harmful effects. Among other things, it motivates scientists to conduct “quick and dirty” studies, so that they can be the first to publish. This reduces the quality of their work and harms the reliability of science as a whole.
The model also suggests that scoop protection, as introduced by PLOS and eLife, works.
“It reduces the temptation to rush the research and gives researchers more time to collect additional data,” Tiokhin said. “However, scoop protection is no panacea.”
This is because scoop protection motivates some scientists to continue with a research line even after several results on that topic have been published, which reduces the total number of research questions the scientific community can address.
The ‘Benefit’ of Inefficiency
Scoop protection reforms in themselves, while helpful, are not sufficient to guarantee high-quality research or a reliable published literature. The model also shows that even with scoop protection, scientists will be tempted to run many small studies if new studies are cheap and easy to set up and the rewards for negative results are high. This suggests that measures that force scientists to invest more heavily in each study, such as asking scientists to preregister their studies or get their research plans criticized before they begin collecting data, can help.
“We also learned that inefficiency in science is not always a bad thing. On the contrary — inefficiencies force researchers to think twice before starting a new study,” Tiokhin said.
Another option is to make large-scale data collection so straightforward that there is less incentive to skimp on data, alternatively, reviewers and journals could be more vigilant in looking out for “underpowered” studies with small sample sizes.
Metascience
This project is an example of metascience, the use of the scientific method to study science itself.
“It was a great pleasure to be part of this project. I got to use my modeling skills not only to make specific scientific discoveries, but also to shed light on how the scientific procedure itself should be designed to increase research quality and credibility. This benefits the whole scientific community and ultimately, the whole society,” said Yan, a graduate student in the School of Human Evolution and Social Change.
Reference: “Competition for priority harms the reliability of science, but reforms can help” by Leonid Tiokhin, Minhua Yan and Thomas J. H. Morgan, 28 January 2021, Nature Human Behaviour.
DOI: 10.1038/s41562-020-01040-1
Written by Julie Russ (ASU) and H.G.P van Appeven (Eindhoven University of Technology).
Never miss a breakthrough: Join the SciTechDaily newsletter.
Follow us on Google and Google News.
2 Comments
I think it is very misleading to consider anything related to “evolution” to be helpful in purifying or improving the scientific methods which seem to have gone too far afield. Most people think of evolution in Darwinian terms, but this is the evolving of an idea – not evolution if one looks at it critically.
… for evolution, intelligence was a hard thing to create, humans as a outstanding example…
… the savages will not tunnel to a another level…