image: Science is society's best method for understanding the world, yet many people in the field are unhappy with the way it works. Rules meant to promote innovative research have perverse side-effects that harm science and scientists. One of these - the 'priority rule' - rewards scientists who make discoveries disproportionately. Researchers at University of Technology Eindhoven and the Arizona State University have developed a model to see if current reforms to improve the system work. view more
Credit: Image: TU/e
Science is society's best method for understanding the world, yet many people in the field are unhappy with the way it works. Rules and procedures meant to promote innovative research can have perverse side-effects that harm both science and scientists. One of these - the 'priority rule' - rewards scientists who make discoveries with prestige, prizes and better career opportunities, depriving the runners-up of similar perks. Researchers at University of Technology Eindhoven (TU/e) and the Arizona State University in the US have developed a new model to better understand this rule, and see if current reforms to improve the system actually make sense. Their study was published in Nature Human Behaviour.
"Over the past decade, there have been growing concerns that something is "rotten" in the state of science", says Leonid Tiokhin, researcher at the Human Technology Interaction group at TU/e and lead author of the paper. "Scientists are realizing that many rules and procedures are dictated by norms and historical precedent, rather than principled reasons that serve efficiency and reliability. Worse, there is growing evidence that some norms have adverse side-effects that can hurt both science and scientists."
A well-known example is the general preference for positive over negative results. If journals only publish positive results, this creates 'publication bias', where the only results that get published are those that convincingly show correlation or causation, leaving out negative results that are just as relevant.
Another practice that may do more harm than good is valuing priority of discovery, which rewards scientist who are first to publish their findings with disproportionate prestige, prizes and career opportunities.
The winner takes it all
"Many scientists have sleepless nights worrying about being 'scooped' - fearing that their work won't be considered "novel" enough for the highest-impact scientific journals," says Tiokhin. The priority rule has been around for centuries. In the 17th century, Newton and Leibniz haggled over who invented calculus. And in the 19th century, Charles Darwin rushed the publication of his work on evolution by means of natural selection to avoid being scooped by Alfred Russel Wallace.
"Rewarding priority is understandable and has some benefits. However, it comes at a cost," says Tiokhiin, "Rewards for priority may tempt scientists to sacrifice the quality of their research and cut corners."
"This is partly why some academic publishers, such as PLOS and eLife, decided to offer 'scoop protection', allowing researchers to publish findings identical to those already published within a certain timeframe. The problem is that we don't yet have a good idea about whether these reforms make sense."
Modelling the priority rule
To figure out the effects of rewarding priority (and whether recent reforms offer any solution for its potential drawbacks) Tiokhin and his colleagues developed an 'evolutionary agent-based model'. This computer model simulates how a group of scientists investigate or abandon research questions, depending on factors such as the type of results (positive or negative) and the novelty of the research (the cost of being 'scooped').
The scientists were simulated as "agents" who are more likely to progress in their careers if they conduct research in a way that nets them sufficient rewards (such as money or prestige). By varying the way that science was structured and which types of findings were most rewarded, the researchers were able to examine how agents changed their behavior over time and what the consequences of these changes were for science as a whole.
No panacea
The researchers found that a culture that rewards priority - where there is a big cost to being scooped - can have harmful effects. Among other things, it motivates scientists to conduct 'quick and dirty' studies, so that they can be first to publish. This reduces the quality of their work and harms the reliability of science as a whole.
The model also suggests that some form of scoop protection, as introduced by PLOS and eLife, works. "It reduces the temptation to rush the research and gives researchers more time to collect additional data," says Tiokhin. "However, we should keep in mind that scoop protection is no panacea."
This is because scoop protection motivates some scientists to continue with a research line even after several results have been published, which reduces the total number of interesting questions investigated. The model also shows that even with scoop protection, scientists can be tempted to run many small studies if the costs of starting a new project (start-up costs) are low and the rewards for negative results are high.
The 'benefit' of inefficiency
"So while scoop-protection reforms are helpful, they are not sufficient to incentivize high-quality research or a reliable published literature," according to Tiokhin. "Our results suggest that we should also consider having some form of start-up costs, such as asking scientists to pre-register their studies or get their research plans criticized before they begin collecting data."
"We also learned that inefficiency in science is not always a bad thing. On the contrary: inefficiencies force researchers to think twice before starting a new study."
Other possible solutions to offset the negative effects of the priority rule are reducing data-collection inefficiencies (to make it easier to gather lots of data) and rewarding study quality (to avoid statistically 'underpowered' studies).
Metaresearch
Tiokhin has a personal motive for his work, which is known as 'meta-research', the use of the scientific method to study research itself. "When I was doing my PhD research in the US, I was frustrated by the way that science works. Like many researchers, I was idealistic when I got into science. But I soon found out that research is often not about figuring out the nature of reality, but about playing by the rules of the game. That often means getting your work published in the "right" journals, preferably with results that are novel and statistically significant."
"The problem is that I don't like playing by arbitrary rules. With my research, I want to understand what rules for recognizing and rewarding scientists actually make sense and generate better outcomes for science as a whole."
###
More info
Leonid Tiokhin, Minhua Yan, Thomas J.H. Morgan, Competition for priority harms the reliability of science but reforms can help, Nature Human Behaviour (DOI: 10.1038/s41562-020-01040-1)
Journal
Nature Human Behaviour