Group calls for more transparency in science research, announces guidelines

An international group of academic leaders, journal editors and funding-agency representatives and disciplinary leaders, including Rick Wilson, the Herbert S. Autrey Chair of Political Science and professor of statistics and psychology at Rice University, has announced guidelines to further strengthen transparency and reproducibility practices in science research reporting.

The group, the Transparency and Openness Promotion (TOP) Committee at the Center for Open Science in Charlottesville, Va., outlined its new guidelines in a story published in this week's edition of the journal Science.

While transparency, openness and reproducibility are readily recognized as vital features of science and embraced by scientists as a norm and value in their work, a growing body of evidence suggests that those qualities are not necessarily evident today.

"A likely culprit for this disconnect is an academic reward system that insufficiently incentivizes open practices," Wilson said. "In the present reward system, the emphasis on innovation undermines practices that support openness. Too often, publication requirements -- whether actual or perceived -- fail to encourage transparent, open and reproducible science."

TOP's objective is to translate scientific norms and values into concrete actions and change the current incentive structures to drive researchers' behavior toward more openness.

"We know the disciplines differ in what is emphasized, so we sought to produce guidelines that focus on what is shared across disciplines," Wilson said.

Each of eight standards has three levels of adoption in the TOP guidelines; each moves scientific communication toward greater openness, according to the article in Science. These standards are modular, facilitating adoption in whole or in part. However, they also complement each other; commitment to one standard may facilitate adoption of others.

The standards include citation standards for journals, data transparency, analytic methods (code) transparency, research materials transparency, design and analysis transparency, preregistration of studies, preregistration of analysis plans and replication.

Two standards reward researchers for the time and effort they have spent engaging in open practices. Citation standards extend current article citation norms to data, code and research materials. Regular and rigorous citation of these materials credits them as original intellectual contributions. Replication standards recognize the value of replication for independent verification of research results and identify the conditions under which replication studies will be published in the journal.

Four of the standards describe what openness means across the scientific process so that research can be reproduced and evaluated. Reproducibility increases confidence in findings and also allows scholars to learn more about what results mean. Design standards increase transparency about the research process and reduce vague or incomplete reporting of the methodology. Standards for research materials encourage the provision of all elements of that methodology, and data-sharing standards give authors an incentive to make data available in trusted repositories.

The final two standards address the values resulting from preregistration. Standards for preregistration of studies facilitate the discovery of research, even unpublished research, by ensuring that the existence of the study is recorded in a public registry. Preregistrations of analysis plans certify the distinction between confirmatory and exploratory research, or what is also called hypothesis-testing versus hypothesis-generating research. Making the distinction between confirmatory and exploratory methods transparent can enhance reproducibility.

"The guidelines are sensitive to concerns by both journals and researchers," Wilson said. "For example, we encourage journals to state exceptions to sharing because of ethical issues, intellectual property concerns or availability of necessary resources. We encourage journals to pick and choose among the different levels and standards in order to define what they expect of the researchers.

"We acknowledge the variation in evolving norms about research transparency. Depending on the discipline or publishing format, some of the standards may not be relevant for a journal. Journal and publisher decisions can be based on many factors -- including their readiness to adopt modest to stronger transparency standards for authors, internal journal operations and disciplinary norms and expectations," Wilson said.

The present version of the guidelines is not the last word on standards for openness in science, according to the report. "As with any research enterprise, the available empirical evidence will expand with application and use of these guidelines," the TOP Committee wrote. "To reflect this evolutionary process, the guidelines are accompanied by a version number and will be improved as experience with them accumulates."

An information commons and support team at the Center for Open Science is available (top@cos.io) to assist journals in selection and adoption of standards and will track adoption across journals. Adopting journals may also suggest revisions that improve the guidelines or make them more flexible or adaptable for the needs of particular subdisciplines.

To read the complete guidelines, go to https://osf.io/ud578/.

###

For more information or to schedule an interview with Wilson, contact:

Follow Rice News and Media Relations via Twitter @RiceUNews.