Scientific reproducibility is hampered by a lack of specificity of the material resources

A key requirement when performing scientific experiments is the accessibility of material resources, including the reagents or model organisms, needed to address a specific hypothesis. The published scientific literature is a source of this valuable information, but frequently lacks sufficient detail to the extent that researchers are unable to identify material resources used to perform experiments.

A study, published today in PeerJ, demonstrates the magnitude of the problem – a problem that negatively affects the ability of scientists to reproduce and extend reported studies. The study showed that a large number of scientific resources are unidentifiable based on the information reported within the journal articles.

"The stories we tell in scientific publications are not necessarily instructions for replication." said Melissa Haendel, Ph.D., an ontologist and assistant professor in the Library and Department of Medical Informatics & Clinical Epidemiology at Oregon Health & Science University and senior author on the study. "This study illuminates how if we aim to use the literature as the scientific basis for reproducibility, then we have to get a lot more specific."

The study, led by Haendel and Nicole Vasilevsky, Ph.D., project manager and biocurator in Oregon Health & Science University's Ontology Development Group, examined nearly 240 articles from more than 80 journals spanning five disciplines: neuroscience, immunology, cell biology, developmental biology and general science. The articles were evaluated to determine if the reported research resources could be uniquely identified based on the information that was provided in each article, its supplemental data, or prior references. Specific criteria were developed to determine if antibodies, cell lines, constructs, model organisms, and knockdown reagents were identifiable. Based on these criteria, Haendel, Vasilevsky and their team of researchers also developed guidelines for reporting of research resources. These guidelines are available online (http://www.force11.org/node/4433) and can be used as a new data standard by authors, reviewers, publishers, and other data contributors to aid reproducibility.

The study showed that just under 50 percent of scientific resources used in previously published articles were unidentifiable, a percentage which varied across resource types and disciplines. The study also found no increased level of identification in journals that had more stringent reporting guidelines.

"We hope that quantifying the problem through this study will highlight to the research community that there is a significant and pressing need to make material resource information more accessible going forward," said Vasilevsky.

Citation to the article: Vasilevsky et al. (2013), On the reproducibility of science: unique identification of research resources in the biomedical literature. PeerJ 1:e148; DOI 10.7717/peerj.148

PeerJ is an Open Access publisher of peer reviewed articles, which offers researchers a lifetime membership, for a single low price, giving them the ability to openly publish all future articles for free. The launch of PeerJ occurred on February 12th, 2013. PeerJ is based in San Francisco, CA and London, UK and can be accessed at https://peerj.com/.

All works published in PeerJ are Open Access and published using a Creative Commons license (CC-BY 3.0). Everything is immediately available—to read, download, redistribute, include in databases and otherwise use—without cost to anyone, anywhere, subject only to the condition that the original authors and source are properly attributed.

PeerJ Media Resources (including logos) can be found at: https://peerj.com/about/press/

Oregon Health & Science University (http://www.ohsu.edu/) is the state's only public academic health and research university. As one of Oregon's largest employers with more than 14,000 employees, OHSU's size contributes to its ability to provide many services and community support not found anywhere else in the state. OHSU serves patients from every corner of Oregon and is a conduit for learning for more than 4,400 students and trainees. OHSU is the source of more than 200 community outreach programs that bring health and education services to each county in the state.

The Oregon Health & Science University (OHSU) Library is the largest health sciences library in Oregon and serves to advance knowledge and improvement of health and teach skills to navigate, interpret, and analyze the information landscape to the community. See: http://www.ohsu.edu/xd/education/library/

For the Authors:

Mirabai Vogt
Email: vogtmi@ohsu.edu
Phone: +1 503-494-8231

For PeerJ:

press@peerj.com
https://peerj.com/about/press/

Scientific reproducibility has been at the forefront of many news stories and there exist numerous initiatives to help address this problem. We posit that a contributor is simply a lack of specificity that is required to enable adequate research reproducibility. In particular, the inability to uniquely identify research resources, such as antibodies and model organisms, makes it difficult or impossible to reproduce experiments even where the science is otherwise sound. In order to better understand the magnitude of this problem, we designed an experiment to ascertain the "identifiability" of research resources in the biomedical literature. We evaluated recent journal articles in the fields of Neuroscience, Developmental Biology, Immunology, Cell and Molecular Biology and General Biology, selected randomly based on a diversity of impact factors for the journals, publishers, and experimental method reporting guidelines. We attempted to uniquely identify model organisms (mouse, rat, zebrafish, worm, fly and yeast), antibodies, knockdown reagents (morpholinos or RNAi), constructs, and cell lines. Specific criteria were developed to determine if a resource was uniquely identifiable, and included examining relevant repositories (such as model organism databases, and the Antibody Registry), as well as vendor sites. The results of this experiment show that 54% of resources are not uniquely identifiable in publications, regardless of domain, journal impact factor, or reporting requirements. For example, in many cases the organism strain in which the experiment was performed or antibody that was used could not be identified. Our results show that identifiability is a serious problem for reproducibility. Based on these results, we provide recommendations to authors, reviewers, journal editors, vendors, and publishers. Scientific efficiency and reproducibility depend upon a research-wide improvement of this substantial problem in science today.