There are lots of ways to mess with the heads of undergraduate students. (This is a recent favorite). Giving them a research assignment and failing to specify a minimum number of references needed is just one example.

"Include as many sources as you need to make your point and illustrate your thesis."

For students, finding one scholarly article on their topic often seems to be enough. Researchers did an experiment, got some results, and answered the research question the student started with. All done, all set, time for dinner.

But science doesn't work that way. One experiment may suggest something interesting, but it doesn't prove anything. In fact, it is quite easy to point to many examples of intriguing scientific studies that were either proved false or that couldn't be reproduced later on. Scientific ideas that are true should be reproducible: other researchers should be able to repeat the experiments and get similar results or use other methods to arrive at the same conclusions. You can't say that you discovered something new if someone else can't reproduce your result.

This fundamental scientific idea, reproducibility, may be in crisis. A recent article by Vasilevsky et al. in the journal PeerJ suggested that many scientific journal articles don't provide the information that other scientists would need in order to replicate their results. Key information about chemicals, reactants or model organisms is often missing, despite journal requirements to include such information (Vasilevsky et al., 2013). And a recent item in The Economist suggests that this might not matter that much. The emphasis placed on new research (by funding agencies and tenure and promotion committees) means that few scientists even attempt to replicate the work of others ("Unreliable research: Trouble at the lab," 2013).

All of this means trouble from the very beginning of a research project, before an experiment is even designed, when scientists start to do background research on their topics. In the same way that experimental scientists can't rely on the results of just one experiment to prove something, relying on just one information source for knowledge is a sure way to end up with unreliable information. Journalists look for corroborating sources, wikipedia flags articles that need a wider variety of citations, and scholars need to find multiple scholarly articles to support their ideas.

Some innovative people, companies and publishers are trying to sort this mess out. A collaboration between PLOS ONE, Mendeley, Figshare and the Science Exchange will be attempting to replicate the results of selected projects as a part of the Reproducibility Initiative. The Reproducibility Project is a crowdsourced effort to evaluate the reproducibility of experimental results in psychology. And the Reproducible Science project aims to make the results of computational experiments reproducible by ensuring the sharing of code and data and by making that information available to reviewers who can test the results described in a manuscript they are reviewing.

Unfortunately, these innovative programs are just a drop in the bucket of modern science. Funding agencies, publishers and tenure and promotion committees still value original work more highly than verification work. Scientists who concentrated on replicating the work of others would risk their careers.

As a result it is important for students and scholars to be aware of the challenges facing the reproducibility of science. We teach students in introductory science classes that reproducibility is one of the hallmarks of science. As they learn more about their disciplines, they need to be aware of the practical challenges involved in reproducing the work of others, and the importance of finding multiple sources about a topic needs to be emphasized.

As a librarian, part of my job is to help students find additional sources related to their research topics, even if there isn't a published reproduction of an original source. This isn't about which database to use or whether to put quotes around a phrase. It is about getting them to think critically about their topics. For example, while there might not be a second study that repeated the experiment of the first, students can look for:

  • Studies that examined the same topic in a different way
  • Studies that used the same methodology on a different species, geographic area, etc.
  • Background studies on individual aspects of their research question, including the statistical analyses used
  • Studies that cite the original study (even if no one has tried to reproduce the results, other scholars might express doubts about their conclusions when they cite the original).

The issues surrounding reproducibility in science won't be solved overnight, and it will take a concerted effort from scientists at all levels of the modern scientific enterprise to steer this very big ship. In the meantime, students and scholars can make special efforts to ensure that they are using the highest quality information available as the basis of their original studies.

Works Cited:

"Unreliable research: Trouble at the lab." (2013, October 19). The Economist, 409(8858), 26-30.

Vasilevsky, N. a, Brush, M. H., Paddock, H., Ponting, L., Tripathy, S. J., Larocca, G. M., & Haendel, M. A. (2013). On the reproducibility of science: unique identification of research resources in the biomedical literature. PeerJ, 1, e148. doi:10.7717/peerj.148.