This article was published in Scientific American’s former blog network and reflects the views of the author, not necessarily those of Scientific American
In my last post, I made the argument that peer review makes science better. Every article is reviewed by at least a couple of experts prior to publication, and this helps prevent really bad science from appearing alongside the good stuff. Most scientists agree that peer review helps scientific communication (82%) or are satisfied with the peer review system (64%).
But peer review isn't perfect. In fact, there are some significant problems with the way that peer review is traditionally done.
For example:
On supporting science journalism
If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.
While peer review can do a pretty good job of separating good methodology and data analysis from bad, it isn't very good at anticipating how revolutionary a particular idea may turn out to be. The history of science is filled with examples of discoveries whose importance was not immediately understood.
Peer review isn't good at catching outright fraud - the reviewers aren't charged with actually repeating the experiments described, and deliberate manipulation of data and tables can slip by even the most careful of reviewers. When fraud is discovered, articles are typically retracted.
Accountability is a challenge. Reviewers are typically anonymous and their comments are kept private. As a result, readers don't typically get to see how a manuscript has changed from the initial version. The anonymous nature of review can also leave the door open for abuse. However, studies have shown that double blind peer review is likely to produce high quality reviews (e.g. Alam et. al 2011).
It is also difficult to evaluate the effectiveness of peer review, partially because it is difficult to define exactly what a good review will do versus a bad review. Several studies have tried to evaluate the effectiveness of various interventions (training, etc.) on the quality of reviews, but many results have been inconclusive.
Despite these challenges, most scientists are not in favor of ditching the peer review system. Over the last 20 years, the switch from traditional print publication to online has granted the scholarly community many opportunities to experiment with the peer review process. While most scholars seem to think that peer review is good, many also think that additional ways of evaluating articles would be beneficial to the scientific process.
Altmetrics, open peer review, and post-publication review are just three strategies aimed at improving how science is recognized and evaluated. It will be exciting to see how these strategies progress.