About the SA Blog Network

Information Culture

Information Culture

Thoughts and analysis related to science information, data, publication and culture.
Information Culture HomeAboutContact

But peer review isn’t perfect

The views expressed are those of the author and are not necessarily those of Scientific American.

Email   PrintPrint

In my last post, I made the argument that peer review makes science better. Every article is reviewed by at least a couple of experts prior to publication, and this helps prevent really bad science from appearing alongside the good stuff. Most scientists agree that peer review helps scientific communication (82%) or are satisfied with the peer review system (64%).

CC image courtesy of Research to Action, posted to Flickr by user AJC1

But peer review isn’t perfect. In fact, there are some significant problems with the way that peer review is traditionally done.

For example:

  • While peer review can do a pretty good job of separating good methodology and data analysis from bad, it isn’t very good at anticipating how revolutionary a particular idea may turn out to be. The history of science is filled with examples of discoveries whose importance was not immediately understood.
  • Peer review isn’t good at catching outright fraud – the reviewers aren’t charged with actually repeating the experiments described, and deliberate manipulation of data and tables can slip by even the most careful of reviewers. When fraud is discovered, articles are typically retracted.
  • Accountability is a challenge. Reviewers are typically anonymous and their comments are kept private. As a result, readers don’t typically get to see how a manuscript has changed from the initial version. The anonymous nature of review can also leave the door open for abuse. However, studies have shown that double blind peer review is likely to produce high quality reviews (e.g. Alam et. al 2011).
  • It is also difficult to evaluate the effectiveness of peer review, partially because it is difficult to define exactly what a good review will do versus a bad review. Several studies have tried to evaluate the effectiveness of various interventions (training, etc.) on the quality of reviews, but many results have been inconclusive.

Despite these challenges, most scientists are not in favor of ditching the peer review system. Over the last 20 years, the switch from traditional print publication to online has granted the scholarly community many opportunities to experiment with the peer review process. While most scholars seem to think that peer review is good, many also think that additional ways of evaluating articles would be beneficial to the scientific process.

Altmetrics, open peer review, and post-publication review are just three strategies aimed at improving how science is recognized and evaluated. It will be exciting to see how these strategies progress.

Bonnie Swoger About the Author: Bonnie J. M. Swoger is a Science and Technology Librarian at a small public undergraduate institution in upstate New York, SUNY Geneseo. She teaches students about the science literature, helps faculty and students with library research questions and leads library assessment efforts. She has a BS in Geology from St. Lawrence University, an MS in Geology from Kent State University and an MLS from the University at Buffalo. She would love to have some free time in which to indulge in hobbies. She blogs at the Undergraduate Science Librarian. Follow on Twitter @bonnieswoger.

The views expressed are those of the author and are not necessarily those of Scientific American.

Rights & Permissions

Comments 1 Comment

Add Comment
  1. 1. kmahawar 3:06 pm 06/23/2012

    Peer Review isn’t perfect and it does not have to be as long as things can be put into their proper perspective. Peer review as a gatekeeper of sccience is not very effective, arbitrary, sometimes biased, and costly. You take that function away from it and it is a great tool that will help scientists.
    Author of this article mentions Post publication peer review and I believe this is a tool we need to use more and more. Publish everything, then let peers review it and finally authors can improve their work (if they wish to) and publish it elsewhere for targetted audience or academic recognition. WebmedCentral has attracted some attention with its publishing model based on post publication peer review and others are thinking of launching similar ventures. Time will tell how things pan out!

    Kamal Mahawar

    Link to this

Add a Comment
You must sign in or register as a member to submit a comment.

More from Scientific American

Email this Article