Scholarly scientific publications have a pretty standard structure: Abstract, Introduction, Methods, Results, Discussion, Conclusion, References. One of the primary purposes of this structure is to make the work understandable to others. Ideally, we will understand the context from the introduction, see how the experiments were performed from the methods section, and learn how future research may aid our understanding of the topic in the discussion or conclusion. All of this makes the scientific process more transparent.

But the way in which the publication came to be is often shrouded in mystery, especially for those who haven't tried to publish something in an academic journal.

The peer review process is inherent in the "scholarly" part of "scholarly publication." But when you examine a standard journal article, there is nothing to indicate if the text has been through such a review process.

In class earlier this week, I showed students this article and asked them to tell me why it was considered scholarly:

After answers related to the occupation of the authors and the presence of a reference section, one student said "Because it's peer-reviewed."

"Really," I said, "how do you know it has been through peer review?" The student didn't have a response.

It's a trick question, really, because there is nothing in the article that indicates it is peer-reviewed.

In addition to the lack of information within articles themselves, the websites of many journals make it very difficult to determine if the articles they publish have been through some kind of peer review. You almost always have to dig deep into the journal website to find out. Sometimes the "About" page will say that "Journal of Scholarly-ness is a peer reviewed journal focused on…" But more often, readers actually have to dig into the "Author Guidelines" or "Reviewer Guidelines" sections to determine what review process happens prior to publication. This gets even more complicated if a publication includes some peer-reviewed content and some not-peer-reviewed content (e.g. book reviews, news stories, etc.).

Thankfully, some publications are making this a bit clearer by labeling articles as "Peer-Reviewed," like this PLOS ONE article:

Even when you find a statement saying "This journal uses peer review," that doesn't tell you a whole lot. What were the peer reviewers looking for? How did they evaluate the manuscript? How did the reviewers praise or criticize the paper?

Peer review is difficult to study (Couzin-Frankel, 2013), and the process remains mysterious even for some who have been through it. For students and the general public, peer-review can often be confused with basic proof-reading. The secrecy of the peer-review process doesn't help students as they learn to explore and think critically about the scientific literature.

I'm happy to see new publications like PeerJ and F1000Research making reviews available to the public, alongside the earlier versions of the manuscript. Teachers can use these publications to teach students about the peer-review and publication process. And third-party peer review services are creating peer-review rubrics to guide reviewers through the process. Happily, students can examine these rubrics, and even use them on published papers they read in class to help them critically analyze the reports.

Without wading into the debate about how the peer-review process can be improved, I suggest three really simple things publishers can do to assist readers:

  • Add a note indicating if an individual article is peer-reviewed to the HTML and PDF versions of the article
  • Include the words "peer-reviewed" in the "About" section of your journal
  • Prominently post any guidelines you provide to reviewers.

My students would certainly appreciate it.

Couzin-Frankel, J. (2013). Secretive and Subjective, Peer Review Proves Resistant to Study. Science, 341(September), 1331. doi:10.1126/science.341.6152.1331