This article was published in Scientific American’s former blog network and reflects the views of the author, not necessarily those of Scientific American
Dear Dr. Shanahan,Thank you for your application to the Summer Institute on Unicorn Science. We appreciate the effort that went into all of the applications. We received over 1,000 excellent submissions and had a very difficult decision to make. Unfortunately, we were only able to select 10 applications and yours was not among those that were chosen.
We thank you for your interest and effort and encourage you to resubmit in future years.
Best wishes,
Rainbow Unicorn, PhD
Director of the Summer Institute on Unicorn Science
This is an example of a rejection e-mail I recently received. Okay, you’ve caught me, I didn’t really apply to a Summer Institute on Unicorn Science but the rest of the letter is real. Almost everyone has received a letter or e-mail like this one at one time or another. Maybe you didn’t get into a particular college program or didn’t receive a scholarship you applied for. Perhaps you’ve received some other form of bad news in writing such as a negative report card comment from your or your child’s teacher ("While Kayla’s quiz marks have begun to improve, she must continue to develop her organization skills so that her lab reports reflect the same level of achievement"). One thing that these letters and e-mails have in common is that, despite delivering bad news, they usually begin with some sort of praise for the recipient. Dr. Unicorn told me that she recognizes the effort I put into my submission and Kayla’s teacher notes that her quiz marks have gone up.
On December 2, NASA held a press conference to announce that a team of researchers had evidence of a bacterium using arsenic in the place of phosphorous in its DNA. A paper was simultaneously published in the journal Science. Since that day there has been a considerable amount of criticism levelled against the methods of the study and the strength of the evidence. The form that these criticisms took got me thinking about the meaning and impact of the bad news letter style.
On supporting science journalism
If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.
Much of the early criticism was posted online either on blogs (e.g., Dr. Rosie Redfield’s lab blog RRRResearch, Texas A&M researcher Jim Hu), or in other online venues (e.g., such as Carl Zimmer’s piece on Slate.com). From there it was also picked up by media outlets such as the CBC. Writing for The Guardian, Martin Robbins provides an excellent summary of the timeline and venues of the criticism.
Despite the emergence of these critiques, however, the research team initially resisted responding (A Q+A response document was later posted on Wolfe-Simon’s Web site).
Certainly, the volume of questions and comments was overwhelming. Felisa Wolfe-Simon, the lead author of the Science paper, compared it to a denial of service attack, where perpetrators overload a computer network such that it cannot respond to any requests, legitimate or otherwise. Responses from NASA and from Wolfe-Simon herself suggest, however, that there was more than just volume that held them back from responding directly to the criticism. Journalist Carl Zimmer, sought feedback on the critical comments made to him by experts in the field. This is what he wrote about the reply that he received:
"Any discourse will have to be peer-reviewed in the same manner as our paper was, and go through a vetting process so that all discussion is properly moderated," wrote Felisa Wolfe-Simon of the NASA Astrobiology Institute. "The items you are presenting do not represent the proper way to engage in a scientific discourse and we will not respond in this manner."
The message in these statements is that critique outside of peer-reviewed letters in scientific journals is not proper critique. Reading them one might assume that NASA’s biggest concern was comments coming from outside of the expert community. Perhaps encouraging responses through Science would solve the problem by allowing only those with peer status to comment.
The most pressing online criticisms already came, however, from peers - from expert scientists in related fields or journalists who had sought the opinions of such peers. These are the same people who might also write formal criticism letters in the scientific journals.
If these critiques come from peers with relevant expertise, what is different about them? Why were they so concerning to the NASA team? And, if they are different, do they have something new to offer to the science criticism and review process?
One obvious difference is the process of peer review, but if these critiques are written by peers, the lack of process doesn’t mean that they are necessarily illegitimate. We need to look beyond that process at other more subtle differences. One possibility is audience, and it is audience that brings me back to the bad news letter.
A bad news letter is usually directed at an individual or small group (e.g., a researcher or research team applying for a grant or a student applying for a scholarship). It must convey a piece of negative information while also working to maintain some sort of personal or community relationship. The teacher offering a negative comment to his student must carefully word the criticism so that it conveys the desired meaning but also does not discourage the student too much. A summer institute (such as the Summer Institute on Unicorn Science) or scholarship program may reject a lot of applicants. To maintain the program, the organizers want to make sure that those who are rejected aren’t too discouraged to apply again next year. In a small research field the rejected applicants may also be personally acquainted with those writing the rejection. It is important for the community and for possible future work together that the applicant not feel offended by the rejection they receive from a colleague. That is the purpose of the praise at the beginning of these letters—it softens and makes more palatable the criticism implied in the bad news.
Published peer reviews are no different. They take the same form as a bad news letter. Ken Hyland, from the University of Hong Kong, examined over 160 published peer reviews (mostly book reviews) from different academic fields, including science and engineering. In his collection of reviews, opening and closing with a praising comment was common to the point of being routine, regardless of the overall degree of criticism in the review. Even the most negative reviews usually opened with a praising comment. This pattern continues beyond the opening statements as well, with over 65% of all critical statements mitigated through praise or some other means such as overly tentative language. This was particularly evident in reviews published in science and engineering journals, which contained more praise and less criticism than those in social science and humanities journals. Francis Crick described the importance of these social niceties to science in describing the interactions that he had with other scientists while working to find the best methods for understanding the structure of DNA.
"I learned that if you have something critical to say about a piece of scientific work, it is better to say it firmly and nicely and to preface it with praise of any good aspects of it. I only wish I had stuck to this useful rule. Unfortunately I have sometimes been carried away by my impatience and expressed myself too briskly and in too devastating a manner" (Crick, 1988, p. 49)
Taking Crick’s advice is important in published peer reviews because, while the reviews may be read by anyone who reads that particular journal, the main audience for the review consists of the journal editors and the authors of the paper being criticized. Like a bad news letter, published peer reviews must navigate a difficult path between being critical and maintaining collegiality.
Are online critiques like the ones published about the arsenic-based life paper the same?
Redfield provides a great example for answering this question because she chose to engage in both types of critique. She published a first reaction on her blog RRResearch and then a few days later she posted a formal letter that she composed for submission to Science. The formal letter submitted for peer review opens and closes with the following sentences:
"Wolfe-Simon et al. (1) meticulously eliminated contamination of the reagents and equipment used in their elemental analyses, but they made much less effort to eliminate contamination in their biological samples. ... Omission of the gel-removal step for these critical samples is surprising because the authors did use it in preparing the rDNA fragments they sequenced for their phylogenetic analysis."
Notice the form of these two sentences: just like that of the bad news letter. Even in a very restricted space (Science limits letters to 300 words), she has taken steps to follow Crick’s advice. She praises the research team for their meticulous elimination of some types of contamination before criticising their approach to another type.
It may not have been explicitly intentional but using such limited space in this way shows that there are important interpersonal elements to this letter, elements that confirm the intended audience: the authors of the paper, the editors of the journal and members of the specific research community (or communities) affected by the paper. Anyone writing a published peer review like this must maintain a professional relationship with these audiences. It walks this difficult line by showing that the writer is a reasonable critic who recognizes both strengths and weaknesses.
The remainder of the letter also reinforces this interpretation of the intended audience. All of the other sentences consist of very specific, concisely stated critical comments (e.g., "The reagents used for the culture media were not pure."). None of the space is used for neutral comments, such as descriptions of the procedure or findings. The reader of the letter is assumed to have a detailed understanding of the paper so neutral details don’t need to be included.
This sits in sharp contrast to the opening paragraph of her blogged response:
"Here's a detailed review of the new paper from NASA claiming to have isolated a bacterium that substitutes arsenic for phosphorus on its macromolecules and metabolites. ... NASA's shameful analysis of the alleged bacteria in the Mars meteorite made me very suspicious of their microbiology, an attitude that's only strengthened by my reading of this paper. Basically, it doesn't present ANY convincing evidence that arsenic has been incorporated into DNA (or any other biological molecule)."
This is a strongly worded negative assessment, with a critical tone and word choice (e.g., ‘shameful’, ‘very suspicious’, ‘ANY convincing evidence’). There seems to be little effort to maintain any interpersonal relationships by softening the criticism with praise. Why? - Because her blog post is written for a completely different audience than the peer reviewed letter. This intended audience is also evident in the rest of the post. There are long sections of neutral text that describe the paper and the procedures that the researchers used. Redfield uses this description to support the critical points that she makes. For example, she introduces a criticism (the contamination problem) and supports it by explaining that the usual steps have not been taken. The assumption is that the audience may not be aware of the usual steps:
"An independent contamination problem is the omission of standard DNA purification steps when testing for As in DNA (2). Contamination is typical in DNA/RNA pellets produced by ethanol precipitation of the aqueous phases from phenol:chloroform extractions. ... Yet the usual step of washing the pellets was omitted, and the dried pellets were simply resuspended in water and loaded on an agarose gel."
Redfield introduced the blog post by saying that she wrote it to clarify her own thinking and that she expected only a few researchers to read it. The way that it is written, though, suggests a much wider audience—an audience that has not necessarily read the original paper. This could include the broader research community and also those that we might call the interested public—those who follow science news and have a personal interest in science discoveries but may or may not be involved in science. In a December 6 CBC news story, Redfield is quoted as saying "I blog openly…to bring this stuff more into the open where everybody can see it." This wide audience responded to her post, which was viewed by thousands of people and receive hundreds of comments both from scientists and non-scientists.
This online critique was not for the authors of the paper or for the editors of science—it was for outsiders, those who might not read the paper or the published peer review in Science. One of the main reasons we can see the difference is that it doesn’t look and sound like a bad news letter.
The lack of bad news letter form might also have a greater impact than this. One of the major elements of this story as it was reported throughout December was that the research team felt attacked and overwhelmed with criticism that they felt broke the conventions of scientific critique. NASA generally, and the authors of the paper in particular, initially refused to respond to criticisms made online, encouraging critics to turn to conventional peer-reviewed venues. This response may in some part be due to lack of bad news letter style. Because they were not written with the authors in mind as the audience, the blogged critiques didn’t soften their language. Crick might have described the tone of much of the online critique as "devastating" and that is exactly how they might feel to someone personally involved in the research.
So is online critique like Redfield’s blog illegitimate science critique? My view is that it is not—it is just a different form of communication, with different goals and, importantly, a different intended audience. There have been discussions of the value of online science communication for what is called post publication peer review, where online venues allow peers to critique and discuss findings in a public forum that can be viewed by all. There has been some resistance, however, to efforts that try to formalize these online reviews. For example, a study of efforts to allow support commenting and author responses on studies published in the British Medical Journal showed the authors and readers were reluctant to engage in this way. Redfield has described on her blog her own resistance to these formal online venues . She also points out that despite all of the online activity, Wolfe-Simon’s Science paper hasn’t received any comments on The Third Reviewer, a Web site dedicated to post publication peer review.
I think that the online comments that have been generated by this story illustrate something different. This isn’t post publication peer review for the community—it is review for the public, for anyone who is interested in reading it. In many ways it represents a throw-back to the origins of review writing. The history of published peer review began with summaries for the educated public of published works, work that was exploding in availability mid 18th century. The first reviews (published in, for example, Analytical Review and the Monthly Epitome) were typically non-critical summaries. They were descriptions with large transcribed sections of the original publication included without comment.
The subtitle of the Monthly Epitome was "Readers their own reviewers"—They aimed to publish the reviews so that readers could make up their own minds. Analytic Reviewannounced similar intentions in its inaugural issue: "The true design of a Literary Journal is, in our opinion, to give such an account of new publications, as may enable the reader to judge them for himself."
To achieve this, they believed that review writers should only summarize and quote from the works rather than providing any opinions of their own. The specialized nature of contemporary science makes this assumption impossible, no one (scientists included) can be expected to have the expertise to decipher the strengths and weaknesses of all research. We need the reviewers to provide their expert opinions and, like Redfield does, explain how the procedures differ from what is normally done. But we also need these critiques to be open and available and wide ranging. This allows the interested reader to make up their own minds. They can read, consider and weigh the supporting and contradicting opinions of several critics and decide for themselves.
Instead of rejecting these critiques as illegitimate because they do not follow the traditional procedure and form (such as rejecting the conventions of the bad news letter) another option is to embrace their intent to address a different audience. Might I suggest an adoption of the slogan of Monthly epitome: Online science critique—readers their own reviewers!
Adopting this view, however, requires a change in the interpersonal aspects of science critique (and critical analytic skills on the part of the reader, but that’s another issue). Researchers and scientific authors need to understand that when it comes to these reviews, they are no longer the primary audience and accept the interpersonal consequences both of writing them and receiving them. This isn’t an easy task. This post from A Blog Around the Clock provides a great overview of the challenges presented by the language of online science communication.
Looking back on the proposal that I submitted to the Summer Institute on Unicorn Science, I can see several weaknesses. I am relieved though that the institute does not have a blog and does not publish their critiques. Despite my own recognition of the problems in my proposal, I would be devastated to come across a review such as:
Dr. Shanahan’s proposal suffers from several crippling weaknesses. It opens with vague and unsupported generalizations, missing the usual step of including supporting quotes from the research participants before jumping into a report of the survey statistics.
I am saved by the fact that there would be little value in the Institute publishing reviews like this. This is not the case for high profile scientific claims that have been widely published and discussed in the public sphere. The audience for the initial reporting of the claims was broad and that broad audience needs access to corresponding critique that is written for them. There are no simple solutions to the interpersonal problems of critique but the value to the wider research community and the public of having open and forthright critique available surely outweighs the personal challenges that come from encountering potentially devastating critiques. The bad news letter serves an important purpose for one audience but there is important value in abandoning it when addressing another.
References and Further Reading
Crick, F. (1988). What mad pursuit: A personal view of scientific discovery. New York: Basic Books.
Hyland, K. (2004). Disciplinary discourses: Social interactions in academic writing. Ann Arbor, MI: University of Michigan.
Orteza y Miranda, E. (1996). On book reviewing. Journal of Educational Thought, 30, 191-202.
About the Author: Marie-Claire Shanahan is an assistant professor of science education at the University of Alberta in Edmonton. She is interested in all of the ways that language impacts the interactions that people have with each other in science - in meeting rooms, classrooms and online. She blogs at Boundary Vision and tweets at @mcshanahan. When she isn't teaching, visiting research sites or writing, she can be found exploring the Edmonton river valley with her dogs who, despite her best efforts, have not yet developed the ability to ask scientific questions.
The views expressed are those of the author and are not necessarily those of Scientific American.