June 16, 2011 | 7
"Science is messy. And the bigger the claims, the more intense the criticism." This is how Brian Vastag opened his Washington Post article chronicling the publication of NASA’s arsenic bacteria paper along seven critical comments and a follow-up response. It describes the situation – and science – well, but it’s not the story that those outside of science usually hear.
In 1989, I was a shy but keen 9th Grade science student at a rural high school, working hard on a term project on the dangers of Acid Rain. That March, my teacher Mr. Sklepowicz brought in some newspaper clippings about a press conference held to announce supporting evidence for cold fusion.
"Cool!" I thought and I asked if it could be true. He told me that other labs would surely be trying to replicate the results and we’d know soon enough. When the supporting evidence never materialized, the results were widely criticized within the physics and electrochemistry communities – but I never heard about it.
It was a long time before I asked another teacher and finally heard that they had been discredited. I didn’t learn much about the processes of science or how results are critiqued and argued, just that sometimes people do bad science and they’ll eventually be found out. There are much different and better lessons to glean from the current scientific controversy that Vastag was writing about.
The changing nature of scientific critique
Beginning with the NASA’s original press release, the arsenic bacteria results have been widely discussed online, including early speculation that an extra-terrestrial life form had been found. Following the press conference, the online discussion quickly turned to comments and critique.
Rosie Redfield from the University of British Columbia, who has explained the findings and her criticisms in today’s accompanying post, wrote strong critical comments about the methods and interpretation of the paper, just two days after the press conference.
Journalist Carl Zimmer, who had already alluded to scientists’ skepticism about the findings on the day of the press conference, published a summary of criticisms from scientists in Slate and later posted on his blog the scientists’ exact comments from email exchanges.
Coverage of the critique on science related blogs was extensive and picked up by institutional media outlets like the Canadian Broadcasting Corporation (CBC) and the New York Times. One of the defining characteristics of this story has been the "coverage of the coverage". There was a lot of hype not only about the original press release but also about the importance of online communication about the story including the blog coverage and the ongoing twitter discussions linked with hashtag #arseniclife.
Zimmer, in a piece entitled "The Discovery of Arsenic-Based Twitter: How #arseniclife changed science" makes a strong comparison between this case and a typical cycle of scientific critique:
"In earlier times, such critics didn’t have many options. They could write to Science and hope that their letter would be published long after the public’s attention had turned to other things. They could write to their local newspaper and try to sum up their objections in 50 words. They could grouse over a beer with likeminded colleagues. Now, however, they can form an online community. Blogging scientists read the #arseniclife paper and aired their complaints. On Twitter, they kept each other up to date on new developments in the story. Within a couple weeks the New York Times and the Washington Post were reporting not on the Science paper, but on the online debate. The center of gravity had shifted."
But has it really? Has online communication and critique really changed science? In some way yes, in others no.
Scientists have always had networks for sharing criticisms and discussing results. Holly Tucker’s book Blood Work includes wonderful descriptions of the back and forth publication of claims and refutations related to 17th century blood transfusions in journals, personal letters, and pamphlets.
Similarly, while 1989 might seem like a long time ago by web standards, quickly after the media blitz of Pons and Fleischmann’s Cold Fusion press conference researchers were using electronic back channels to communicate with each other. Electronic newsletters, like the "Cold Fusion Newsletter" were distributed by email to interested colleagues and posted to the alt.fusion and sci.physics.fusion newsgroups.
Online (or offline) correspondence communities are not new for scientists and neither are rapid communications used to keep each other up to date on developments. What is new is that in the case of the arsenic story, these communications have happened in the open.
Anyone who is interested has the opportunity to see them and try to make sense of them. As Zimmer points out, there isn’t a clear boundary between what was eventually published in the peer reviewed technical comments and what had already been said in other venues. Steve Benner, for example, authored one of the peer reviewed technical comments published by Science and, in parallel, wrote a blog post summarizing the concerns that he and others had about the study.
Not only were the critiques themselves open and available, the processes that led to these critiques have been as well.
Redfield’s critique process has been open from the start. After an initial critical blog post, she posted a draft of her letter to science and asked readers for comments and suggestions. The process of publishing this letter as a Technical Comment is also described in a more recent post:
"The brief Letter to Science that I composed here (with help from some readers) was one of the eight published today. The Letter was converted to a ‘Technical Comment’ by the Science Editorial Office, I guess because it contained technical comments, and had one-paragraph peer reviews from 5 (!) reviewers. I think Science must have sent all the submissions to the same group of reviewers, who gave each a very brief review."
This is what is new about the arsenic life story. Arsenic isn’t so much a story about post-publication peer review but instead of post-publication peer review in the open.
What’s the importance of it being in the open?
In thinking about this story, I come back to the kid that I was in 9th grade. I regret my hairstyle choice, sure, but I think back to what I could have learned about science if the post-publication peer review of cold fusion had happened in the open.
One of the most obvious outcomes of these online critiques has been the significant attention that institutional news outlets have paid not only to the original press release but also to the ongoing narrative of criticism. In doing so, they have told the story of the people and processes of the arsenic bacteria research, while simultaneously pouring significant effort into explaining the findings, the specific criticisms and the research team’s responses.
As one example, the CBC published a helpful comparison table to show the critical comments and the responses provided by the Wolfe-Simon team. This way even teachers and students not familiar with science blogs or a part of Twitter communities start to have access to its impacts.
So, if it had happened in 1989, I probably wouldn’t have had to wait a few years to hear about the critique. More significantly though, post publication peer review brings to the forefront some important issues for science teaching.
There’s more to this than science teachers having better access to media coverage of critical comments. The #ArsenicLife saga provides a narrative with tremendous educational value for students, like me, hungry to understand how science works.
Current science education guidelines advocate that students learn scientific concepts by engaging in inquiries and that they learn about the nature of science as it is practiced. Issues of communication and critique apply to both and should be a part of school science. Examining science standards though suggests that they might not do so adequately in a science media environment that is changing.
1. Scientific communication is not an afterthought
Communication is an included, if not emphasized part of most science programs. When students learn to do science, the typical last step is communicating their results to the class or their teacher or another audience. For example, the National Science Education Standards describes what might be included when students perform their own investigations:
"The investigation may…require student clarification of the questions, method, controls, and variables; student organization and display of data; student revision of methods and explanations; and a public presentation of the results with a critical response from peers" (p. 175)
Similarly, the Skills Required for Success in Inquiry (SRSI) is a checklist sometimes used to assess students’ scientific skills. It addreses communication only once, providing a checkbox to ensure that the student "Communicates observations and findings".
Structured in this way, students can be left to see "communication" as a school science step – a report writing exercise so that teachers have a product to evaluate, a report that may differ only in style or structure from a report written in any other course. If they are not encouraged to understand this step as a true element of science, they may gather that scientific reports are the final word on a subject: After everything has been completed, a report is written. It might receive a critical response (much like a film or book might) but the process is over.
The arsenic story illustrates scientific communication as an ongoing and multifaceted process. Some of the scientists involved have not only reported on their own work but spoken and written about their concerns regarding the methods and interpretation of the study on multiple platforms and documented their changing thinking on the subject.
Redfield, for example, in one of her recent posts expresses changing views on the value of replication testing: "I’ve been saying that researchers shouldn’t invest the time and resources needed to test Wolfe-Simon et al’s claims, because of the vanishingly small probability that they are correct. But I’m having second thoughts, because the most important claims can, I think, be very easily tested."
Similarly, her post today is not a final report of an investigation. It details her future plans and ongoing reasoning. She describes the replication activities she intends to do in her own lab, why she is planning them and what she expects to find as a result. There is even a promise of continued communication as she documents the replication process on her blog.
This is very different from the lab report that most often exemplifies school science communication. It is not an afterthought but an essential and ongoing part of the scientific process. And like the whole process of post publication peer review in the open, it can be a resource both for learning about biology and, critically, about science communication.
2. Scientific critique is vital to science and has some common elements
Another feature of the standards quoted above is the cursory mention of critique. They acknowledge that responding to criticism is part of science but the details are slim.
"Students in school science programs should develop the abilities associated with accurate and effective communication. These include writing and following procedures, expressing concepts, reviewing information, summarizing data, using language appropriately, developing diagrams and charts, explaining statistical analysis, speaking clearly and logically, constructing a reasoned argument, and responding appropriately to critical comments" (p. 176)
There are a couple of elements missing here that are important in the arsenic story. Again, it is found only at the end of the process. There is no indication here that critique itself leads to more investigation. The plans that Redfield describes in her post were initiated as a result of critical views of the original study. It is not just the comments and questions that an audience poses or the answers that an author gives that matter but also what is done as a result. The arsenic story lays bare that critique is not the end-product of science but an essential step.
There is also no indication in the standards of what the critical comments might be, nor that students should develop their skills in composing appropriate critique and deciding whether the critical comments have been answered. The open post publication peer review of the arsenic study provides valuable resources in this regard – resources likely inaccessible before.
For example, the Vastag’s Washington Post article summarizes the nature of some of the critique:
"The assertions ranged widely: Wolfe-Simon’s methods were sloppy, her conclusions an overreach. Science magazine had failed to vet the research properly. NASA had hyped the finding."
These same elements: sloppy methods, overreaching conclusions, rigor in peer review and overly-hyped results can also be seen Zimmer’s collection of emails from scientists. Benner’s post, with the provocative title "Does Arsenic Really Exist in the DNA from GFAJ-1?", goes even further by examining how chemists and geologists might approach both the critique and the response differently, providing a resource for understanding not only appropriate avenues of critique but that the criteria of critique can depend on the disciplinary context.
3. Something is missing in the skills that students are taught
The final lesson is that there may be something missing in the scientific inquiry skills taught to students. Reading and interpreting scientific texts are sometimes overlooked when the emphasis is on getting students actively doing science, especially in high school science. Reading is conspicuously absent from the standards document that has been quoted in this post.
When educators do focus on high school students reading in science, they usually focus on one of two approaches, either learning to read about difficult concepts, or learning about the different types of scientific writing, for example, primary sources (e.g., journal articles) and secondary sources (e.g., science magazines and newspapers). These are important and valuable scientific skills. There is little guidance, however, on how to navigate and make sense of a genuine controversy that plays out in several different communications media.
This is the arsenic lesson that is perhaps most pressing. Accessible post publication peer review of high profile science will likely continue. Students will encounter back and forth critique, rejoinders, and responses that are quickly available to them online. How do they make sense of them? The scientific skills in our curricula need to adapt to this new architecture and we need to begin thinking about how to provide students with these tools.
When I was a 9th grader all that was available were newspaper articles. My teacher dutifully collected them and helped us try to learn to read them with a critical eye so that we could be informed about current events and start to see science as more than a school subject. We couldn’t see the messy side that Vastag describes so it wasn’t essential yet that we learn to make sense of it – now it is.
I know it’s easy to say that there’s more that should be done in school science education. It seems like there’s always more. Scientific literacy, however, is more than just understanding some core scientific concepts; it is also about having the knowledge and skills to deal with scientific issues in the public sphere. The real revolution of the arsenic study has been the movement of scientific critique from the private world of science to the public world of blogs, twitter and online news. It will be essential for students as future citizens to be able to navigate these media astutely. That is one of the biggest changes that #arseniclife has brought.
About the Author: Marie-Claire Shanahan is an assistant professor of science education at the University of Alberta in Edmonton. She is interested in all of the ways that language impacts the interactions that people have with each other in science – in meeting rooms, classrooms and online. She blogs at Boundary Vision and tweets at @mcshanahan. When she isn’t teaching, visiting research sites or writing, she can be found exploring the Edmonton river valley with her dogs who, despite her best efforts, have not yet developed the ability to ask scientific questions.
The views expressed are those of the author and are not necessarily those of Scientific American.
Related at Scientific American:
Note: name Vastag was incorrectly spelled in the original version of this article. Corrected on June 18th, 2011. We apologize for the error.
Give a 1 year subscription as low as $14.99X