January 27, 2014 | 8
Could science’s replication crisis stem in part from how students are taught to perform experiments in college? That’s my suspicion after discussing the issue with students taking Introduction to Science Communication, a course I’m teaching for the first time at Stevens Institute of Technology.
“Replication crisis” refers to the growing recognition that the bulk of peer-reviewed scientific claims are “false,” as statistician John Ioannidis bluntly puts it. Ioannidis has drawn attention to the problem in a series of papers, beginning with his 2005 blockbuster “Why Most Published Research Findings Are False.”
The crisis, which I flicked at in a November column, continues to generate headlines. My buddy George Johnson reflected on it in “New Truths That Only One Can See,” his terrific inaugural contribution to a new Science Times column, “Raw Data.” (Congrats to George and the New York Times on the column!)
“Replication, the ability of another lab to reproduce a finding, is the gold standard of science, reassurance that you have discovered something true,” George writes. “But that is getting harder all the time.” George and I also chewed on the issue in a new Bloggingheads.tv chat.
Last week I asked my 20 science communication students–almost all science or engineering majors–to read a recent Economist cover story, “Unreliable Research: Trouble in the Lab,” which we then discussed in class. My students were not exactly shocked by the Economist expose. Far from it. One mentioned that when doing laboratory work for science classes, he had seen classmates fudge experimental results to align them with professors’ expectations. As he spoke, others nodded their heads. When I asked if anyone else had witnessed such behavior, all but a couple of students raised a hand.
Two students, Anthony and Amira, elaborate on these problems in papers that they posted on our course blog. Both posts are worth reading in their entirety, but here are excerpts.
In “Exploring the Statistical Defamation Sweeping Across Science Through the Eyes of an Undergrad,” Anthony writes: “[I]n several of the classes students are expected to perform experiments in, data is not necessarily as important as the report itself. I have seen students essentially create data to fit into a range given by the professor regardless of what their data actually is. Change a number here, move a decimal point there and voila, you have desirable data. The motivation for these kinds of actions varies. The most obvious is the mentality that GPA is without a doubt the most important thing in college. Students with this kind of mindset will do just about anything to make sure that the letter they receive for the class is as high as possible… [Professors] may be at as much fault as their students. By not making it a priority to correct statistical errors and by pushing aside the importance of effective data collection and analysis, they pass these ideologies on to their students, many of whom eventually put them into practice. I have seen numerous cases where students, after struggling with an experiment for whatever reason, would simply be given a set of data points from their professor so they can move on with what had been planned for the class. By doing this, the professor implies that the data is not necessarily as important as the final result.”
In “Is Irreproducibility an Issue of Academic Negligence?” Amira writes: “There are two contributors to this fiasco, the laboratory instructor and the students. Given an experiment to conduct, usually with ideal results easily predicted, most students’ goal is to get the ‘answer’ and leave. With the experimental results rarely (if ever) holding true to the ideal, some students take it upon themselves to ‘correct’ the mistake by changing the data, rather than risk being told to repeat the experiment; they’re more concerned with the red number on the top corner of the page than with those in the excel spreadsheet boxes.”
Amira notes that some professors, fortunately, emphasize the process rather than final results of experiments. “[I]nstead of being graded on numerical results (given that the procedure was understood and followed),” she writes, “there is a focus on understanding the outcome of the experiment; if reasonable, then it is understood why, and if not, then there is an analysis of real world factors that may have contributed to experimental errors.”
Yes, that’s the way science should be taught. But why isn’t that method universal? The problem, Amira suggests, is the focus on “judging every individual based on the outcomes of exams, whether they are college entrance, graduate entrance, or occupational exams. We’ve depleted inspiration from our work and made everything a competition. The pressure caused by these fixated evaluations drive scientists to one goal, produce and publish anything and everything just to stay in the public sphere, meaning to keep their job and position… [A] cultural transformation away from this uninformative means of evaluation will be the ultimate solution.”
Of course my students represent a minute sample, but I’ll wager that their experiences are not unusual. Ironically, listening to them and reading their papers left me in an oddly upbeat mood. Not to get all sentimental, but as long as science attracts intelligent, conscientious students like Anthony and Amira, there’s hope.