December 4, 2013 | 42
In a series of recent articles published in The Economist (Unreliable Research: Trouble at the Lab and Problems with Scientific Research: How Science Goes Wrong), authors warned of a growing trend in unreliable scientific research. These authors (and certainly many scientists) view this pattern as a detrimental byproduct of the cutthroat ‘publish-or-perish’ world of contemporary science.
In actuality, unreliable research and irreproducible data have been the status quo since the inception of modern science. Far from being ruinous, this unique feature of research is integral to the evolution of science.
At the turn of the 17th century, Galileo rolled a brass ball down a wooden board and concluded that the acceleration he observed confirmed his theory of the law of the motion of falling bodies. Several years later, Marin Mersenne attempted the same experiment and failed to achieve similar precision, causing him to suspect that Galileo fabricated his experiment.
Early in the 19th century, after mixing oxygen with nitrogen, John Dalton concluded that the combinatorial ratio of the elements proved his theory of the law of multiple proportions. Over a century later, J. R. Parington tried to replicate the test and concluded that “…it is almost impossible to get these simple ratios in mixing nitric oxide and air over water.”
At the beginning of the 20th century, Robert Millikan suspended drops of oil in an electric field, concluding that electrons have a single charge. Shortly afterwards, Felix Ehrenhaft attempted the same experiment and not only failed to arrive at an identical value, but also observed enough variability to support his own theory of fractional charges.
Other scientific luminaries have similar stories, including Mendel, Darwin and Einstein. Irreproducibility is not a novel scientific reality. As noted by contemporary journalists William Broad and Nicholas Wade, “If even history’s most successful scientists resort to misrepresenting their findings in various ways, how extensive may have been the deceits of those whose work is now rightly forgotten?”
There is a larger lesson to be gleaned from this brief history. If replication were the gold standard of scientific progress, we would still be banging our heads against our benches trying to arrive at the precise values that Galileo reported. Clearly this isn’t the case.
The 1980’s saw a major upswing in the use of nitrates to treat cardiovascular conditions. With prolonged use, however, many patients develop a nitrate tolerance. With this in mind, a group of drug developers at Pfizer set to creating Sildenafil, a pill that would deliver similar therapeutic benefits as nitrates without declining efficacy. Despite its early success, a number of unanticipated drug interactions and side-effects—including penile erections—caused doctors to shelve Sildenafil. Instead, the drug was re-trialed, re-packaged and re-named Viagra. The rest is history.
This tale illustrates the true path by which science evolves. Despite a failure to achieve initial success, the results generated during Sildenafil experimentation were still wholly useful and applicable to several different lines of scientific work. Had the initial researchers been able to massage their data to a point where they were able to publish results that were later found to be irreproducible, this would not have changed the utility of a sub-set of their results for the field of male potency.
Many are taught that science moves forward in discreet, cumulative steps; that truth builds upon truth as the tapestry of the universe slowly unfolds. Under this ideal, when scientific intentions (hypotheses) fail to manifest, scientists must tinker until their work is replicable everywhere at anytime. In other words, results that aren’t valid are useless.
In reality, science progresses in subtle degrees, half-truths and chance. An article that is 100 percent valid has never been published. While direct replication may be a myth, there may be information or bits of data that are useful among the noise. It is these bits of data that allow science to evolve. In order for utility to emerge, we must be okay with publishing imperfect and potentially fruitless data. If scientists were to maintain the ideal, the small percentage of useful data would never emerge; we’d all be waiting to achieve perfection before reporting our work.
This is why Galileo, Dalton and Millikan are held aloft as scientific paragons, despite strong evidence that their results are irreproducible. Each of these researchers presented novel methodologies, ideas and theories that led to the generation of many useful questions, concepts and hypotheses. Their work, if ultimately invalid, proved useful.
Doesn’t this state-of-affairs lead to dead ends, misused time and wasted money? Absolutely. It is here where I believe the majority of current frustration and anger resides. However, it is important to remember two things: first, nowhere is it written that all science can and must succeed. It is only through failure that the limits of utility can be determined. And second, if the history of science has taught us anything, it is that with enough time all scientific wells run dry. Whether due to the achievement of absolute theoretical completion (a myth) or, more likely, the evolution of more useful theories, all concepts will reach a scientific end.
Two reasons are typically given for not wanting to openly discuss the true nature of scientific progress and the importance of publishing data that may not be perfectly replicable: public faith and funding. Perhaps these fears are justified. It is a possibility that public faith will dwindle if it becomes common knowledge that scientists are too-often incorrect and that science evolves through a morass of noise. However, it is equally possible that public faith will decline each time this little secret leaks out in the popular press. It is a possibility that funding would dry up if, in our grant proposals, we openly acknowledge the large chance of failure, if we replace gratuitous theories with simple unknowns. However, it is equally possible that funding will diminish each time a researcher fails to deliver on grandiose (and ultimately unjustified) claims of efficacy and translatability.
Many of my colleagues worry that honesty and full disclosure will tarnish the reputation of science. I fear, however, that dishonesty will accomplish this much faster. In the end, we must trust that the public and granting bodies can handle the truth of our day-to-day reality. The story of legitimate science may not live up to the ideal—but at least it is the truth. Isn’t that what science purports to be all about?