ADVERTISEMENT
  About the SA Blog Network













Cross-Check

Cross-Check


Critical views of science in the news
Cross-Check Home

The truth we’ll doubt: Does the “decline effect” mean that all science is “truthy”?

The views expressed are those of the author and are not necessarily those of Scientific American.


Email   PrintPrint



As an old hippy I still get a kick out of anarchy, mayhem and challenges to authority. As a father, teacher, journalist and all-around pillar of the community, however, I’ve come to see the upside of the status quo more than I did in my carefree youth. So part of me thrills at WikiLeaks‘s assaults on government secrecy, whereas another part frets that forced transparency may subvert benign as well as malignant government actions.

I feel a similar ambivalence toward challenges to science’s authority. Take, for example, "The Truth Wears Off," a bombshell that journalist Jonah Lehrer just detonated in The New Yorker. (The article is behind a pay-wall, but here is an abstract.) Lehrer reports on the "decline effect," the tendency of scientific claims to receive decreasing support over time. The term was coined by the parapsychologist J. B. Rhine in the 1930s to describe the apparent drop-off in extrasensory perception (ESP) of psychic subjects tested by Rhine for extended periods.

The likely explanation, of course, is that Rhine’s subjects were never psychic; the initial finding of ESP was illusory, vanishing as Rhine’s methods became more rigorous and his data more statistically significant. The decline effect is really a "decline of illusion," Lehrer explains, which "reminds us how difficult it is to prove anything." The decline effect has turned up in a host of fields, including not only squishy sciences like psychology, psychiatry, psychopharmacology and medical genetics, but even physics; measurements of the charge of electrons and a constant governing the decay of neutrons have exhibited the decline effect, according to Lehrer.

Lehrer attributes the effect to several common factors. First, a researcher stumbles on a dramatic correlation—say, between a new pharmaceutical and amelioration of schizophrenia—that actually stems not from causation but just from coincidence. The more dramatic the researcher’s claim, the more likely he may be to get published and to obtain funding for more research. Other researchers jump on the bandwagon, doing follow-up studies that—because of the unconscious bias in favor of the initial claim—often corroborate it. Only gradually does counterevidence emerge, showing that the initial correlation stemmed not from causation but coincidence.

Consequently, science yields not truth but what comedy talk-show host Stephen Colbert calls "truthiness." None of this should surprise veteran science watchers—or anyone who’s taken a course in the history or philosophy of science. I’ve whacked fields such as clinical psychology and behavioral genetics for churning out claims—A new and improved treatment for depression! A gene for liberalism!—that don’t stand up to scrutiny. But Lehrer does a good job pulling together multiple strands into a unifying narrative of doubt. He cites the remarkable recent work of the epidemiologist John Ioaniddis, who has presented evidence that "most published research findings are false."

Lehrer also presents examples of the decline effect that were new to me. One is the link between physiological symmetry and sexual attraction in humans and other animals, which evolutionary psychologists have been touting for almost 20 years. Finally, the neo-Darwinian theory of human nature produced a not-completely obvious result! Except it didn’t. A flurry of positive findings in the 1990s—Women have more orgasms when they couple with symmetrical guys!—gradually gave way to negative reports.

So why does Lehrer’s article make me uneasy? First of all, early on he seems to suggest that the decline effect reflects changes in the phenomenon being measured, which is what Rhine meant by the term; only gradually does Lehrer make it clear that he attributes the effect to reporting bias. Some readers might still conclude that Lehrer is talking about an objective rather than subjective phenomenon; a colleague of mine was left with this impression after hearing Lehrer discuss "The Truth Wears Off" on National Public Radio. Moreover, Lehrer too quickly rules out fraud, especially in the case of reporting on drug trials, where the financial stakes are huge; he attributes the decline effect to "subtle omissions and unconscious misperceptions, as researchers struggle to make sense of their results."

But these are quibbles. My main complaint is that Lehrer makes science as a whole sound much "truthier" than it really is. His article was first pointed out to me by my friend Valerie, who believes in homeopathy and tarot cards. The article confirmed her suspicions that mainstream science and medicine may not be based on evidence any more solid than her supposedly (and IMHO, actually—sorry, Valerie) pseudoscientific beliefs. Lehrer’s broad-brush critique will no doubt also cheer global-warming deniers, creationists, postmodernists and other pesky challengers of scientific orthodoxy.

Lehrer himself seems to have realized that he went too far. On his blog The Frontal Cortex, he dismisses the notion that "The Truth Wears Off" implicitly undermines the status of the theory of evolution by natural selection and global warming, which are "two of the most robust and widely tested theories of modern science." He also denies that he is "some sort of Derridean postmodernist, trying to turn publication bias into an excuse to not believe in anything."

But here is how Lehrer ends his article: "Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. When the experiments are done, we still have to choose what to believe." This assertion is absurd. We may choose to believe in psychoanalysis rather than behaviorism, because both are equally flimsy. But the evidence is rock-solid for quantum mechanics, general relativity, the germ theory of infectious disease, the genetic code and many other building blocks of scientific knowledge, which have yielded applications that have transformed our world. There’s nothing truthy about a hydrogen bomb.

If Lehrer didn’t really mean that belief in a given scientific claim is always a matter of choice, why did he say it? He apparently decided, like many scientists, that truthiness would make a bigger splash than truth.

Image: Wikimedia Commons

Tags:





Rights & Permissions

Comments 16 Comments

Add Comment
  1. 1. promytius 5:48 pm 12/13/2010

    I have also noticed that science writing/reporting has eroded, with bad grammar, incorrect facts and references, poor syntax, generalizations and rambling, better suited/expected to be found in entertainment/gossip reporting. While basic science seems to hold to the center, the ever increasing database of research and reports and findings and discoveries makes it impossible to keep up with and easier to discern how little we really do know. 4% of the universe is what some estimate we grasp; most agree that we really can’t account for 2/3 of the universe; but then again, that could just be bad research, I suppose. The recent recalculation of quantum weights and forces are a good example of refining research that while causing a lot of scrambling, point towards deepening our understanding of the universe and all its stuff. This article was very thought provoking and pleasing to read, knowing that I’m not the only hardass skeptic left on the planet. Still, we need PROOF of this article! With follow-up studies, it may very well erode.

    Link to this
  2. 2. chrisad2 5:55 pm 12/13/2010

    John,
    I am both surprised and disappointed that a writer for this journal would call my science (of psychology) "squishy," and refer to the principles of behaviorism as "flimsy." Is your understanding of my discipline really that meager that you are comfortable making such sweeping generalizations?

    Link to this
  3. 3. jvajda 6:54 pm 12/13/2010

    Lehrer’s thinking is philosophically contradictory. He uses the rational scientific perspective to show that some science ends up being wrong. But he still is relying on the scientific method to determine that.

    I see nothing new here, only the banal observation that some science is more rigorous and trustworthy than other science.

    Link to this
  4. 4. Mr. Peabody II 9:31 pm 12/13/2010

    Obviously, many scintific claims have survived the test of time. Why is it so different now?

    For all the education and research that may have resulted in it, a scientific claim is ultimately nothing more than a personal observation. If it delines into "truthiness", it can only be due to poor research, poor communication, or poor understanding.

    It is totally sequitur for Lehrer to imply that blame for the "decline effect" lies in the failings of science and scientists. In fact, the effect is caused by the rapid decline in the quality of human consciousness as a whole — a growing inability of humans to stop thinking like animals.

    Lehrer is an excellent example.

    Link to this
  5. 5. bamsel 10:38 pm 12/13/2010

    "Lehrer’s broad-brush critique will no doubt also cheer global-warming deniers, creationists, postmodernists and other pesky challengers of scientific orthodoxy."

    This may be true, but pesky challengers of scientific orthodoxy engage in the same flawed selection of information that–as Lehrer pointed out–scientific journals sometimes do. Global-warming deniers sift through the scientific evidence in search of confirmatory evidence that their position is right (and they need to sift far and wide to find it). Certain scientific journals sift through null results to find statistically significant results, and then publish them. More accurately, they don’t need to sift at all, but can sit back and wait for the submissions of statistically significant effects to come to them. And in their defence, they often have no way of knowing the ratio of positive to negative replications of the effects in question.

    Consider the following scenario:

    If 20 independent labs run a single experiment to test some hypothesized effect E, which in fact does not exist, approximately 1 of these labs will have statistical evidence for effect E. If the existence of effect E would be particularly exciting or revolutionary, this lab would naturally have reason for excitement, and may submit a manuscript outlining effect E to a top journal. Given the gravity of effect E, this top journal may choose to publish the report (assuming sound methodology and statistics were evidenced by the authors–including an appropriate sample size). The other 19 labs, however, would be forced to try again, or move on. The ‘successful’ lab has contributed a falsehood to the scientific canon. Did they do anything wrong? Actually, they did no more wrong than the other labs.

    There are at least two ways to minimize the probability of this situation occurring, and each are in practice by only some scientists, and only some disciplines.

    One, replicating an experiment before attempting to publish. In the above example, if every lab performed three replications, it would be unlikely that any of them would find the effect even 2 out of 3 times, let alone all three, which would give each lab reason for pause.

    Two, sharing null results with other scientists, which can be implemented in several mediums: conferences, emails, blogs, open-access journals, etc.). In the above example, if the lab that found the effect subsequently found out about the other 19 experiments, their initial disappointment would be tempered with the knowledge that they avoided publishing a likely bomb.

    Link to this
  6. 6. Bops 3:32 am 12/14/2010

    I don’t understand why people waste their time writing articles that almost everyone rejects. After a while, when you see the authors name, you just click past it.

    Link to this
  7. 7. o_nate 12:07 pm 12/14/2010

    Lehrer has brought up a real problem which is naturally somewhat embarrassing to many scientists – although it probably doesn’t come as a surprise to any of them. Your complaint seems to be mainly that Lehrer doesn’t talk enough about the many other scientific findings which have been firmly established – but that wasn’t his purpose in this article. I think you underestimate the readership of the New Yorker to think that many people will dismiss all of science on the basis of these examples. The troubling cases that Lehrer discusses are real, and they won’t go away just because we’d rather not think about them. It would be good for all of us to realize that new scientific findings should be taken with a grain of salt until they have been solidly established, especially when they influence medical practices that can impact people’s health and well-being.

    Link to this
  8. 8. wcampos 3:16 pm 12/14/2010

    I agree with the author. I agree that the theory of evolution by natural selection is "one of the most robust and widely tested theory of modern science." Unhappily, evidence for "global warming" is not as sound as for the "theory of evolution". The authors has been biased by his beliefs!

    Link to this
  9. 9. BryanBunch 3:41 pm 12/14/2010

    Leher says "Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. " This is essentially a statement of Goedel’s Theorem (and its converse, also a theorem). So that is a good description of where we stand in mathematics. Given the "unreasonable effectiveness of mathematics in the natural sciences" (Wigner), then I certainly would go along with Leher on this, although perhaps not on his main point. There is very little truthiness, if any, in mathematics, but I can’t speak for social sciences.

    Link to this
  10. 10. omnologos 5:05 pm 12/14/2010

    > he dismisses the notion that "The Truth Wears Off"
    > implicitly undermines the status of the theory of
    > evolution by natural selection and global warming,
    > which are "two of the most robust and widely
    > tested theories of modern science."

    I wish people were more confident in their science and less defensive on subjects that they consider "robust" and "widely tested". To me, it is obvious that the "Truth that wore off" about evolution was Eugenics. It has all the characteristics indicated by Lehrer, including Galton’s "dramatic correlation" and a huge bandwagon that was eliminated only by the horrors of WWII.

    Likewise for "global warming": a misnomer as everybody now agrees, should be "climate change" at least, and it has evolved from simplistic claims of an increase in temperatures everywhere to a whole load of nuances and lots of studies still to be carried out at a regional (and even more, local!) level. The "Truth" that is wearing off "climate change" is the idea that it only takes a few years to properly understand the behavior in the free atmosphere of something that can be seen in the lab. Other "global warming Truths"/bandwagons that are slowly disappearing include the notions that (a) every environmental phenomenon is caused by increases in CO2 emissions, (b) we have all the technology we need to stop emitting CO2, (c) cap-and-trade is the solution to CO2 emissions, (d) it is ok to present data devoid of uncertainty for policy reasons, (e) reconstructions of past temperatures can be done without involving statisticians, etc etc.

    Please do note that Evolution (in a modern form) has survived the demise of Eugenics, just like "climate change" will likely survive (in an updated form) all semi-idiotic studies forever linking it to the disappearance of mostly-cute animals.

    Sometimes I feel like we have learned nothing of the useless debates of old, Newtonians vs Leibnitzians, light-is-a-particle vs light-is-a-wave, relativity vs quantum mechanics. Science shouldn’t be a place where people sacrifice themselves and their principles for pet theories, closing their minds rather than accepting the challenge: rather, Science should be an open battlefield where the truly powerful ideas don’t even need defending. But I suppose that might not chime right if the worry is the preservation of the status quo.

    Link to this
  11. 11. rwstutler 7:28 pm 12/14/2010

    Jonah Lehrer writes for pay, and is rewarded for writing that draws the largest audience. The fact that he gets a lot of what he writes about correct is the most noteworthy thing to me. Balancing sensationalism with accuracy is a fine line to walk, and science writers and popularizers have been derided for trying to write to general audiences (thus sensationalizing the science) forever. If Lehrer’s readers aren’t smart enough to read carefully and think criticaly, it is not his fault. His job is to sell papers, and internet subscriptions (or books).

    Link to this
  12. 12. sleeprun 9:21 pm 12/14/2010

    Leher is a pop journalist and, as such, gets paid to write what people want to hear/already believe. That’s fine. Journalists need to eat too.

    Knowledge that contradicts pop beliefs can pretty much only be found in science. We will, effectively, not pay — attention, time or money — to receive information that contradicts our individual belief systems. Is it imperfect? Of course.

    We subscribe to Feynemann’s view "All science is wrong, but some is less wrong."

    It is all metaphor after all and severely flawed as is human brain processing of the world and experience.

    The human brain is the last frontier. Things are always uncharted and scary on the frontier. Another good quote from Ioannidis "Science is a low yield activity."

    What is the alternative? And, in fact, science has had a few successes: airplane flight, medical science, the LHC.

    Link to this
  13. 13. J.Conner 3:02 pm 12/15/2010

    There is a critical distinction between false and falsifiability. Gravity is falsifiable by thought induced levitation and evolution is falsifiable by rabbit fossils in the Cambrian. That is what separates them from divine principles that cannot conceivably be false.

    Climatologists invite attack when they take proposals for falsification parameters as claims that the theory is false. However dangerous scientists believe questioning the theory is, denying the possibility of falsification puts climate change on the same logical ground as canon. If scientists are wondering why the public is becoming so skeptical, it is the attitude that climatology should be both held at the same level as falsifiable theories such as gravity and evolution, yet be exempt from the hypothetical possibility of falsification.

    It is also self defeating to hold back evolutionary science to prevent encouraging creationists. Creationists are driven by the Book of Genesis and they will invent any reason they need to rationalize it. Natural selection is a woefully inadequate mechanism for explaining genetic evolution yet it is treated as the law on stone because of the fear of creationists using it to justify their own beliefs. This marginalizes the genetic evolution that could actually give us real, huge answers.

    On a final note, As for the Ghost of Eugenics Past, Dutch researchers have just discovered that mental retardation comes from <i>de novo</i> mutations and not genetic inheritance. This is evolution happening right here, right now, and without natural selection.

    Link to this
  14. 14. Hyperion2010 4:06 pm 12/15/2010

    I would tend to say that once science knowledge is incorporated into engineering most people tend not to dispute its truth, as unfortunate as that may be. The two examples that come to mind for me are quantum physics and chemistry. These disciplines are integral to products that the majority of people in the US have contact with every day of their lives. If something isn’t useful to the majority then they wont really care whether it is true or not. It is up to the scientific community to get their research into practical applications.

    Link to this
  15. 15. Bossi 10:59 pm 12/15/2010

    This reminds me of a PHD comic from last year:
    http://www.phdcomics.com/comics/archive.php?comicid=1174
    :)

    Link to this
  16. 16. zstansfi 1:00 am 12/22/2010

    "If Lehrer didn’t really mean that belief in a given scientific claim is always a matter of choice, why did he say it? He apparently decided, like many scientists, that truthiness would make a bigger splash than truth."

    Smacks of publication bias. You’ve gotta love the irony.

    Link to this

Add a Comment
You must sign in or register as a ScientificAmerican.com member to submit a comment.

More from Scientific American

Scientific American Dinosaurs

Get Total Access to our Digital Anthology

1,200 Articles

Order Now - Just $39! >

X

Email this Article

X