This article was published in Scientific American’s former blog network and reflects the views of the author, not necessarily those of Scientific American
I recently knocked science journalist Chris Mooney for asserting that "You Have No Business Challenging Scientific Experts." Non-experts have the right and even the duty, I retorted, to question scientific experts, who often get things wrong.
Far from reconsidering his stance, Mooney doubles down on it in a Washington Post column, "The science of why you really should listen to science and experts," that defends not just scientific experts but experts in general. Mooney ends up not boosting experts' credibility but undermining his own.
He cites a study that found that judges and other lawyers show less ideological bias—or "identity-protective cognition"--in their application of the law than law students and lay people. Titled "Ideology' or 'Situation Sense'? An Experimental Investigation of Motivated Reasoning and Professional Judgment," the study was carried out by Yale law and psychology professor Dan Kahan and five other scholars.
On supporting science journalism
If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.
To my mind, the study merely shows that lawyers and judges know the law better than law students and non-lawyers. That's reassuring, but surely it does not mean we should always trust lawyers' legal advice, especially since lawyers so often disagree on interpretations of the law. Consider the rancor of recent debates on health care, immigration, taxes, the environment and other issues in Washington, where more than one third of current Representatives and one half of Senators have law degrees, according to The National Law Journal.
Mooney nonetheless insists that the Kahan study "fits nicely alongside a growing trend toward robustly defending and reaffirming the importance of experts." As an example of this trend, he cites the 2005 book Expert Political Judgment by political psychologist Philip Tetlock.
Mooney's citation of Tetlock is bizarre, because Expert Political Judgment—far from a defense of experts—is a devastating critique of them. Tetlock reports on his long-term study of 284 professional pundits, including academics, government officials and journalists, who comment on politics and related issues in scholarly journals and conferences and via mass media. Over two decades, Tetlock assessed the accuracy of 28,000 predictions by the experts related to wars, elections, economic collapses and other events. In a terrific 2005 review, "Everybody's An Expert," New Yorker writer Louis Menand summarizes Tetlock's conclusions as follows:
"people who make prediction their business—people who appear as experts on television, get quoted in newspaper articles, advise governments and businesses, and participate in punditry roundtables—are no better than the rest of us. When they’re wrong, they’re rarely held accountable, and they rarely admit it, either. They insist that they were just off on timing, or blindsided by an improbable event, or almost right, or wrong for the right reasons. They have the same repertoire of self-justifications that everyone has, and are no more inclined than anyone else to revise their beliefs about the way the world works, or ought to work, just because they made a mistake. No one is paying you for your gratuitous opinions about other people, but the experts are being paid, and Tetlock claims that the better known and more frequently quoted they are, the less reliable their guesses about the future are likely to be. The accuracy of an expert’s predictions actually has an inverse relationship to his or her self-confidence, renown, and, beyond a certain point, depth of knowledge. People who follow current events by reading the papers and newsmagazines regularly can guess what is likely to happen about as accurately as the specialists whom the papers quote."
Menand's review is loaded with gleeful one-liners, including this one: "Human beings who spend their lives studying the state of the world, in other words, are poorer forecasters than dart-throwing monkeys." And yet this is no laughing matter. Consider how "experts" in the government, academia and media helped enable the catastrophic U.S. wars in Afghanistan and Iraq and the economic collapse of 2008. Example: New York Times columnist Thomas Friedman, who just before the U.S. invasion of Iraq expressed the hope that it would lead to "a more accountable, progressive and democratizing regime."
How can Mooney possibly interpret Tetlock's book as a defense of experts? Here's how. He seizes on Tetlock's finding that some experts were better forecasters than others. They tended to be not what Tetlock calls "hedgehogs," who explain the world in terms of one big unified theory, but "foxes." Foxes, Tetlock explains, "are skeptical of grand schemes" and "diffident about their own forecasting prowess."
In other words, the most credible experts are those who, implicitly, warn us to be wary of experts. Mooney is oblivious to this irony. "So experts really do exist," he blithely concludes, "and they really are different from non-experts. Now, all we have to do is listen to them."
I prefer Menand's conclusion. He writes that "the best lesson of Tetlock’s book may be the one that he seems most reluctant to draw: Think for yourself."
Addendum: Listen to me talk about the need to challenge experts on New Hampshire Public Radio: http://nhpr.org/post/4615-challenging-experts-creationism-europe-world-beyond-your-head.