This article was published in Scientific American’s former blog network and reflects the views of the author, not necessarily those of Scientific American
Brain studies are the current darling of the sciences, research capable of garnering tens or even hundreds of millions in new funding for ambitious new projects, the kind of money that was once reserved only for big physics projects.
Except the house of neuroscience, which attracts tens of thousands of attendees each year to the annual meeting of the Society for Neuroscience, may be built on a foundation of clay. Those are the implications of an analysispublished online April 10 in Nature Reviews Neuroscience, which questions the reliability of much of the research in the field.
The study—led by researchers at the University of Bristol—looked at 48 neuroscience meta-analyses (studies of studies) from 2011 and found that their statistical power reaches only 21 percent, meaning that there is only about a one in five chance that any effect being investigated by the researchers—whether a compound acts as an anti-depressant in rat brains, for instance—will be discovered. Anything that does turn up, moreover, is more likely to be false. The low power stems from the small size of the studies and minuscule effects.
On supporting science journalism
If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.
John Ioannidis of Stanford University School of Medicine, says the statistical power of neuroscience studies is actually lower than that found in other areas of biology, which also suffer from the same phenomenon—he cited studies on cancer and cardiology that are powered at 34 percent. "Neuroscience has tremendous potential and it is a very exciting field," Ioannidis says. "However, if it continues to operate with very small studies, its results may not be as credible as one would wish. A combination of small studies with the high popularity of a highly-funded, bandwagon-topic is a high-risk combination and may lead to a lot of irreproducible results and spurious claims for discoveries that are out of proportion."
Update: Moses Chao, a former president of the Society for Neuroscience and a professor of cell biology at New York University Medical School, got back to me with a comment after I posted the blog, which is excerpted here:
"I agree that many published papers in neuroscience are based upon small effects or changes. One issue is that many studies have not been blinded. There have been numerous reports in my field which have not been reproduced, some dealing with small molecule receptor agonists. This has set back progress. The lack of reproducibility is one of the reasons that pharmaceutical companies have reduced their effort in neuroscience research. But irreproducibility also applies to other fields, such as cancer...
"I recently wrote an obituary on Rita Levi-Montalcini, who disdained statistical analysis and only wanted to see results with a big effect!"