ADVERTISEMENT
  About the SA Blog Network













Talking back

Talking back


A science blog, sans blague
Talking back Home

New Study: Neuroscience Research Gets an “F” for Reliability

The views expressed are those of the author and are not necessarily those of Scientific American.


Email   PrintPrint



Brain studies are  the current darling of the sciences, research capable of garnering  tens or even hundreds of millions in new funding for ambitious new projects, the kind of money that was once reserved only for big physics projects.

Except the house of neuroscience, which attracts tens of thousands of attendees each year to the annual meeting of the Society for Neuroscience, may be built on a foundation of clay. Those are the implications of an analysis published online April 10 in Nature Reviews Neuroscience, which questions the reliability of much of the research in the field.

The study—led by researchers at the University of Bristol—looked at 48 neuroscience meta-analyses (studies of studies) from 2011 and found that their statistical power reaches only 21 percent, meaning that there is only about a one in five chance that any effect being investigated by the researchers—whether a compound acts as an anti-depressant in rat brains, for instance—will be discovered. Anything that does turn up, moreover, is more likely to be false. The low power stems from the small size of the studies and minuscule effects.

John Ioannidis of Stanford University School of Medicine, says the statistical power of neuroscience studies is actually lower than that found in other areas of biology, which also suffer from the same phenomenon—he cited studies on cancer and cardiology that are powered at 34 percent.  “Neuroscience has tremendous potential and it is a very exciting field,” Ioannidis says. “However, if it continues to operate with very small studies, its results may not be as credible as one would wish. A combination of small studies with the high popularity of a highly-funded, bandwagon-topic is a high-risk combination and may lead to a lot of irreproducible results and spurious claims for discoveries that are out of proportion.”

Update: Moses Chao, a former president of the Society for Neuroscience and a professor of cell biology at New York University Medical School, got back to me with a comment after I posted the blog, which is excerpted here:

“I agree that many published papers in neuroscience  are based upon small effects or changes.  One issue is that many studies have not been blinded.  There have been numerous reports in my field which have not been reproduced, some dealing with small molecule receptor agonists.  This has set back progress.  The lack of reproducibility is one of the reasons that pharmaceutical companies have reduced their effort in neuroscience research. But irreproducibility also applies to other fields, such as cancer…

“I recently wrote an obituary on Rita Levi-Montalcini, who disdained  statistical analysis and only wanted to see results with a big effect!”

About the Author: Gary Stix, a senior editor, commissions, writes, and edits features, news articles and Web blogs for SCIENTIFIC AMERICAN. His area of coverage is neuroscience. He also has frequently been the issue or section editor for special issues or reports on topics ranging from nanotechnology to obesity. He has worked for more than 20 years at SCIENTIFIC AMERICAN, following three years as a science journalist at IEEE Spectrum, the flagship publication for the Institute of Electrical and Electronics Engineers. He has an undergraduate degree in journalism from New York University. With his wife, Miriam Lacob, he wrote a general primer on technology called Who Gives a Gigabyte? Follow on Twitter @@gstix1.

The views expressed are those of the author and are not necessarily those of Scientific American.





Rights & Permissions

Comments 9 Comments

Add Comment
  1. 1. N a g n o s t i c 11:42 am 04/10/2013

    Didn’t you know? The US Government’s “Decade of the Brain” has been ongoing for 23 years now.

    What should get an “F” is the ever-expanding number of publicised studies that confirm common knowledge. I guess these researchers need to make a buck, just like the NASA engineers currently working on an asteroid mission that’ll never come to pass.

    Link to this
  2. 2. PatriciaJH 12:07 pm 04/10/2013

    fMRI is expensive, so n is small. I’d wondered about all those small-n studies, seems I was right to wonder.

    Link to this
  3. 3. NeuroJoe 12:22 pm 04/10/2013

    There’s a famous quote by Einstein which says that “Common sense is the collection of prejudices acquired by age eighteen.” There are a lot of phenomena that seem like common sense that aren’t, and vice versa. Paraphrasing Simon Singh from his 2008 book about alternative medicine, is it common sense that driving my car is going to cook the Earth? It is it common sense that injecting viruses into our bodies can save us from disease? Is it common sense that everything on Earth has a microbe as a great great…..grandparent?

    Link to this
  4. 4. swagner 12:25 pm 04/10/2013

    This is a healthy zoom out. It affected me as I read again this morning the recent assertion that tCDS (electrical stim, here of the ACC) can improve “creativity.” Small study, with small results that nevertheless quoted one researcher as being surprised at how large an effect was shown. I tweeted about the results awhile back, saying something about the first outside stimulation shown to affect normal people positively; if I heard about the study today, I might not have tweeted it.

    Score one a little closer to home for the neurobollocks crowd. A welcome ‘physician, heal thyself’ message for the keepers of the temple. I do think that new research money, tools and especially strategies will circumvent the incredible challenge of putting together larger studies. Hopefully, such progress will help swerve us from focusing on small effects designed around potential news worthiness. Just as psychology is slowly turning more toward replication and validation after revelations of its research issues, hopefully neuroscience can benefit from this feedback.

    In a way, this finding supports the movement to prevent neurobabble from taking over the world by showing that even basic neuroscience is incredibly difficult to accomplish as a funded, educated professional. It sends the clear message that dependence on cutting-edge neuroscience is usually misplaced (hard to write, let alone distill as a first principle). It also leaves society with less ability to depend on scientists to get their brain fix, and any resultant relative caution or silence may lead to even more emphasis on pseudoscience and semi-science applications that fill the news gap.

    One healthy takeaway: there is plenty of solid neuroscience worth studying that can help those of us outside the lab to be practical, efficient monists (people who trust that the nervous system will be revealed to be the key enactor of our individual souls, as opposed to a sophisticated conduit for it). We just need to be very patient with our beloved; less demanding this particular century. Not hovering outside the lab so much, peering through the windows and lipreading. The work’s hard enough as is.

    Link to this
  5. 5. Quantumburrito 2:37 pm 04/10/2013

    “What should get an “F” is the ever-expanding number of publicised studies that confirm common knowledge.”

    Well said. I have lost track of the number of papers published in top journals and publicized heavily which just seem to confirm the obvious.

    Link to this
  6. 6. jtdwyer 2:44 pm 04/10/2013

    Excellent report. IMHO, neuroscience, social science, etc., practitioners should all be required to undergo emergency remedial training in the scientific method and statistical evaluation.

    Also see http://blogs.scientificamerican.com/the-curious-wavefunction/2013/04/09/do-you-need-to-know-math-for-doing-great-science/

    Link to this
  7. 7. N a g n o s t i c 5:14 pm 04/10/2013

    Common sense naturally involves the already-known, not the yet-to-be-known. As such it, it can be employed while engaging in healthy prejudice. It is often invoked by the ignorant.

    Link to this
  8. 8. N a g n o s t i c 5:16 pm 04/10/2013

    …Common sense should evolve, as science does.

    Link to this
  9. 9. billinsandiego 2:15 am 04/18/2013

    Thank you thank you thank you. I thought I was a lone wolf complaining about the apparent unchallenged “findings” of neuroscientists. Several have been making bold claims based on limited observations (observations should not be confused with “findings”). I had asked on another blog if they had some secret Rosetta Stone of brain function (via fMRI) that allowed them make statements of such absolute certainty.
    Sherrington would be ashamed of some of the “research studies” being published over the past decade.

    Link to this

Add a Comment
You must sign in or register as a ScientificAmerican.com member to submit a comment.

More from Scientific American

Scientific American Holiday Sale

Black Friday/Cyber Monday Blow-Out Sale

Enter code:
HOLIDAY 2014
at checkout

Get 20% off now! >

X

Email this Article

X