About the SA Blog Network

Talking back

Talking back

A science blog, sans blague
Talking back Home

Statistical Flaw Punctuates Brain Research in Elite Journals

The views expressed are those of the author and are not necessarily those of Scientific American.

Email   PrintPrint

Neuroscientists need a statistics refresher.

That is the message of a new analysis in Nature Neuroscience that shows that more than half of 314 articles on neuroscience in elite journals   during an 18-month period failed to take adequate measures to ensure that statistically significant study results were not, in fact, erroneous. Consequently, at  least some of the results from papers in journals like Nature, Science, Nature Neuroscience and Cell were likely to be false positives, even after going through the arduous peer-review gauntlet.

The problem of false positives appears to be rooted in the growing sophistication of both the tools and observations made by neuroscientists.  The increasing complexity poses a challenge to one of the fundamental assumptions made in statistical testing, that each observation, perhaps of an electrical signal from a particular neuron, has nothing to do with a subsequent observation, such as another signal from that same neuron.

In fact, though, it is common in neuroscience experiments—and in studies in other areas of  biology—to produce readings that are not independent of one another. Signals from the same neuron are often more similar than signals from different neurons, and thus the data points are said by statisticians to be clustered, or “nested.” To accommodate the similarity among signals, the authors from VU University Medical Center and other Dutch institutions suggest that a technique called multilevel analysis is needed to take the clustering of data points into account.

No adequate correction was made in any of the 53 percent of the 314 papers that contained clustered data when surveyed in 2012 and the first half of 2013. “We didn’t see any of the studies use the correct multi-level analysis,” says Sophie van der Sluis, the lead researcher. Seven percent of the studies did take steps to account for clustering, but these methods were much less sensitive than multi-level analysis in detecting actual biological effects.  The researchers note that some of the studies surveyed probably report false-positive results, although they couldn’t extract enough information to quantify precisely how many.  Failure to statistically correct for the clustering  in the data can increase the probability of false-positive findings to as high as 80 percent—a risk of no more than 5 percent is normally deemed acceptable.

Jonathan D. Victor, a professor of neuroscience at Weill Cornell Medical College had praise for the study, saying it “raises consciousness about the pitfalls specific to a nested design and then counsels you as to how to create a good nested design given limited resources.”

Emery N. Brown, a professor of computational neuroscience in the department of brain and cognitive sciences at the MIT-Harvard Division of Health Sciences and Technology, points to a dire need to bolster the level of statistical sophistication brought to bear in neuroscience studies. “There’s a fundamental flaw in the system and the fundamental flaw is basically that neuroscientists don’t know enough statistics to do the right things and there’s not enough statisticians working in neuroscience to help that.”

The issue of reproducibility of research results has preoccupied the editors of many top journals in recent years. The Nature journals have instituted a checklist to help authors on reporting on the methods used in their research, a list that inquires about whether the statistical objectives for a particular study were met. (Scientific American is part of the Nature Publishing Group.) The one clear message from studies like that of van der Sluis and others is that the statistician will take on an increasingly pivotal role as the field moves ahead in deciphering ever more dense networks of neural signaling.

Image Source: Zache

Gary Stix About the Author: Gary Stix, a senior editor, commissions, writes, and edits features, news articles and Web blogs for SCIENTIFIC AMERICAN. His area of coverage is neuroscience. He also has frequently been the issue or section editor for special issues or reports on topics ranging from nanotechnology to obesity. He has worked for more than 20 years at SCIENTIFIC AMERICAN, following three years as a science journalist at IEEE Spectrum, the flagship publication for the Institute of Electrical and Electronics Engineers. He has an undergraduate degree in journalism from New York University. With his wife, Miriam Lacob, he wrote a general primer on technology called Who Gives a Gigabyte? Follow on Twitter @@gstix1.

The views expressed are those of the author and are not necessarily those of Scientific American.

Rights & Permissions

Comments 49 Comments

Add Comment
  1. 1. Jerzy v. 3.0. 8:09 am 03/28/2014

    Not just brain research, most fields of medical research.

    Link to this
  2. 2. Martinwilson 4:03 am 03/29/2014

    obviously, research on different fields helping us to explore more about it. One should only work on result oriented matters. Do my assignment –

    Link to this
  3. 3. MSkigen 7:11 am 03/29/2014

    I especially have noticed the studies where they take people who are already diagnosed with something, especially something that only really manifests outwardly through behavior, and mistake the identified population (people who have behaved in this way significantly enough for a diagnosis to be pursued) with everyone who has that situation. Therefore, weird stuff like “there are more autistic boys than girls,” when girls’ behaviors are less likely to be pursued, are more inwardly directed and therefore less noticeable, etc.

    Link to this
  4. 4. fustbariclation 12:08 am 03/30/2014

    This isn’t quite as clear as it could be. It says ‘Failure to statistically correct for the clustering in the data can increase the probability of false-positive findings to as high as 80 percent—a risk of no more than 5 percent is normally deemed acceptable.’. Forgetting the split infinitive (‘failure statistically to correct’ would be better), it raises a few questions for me.

    Firstly, the 5% deeming seems quite arbitrary. In Bayesian statistics, you decide on the ‘acceptable’ level based on likely outcomes, not on an arbitrary figure.

    The comment about the false positive isn’t clear. There are four outcomes possible from an experiment – true positive, false positive, true negative, false negative. If these were equiprobable then the chance of a false positive would be .25. They obviously, though, aren’t equiprobable. You need to consider experimental design, limits of measurement and the various sources of error. This will give you a picture of the relative probabilities. If, as they’re saying, the probability of a false positive is .8, then that leaves only .2 between the other probabilities. So, if they’re equiprobable, it’d mean that true positives, true negatives and false positives would be .07 probable.

    Now, if false negatives are .07 probable, which isn’t all that far from .05, then there’s a simple solution. Simply frame the null hypothesis the other way around and, hey-presto, suddenly you’ll have really accurate results. What were your unacceptably high false positives are now your quite reasonable small level of false negatives.

    This doesn’t make a lot of sense, so there must be more to what this .8 probability means than the article is saying.

    Link to this
  5. 5. BeatriceDavey 12:45 am 04/23/2014

    Brains works on thousands of neuron which we are unable to count yet. Its approximately 10^11 which still a large number.|Custom Dissertation by Mighty Dissertations||Write my Dissertation For me|

    Link to this
  6. 6. amyhamilton 4:41 am 05/6/2014

    Nice to be visiting your blog again, it has been months for me. Well this article that i’ve been waited for so long. I need this article to complete my assignment in the college, and it has same topic with your article.
    Essay RHN

    Link to this
  7. 7. JessicaValdezes 7:24 am 05/27/2014

    Its been a great research and very much helpful to make research paper. I was planing to buy cheap research papers but your article made it very easy for me to make my research paper on time.

    Link to this
  8. 8. Vol1234 11:40 am 07/30/2014

    I hope you can continue this type of hard work to this site in future also..Because this article is really very informative and it helps me lot.
    Essay writing services –

    Link to this
  9. 9. JeannieWebb 2:06 pm 08/19/2014

    It is absolutely difficult for me to get the points. I simply recommend to buy medicines online form so its easy to get your self relax and happy.

    Link to this
  10. 10. aurtherlewis 3:06 pm 09/2/2014

    Working on the give topic is really a difficult task but your this tutorial made it easy for me to handle it without any problem. If anyone seeking information then this blog is the best place for him as it has quality content for your required information. Thanks for sharing.
    help with essay

    Link to this
  11. 11. aurtherlewis 6:02 pm 09/4/2014

    It is absolutely difficult for me to get the points.Nice to be visiting your blog again, it has been months for me.i’ve been waited for so long.Great site and a great topic as well i really get amazed to read this.Order Assignment

    Link to this
  12. 12. sharenstuart 3:52 am 09/6/2014

    This message will be most reassuring if research analyzing always follow through with their promise to meet with these statistics so the others are doing independent work.Thanks for sharing.
    Dissertation Mall

    Link to this
  13. 13. ShaunWalker 1:39 am 09/12/2014

    A recent paper in Nature Neuroscience is suggesting that Systems Neuroscience has nearly as big of a problem with statistics as the fMRI field used to.In neuroscience, experimental designs in which multiple observations.
    Assignment Writing-Assignment Kingdom

    Link to this
  14. 14. aurtherlewis 6:38 pm 09/23/2014

    I love your designs! I would love to have this as a personal planner.I am glad I read this.
    Order Assignment

    Link to this
  15. 15. kolinjennis 4:55 am 10/7/2014

    Fascinating information. But I don’t understand everything as I’m only entering the world of neuroscience. I recently wrote an entering essay with the best essay writing service.

    Link to this
  16. 16. vanessahill70 8:11 am 10/13/2014

    I especially have noticed the studies where they take people who are already diagnosed with something, especially something that only really manifests outwardly through behavior, and mistake the identified population (people who have behaved in this way significantly enough for a diagnosis to be pursued) with everyone who has that situation.
    Life Experience Degree

    Link to this
  17. 17. JohnJames12 3:07 am 10/14/2014

    Nice blog about research. Medical research also covers a wide area of different fields. Thanks
    Marketing Assignment Help

    Link to this
  18. 18. danielmm 7:46 am 10/14/2014

    Research on different topics are more important these days we found many new things with the help of research

    Custom Essay Writing

    Link to this
  19. 19. michaelrodgers 7:48 am 10/16/2014

    Research Work Always Difficult for Scholars, but This post really helpful. Interesting to Read.Research Assignment Writing

    Link to this
  20. 20. MrKapilSharma 11:11 am 10/19/2014

    After cutting the Corpus Callosum, the left and right hemispheres of the brain still works well, but not knowing what the other is doing.

    Link to this
  21. 21. jacquelinef 1:37 am 11/8/2014

    Thank you for your post, I look for such article along time, today i find it finally. this post give me lots of advise it is very useful for me. nice i like it.educational management software

    Link to this
  22. 22. Teachers 8:03 am 11/10/2014

    Great blog for the visitor they really like it such a nice information here.
    MBA Assignment Help

    Link to this
  23. 23. balfourjohn 11:53 pm 11/17/2014

    Research Work Always Difficult for Scholars, but This post really helpful. Interesting to Read.
    order research paper online

    Link to this
  24. 24. tzwriting 6:35 am 11/18/2014

    Please write my paper for me tell us where you read or heard it.

    Link to this
  25. 25. Yourtutors 5:42 am 11/22/2014

    nice stuff for the sharing great blog to read.
    Order Online Assignment Help

    Link to this
  26. 26. Educators 8:25 am 11/22/2014

    super post I like it.
    Do Collage Assignment Help

    Link to this
  27. 27. Phil88 3:08 am 11/24/2014

    In my opinion, this is really a much detailed, informative and authentic article post shared to all. Buy Dissertation Online

    Link to this
  28. 28. waltersmith 1:03 am 11/28/2014

    our whole basis of humanity rests on the notion of free will – the law, the accreditation of praise or blame, the whole notion of history Pay for College Homework Help Online?

    Link to this
  29. 29. Maryfernando12 3:05 am 12/1/2014

    Let us make all your dreams turn into reality, by providing you with the content that will let you score an A grade on your academic course!Online Pay money to write papers

    Link to this
  30. 30. michaeljayden 3:06 am 12/1/2014

    Very true. Research plays an important role these days many universities or companies do research for their betterment.

    Professional busniness plan writer

    Link to this
  31. 31. AdamMartin 4:32 pm 12/9/2014

    Thank you for your placement, I appear for such editorial down instance, nowadays i locate it lastly. This position offers me plenty of advice it is very practical for me. Nice i approximating it… Geography Coursework Writers

    Link to this
  32. 32. jpbidding 6:04 am 01/22/2015

    Jpbidding is the all-round deputy & export company from Japan.We take the hassle out of buying, contact to seller, packing, and shipping.
    yahoo auction agency:

    Link to this
  33. 33. customassignment 1:22 am 02/3/2015

    We provide essay writing service for all of the students to prepare term papers or college papers.

    Link to this
  34. 34. PauletteDodson 5:01 am 02/4/2015

    I been looking for that kinda stuff, actually a friend on mine was asking me to do may assignment for me. topic was really irritation first i heard about it.. but after reading your research got max in my Neuroscience :)

    Link to this
  35. 35. michaelrodgers 2:18 am 02/5/2015

    A nice blog on brain procedure to work it works on thousands of neurons. Such a well or in depth written blog, if you are going to research on this kind of topic and need help.. you can apply from Assignment Writing in future.

    Link to this
  36. 36. ameeliabrown 3:31 am 02/13/2015

    Very nice post Gary, readers found this post very informative


    assignment help

    Link to this
  37. 37. kolinjennis 4:55 am 02/13/2015

    Those are very interesting points. The author did a great job researching the material as I didn’t find some of those facts while preparing a paper for .

    Link to this
  38. 38. SunnyEssays 3:31 am 02/17/2015

    Genuine. Exploration assumes a vital part nowadays numerous colleges or organizations do research for their improvement.

    Link to this
  39. 39. RobertGrace 8:55 am 02/27/2015

    I always want to know the entire process of universe growing but I suppose that it would be discovered not in our age. But still space is charming and amazing so to attract more people for studying it we need more popularization of this topic. As I am working as a proofreader at GrabMyEssay I’ve noticed that people are interested in galaxies and stars so I fave a hope for our future.

    Link to this
  40. 40. Abelee 12:46 pm 03/16/2015

    Hello, an amazing Information dude. Thanks for sharing this nice information with us.
    Buy Percocet online

    Link to this
  41. 41. liamjayden011122 10:16 am 03/27/2015

    This makes a lot of sense and learned from the post which is most likely useful for me. Thanks much
    please visit

    Link to this
  42. 42. Simon123 2:31 am 04/2/2015

    We give paper composing administration to the majority of the understudies to get ready research projects or school papers.Need help for academic research paper writing

    Link to this
  43. 43. Assignmentwriting 12:56 am 04/3/2015

    A nice piece of writing, i enjoyed reading this blog…Thank you for sharing this post with us…Assignment writing services

    Link to this
  44. 44. EssayMaxi 4:25 pm 04/7/2015

    Nice writing, I also try to write in Essay Writing Service, but it is not easy work

    Link to this
  45. 45. Tobyyoung 2:13 am 04/18/2015

    I trust you can proceed with this sort of diligent work to this site in future also.Because this article is truly extremely instructive and it helps me part.
    Dissertation Topics

    Link to this
  46. 46. TopQualityAssignment 7:08 am 04/22/2015

    Top quality assignment provides excellent quality of assignment help Australia to help students in attaining better grades.
    Assignment Help Australia for Hire

    Link to this
  47. 47. SunnyEssays 4:33 am 04/24/2015

    This bodes well and gained from the post which is in all probability valuable for me. Says thanks to you to such an extenthomework

    Link to this
  48. 48. MBAThesis 6:51 am 04/24/2015

    Certainly, Statistical part of a paper or thesis is one of the most difficult parts. Hence taking professional help is never a bad idea.

    Link to this
  49. 49. EditingAndProofreading 7:07 am 04/24/2015

    Nice information.
    Apart from the statistical part, the quality of writing should be excellent. EditingAndProofreading provides complete professional help pertaining to editing the research document.

    Link to this

Add a Comment
You must sign in or register as a member to submit a comment.

More from Scientific American

Email this Article