September 25, 2012 | 9
I have to say that I am incredibly pleased that this study won the Ignobel. Not just because it’s a really fun study, but also because it really is one of those studies that makes you laugh, and then makes you THINK. And in the case of this study in particular, it has changed a lot about how we think about making corrections in fMRI, and may have actually really affected the way the data are published. And so, I present to you: the dead salmon study.
Bennett et al. “Neural Correlates of Interspecies Perspective Taking in the Post-Mortem Atlantic Salmon: An Argument For Proper Multiple Comparisons Correction” Journal of Serendipitous and Unexpected Results, 2010.
This study began as a fun trial, and almost never saw the light of day. But since it has, it has become a really important study in the field of functional magnetic resonance imaging, which measures changes in blood oxygenation levels in the brain during tasks (usually in humans). These studies are widely used and very widely cited in the media, and they have told us a great deal about the brain, our mental abilities, and certain disease states. But until this study, not all of these studies were really adequately controlled. And now, it looks like things are getting better. And all because of a dead salmon.
It all started when Bennett et al were setting up their…um…real experiment. They were going to look at humans and their responses to social stimuli. But in order to do this, they have to first test the machine. Apparently, when you usually test an fMRI machine, you put a big balloon in there filled with mineral oil. Just to test it and look for contrast, etc. The authors of this study wanted something different, they wanted something with more contrast and different types of texture.
So first they bought a pumpkin.
They got some good signals, but not very good contrast. They next tried a Cornish Game Hen (dead, defeathered, from the store). That also produced good visuals but wasn’t quite what they needed. The authors needed something with good contrast, but also with several clearly defined and distinguishable types of tissue: fat, bone, muscle, etc.
Enter the salmon.
The lead author, Dr. Craig Bennett, wanted to get something fresh, so he headed in to the grocery story first thing in the morning. At the fish counter, he spoke the words that will echo down the centuries as a testimony to the dedication and drive of neuroscientists throughout the ages:
“I need a full length Atlantic Salmon. For science.”
I am still shocked that the guys at the fish counter didn’t give him a discount. Can’t you get a discount for science?!
Having procured the specimen, the authors placed the salmon in the fMRI and ran all the usual anatomical scans, and then ran the experimental set of the study as well. In this study, the salmon was shown images of people in social situations, either socially inclusive situations or socially exclusive situations. The salmon was asked to respond, saying how the person in the situation must be feeling. The salmon, as far as I can tell from the paper, did not comply with instructions. Naughty salmon.
The results were set aside and not looked at for a good while, until one of the other authors of the study was running a seminar on how to properly analyze fMRI data. They wanted to do some improper analysis on something improbably, and remembered that they had the salmon data on the computer. And a study was born.
Now, to clarify: what exactly were they doing? Well, when you do fMRI studies in the brain, there’s a ton of information there. The information is generally broken down into sections called voxels. Up 130,000 of them in a single study and contrast selection, looking at each one to see if it is ‘activated’ compared to the others. And doing the statistics on these studies gets to be a problem. You have to do thousands of comparisons, and you being to run into something called the “multiple comparisons problem“. If you do a lot of tests, at least some of them will come out positive, even if they are not real. These are called false positives, and they are something you really want to watch out for.
To solve this problem, there are various methods for correcting the multiple comparisons, but this also means that you lose a lot of statistical power. In other words, you get rid of your false positives, but it might mean you don’t see things that are really there, you might find false negatives instead. There is a running debate in the fMRI field over whether false positives or false negatives are more dangerous. The authors of this study contend (and I am inclined to agree) that the false positives are more likely to get overblown and lead to problems down the line. For a really good wrapup on the stats questions, I recommend neuroskeptic’s piece on the topic.
So in the final results, the authors compared the normal multiple comparisons, with the multiple CORRECTED comparisons. When they used the multiple corrected comparisons, the dead salmon showed nothing. When they did the multiple comparisons without the correction, the salmon showed significant increases in “activation”, coincidentally, in the brain and spinal cord. This shows the importance of correcting for multiple comparisons and avoiding false positives.
The original poster almost didn’t make it to a conference, but when it did, it made a major splash, and reactions were very positive. Some people like to use the salmon study as proof that fMRI is woo, but this isn’t the case, it’s actually a study to show the importance of correcting your stats.
And the poster, and the paper that was eventually published, may have had an effect on the field. The authors note that at the time the poster was presented, between 25-40% of studies on fMRI being published were NOT using the corrected comparisons. But by the time this group won the Ignobel last week, that number had dropped to 10%. And who knows, it might, in part, be due to a dead fish.
*I should note that at least one of the authors of this study, Dr. Craig Bennett, reads this blog!! I was so thrilled (and slightly scared, I mean, I know people read it, but…I mean what if I screw up?! Clearly I need to attend the session on “when people start taking your online ramblings seriously” at #scio13) when he told me! He’s on twitter and you can follow his sciency ramblings at @prefrontal. And he also provided the excellently seasonal fMRI photos of the pumpkin. Dr. Bennett also notes that he was not reimbursed for the salmon, but that was because they ate it later.
Secrets of the Universe: Past, Present, FutureX