This article was published in Scientific American’s former blog network and reflects the views of the author, not necessarily those of Scientific American
So Hilda Bastian and I have been going back and forth on this recent exercise and depression study, and getting way past the study itself, and in to the nitty gritty of bad press releases, and what really makes for a good clinical trial. I hope you've been following along! It's interesting and fun, and really shows what kind of conversations we have in science...pretty much all the time.
So to bring you up to speed...
1. Bristol university publishes a press release, and then a paper, about the results of the TREAD study, on exercise interventions and depression scores.
On supporting science journalism
If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.
2. The internet goes wild. "Exercise won't help your depression! We've all been LIED TO!"
3. I write a "scalping" (thanks dude on Twitter!) of the study, critiquing the methodology and particularly the media coverage and press release.
4. Hilda writes a scalping of my scalping.
5. I wrote a critique of Hilda's critique of MY critique!
6. And here we are, at Hilda's spawn of the critique of my critique of her critique of...ACK THERE ARE TOO MANY. But it's fun. :)
Anyway, here she is! Please welcome Hilda for a few more comments on clinical trials!!!
(Courtesy of Hilda and her blog, Statistically Funny, used via Creative Commons License)
The one about why depression and hypothesis testing are both tough nuts to crack
Hilda Bastian
14 June 2012
Tanked up with early morning coffee and sending out both cheers and “hats off” to @Scicurious. If you’ve just joined us, see @Scicurious’ excellent explanation of the previous episodes in her blog post [1], including how a disagreement about the meaning of a particular trial [2] has led to a mutual admiration society.
@Scicurious and I share a passion, not just about science, but also about bad press releases about research. And she surely nailed a culprit to the wall in this statement from one of the authors: “...but this carefully designed research study has shown that exercise does not appear to be effective in treating depression.” Even though their intervention would have likely shown some benefit if exercise per se were very powerful, lack of effect of this intervention is a far cry from proving absence of effect of exercise.
But I don’t think the occasional ill-advised sentence (bad though that one is) is the biggest culprit behind the endless wave of misleading health claims in the media though. It’s not even the tendency of researchers to exaggerate the importance of their own work - or not give the same care to a press release as they do to the other reports of their research.
That sentence was contradicted by the other contents of the press release and the research article itself. More journalists should have been able to see that – and be skilled at countering an imbalanced perspective without throwing more imbalance into the mix.
A bigger problem, it seems to me, is people taking statistical associations from hypothesis-generating analyses, and treating them as though an association proves cause and effect.
I don’t know all the details behind this trial [3] of course. But this trial could only find actual answers to what it was set up to test (as @Scicurious showed). Leaving the zone of the randomized analyses and going on to data dredging after the trial is finished could cause even bigger problems. How this can lead us horribly astray is explained in Testing Treatments [4] or Ioannidis’ classic article on why most research findings are false [5].
We’ve got no shortage of hypotheses about exercise and depression. Dose might be an issue – but then again, maybe that’s not it. Maybe it’s getting outside. Maybe social issues are the main factors. Maybe no matter how many hours you spent alone on a treadmill without leaving the house, like a hamster in a wheel, it wouldn’t help. Maybe that would make it worse.
But even if a particular amount or type of exercise worked if you did it: could people do it if they were depressed? It doesn’t matter if something is theoretically efficacious, but too hard to do. Or only possible at great expense that puts it out of reach. For some, exercising could be a lifeline. That said, even in the best of circumstances we haven’t found very powerful interventions for getting people to exercise who aren’t already into it.
And what are the adverse effects of pushing the exercise idea? This is especially important where the condition might mean it’s a triumph of the human spirit to be getting on with life at all – and stigma or self-blame don’t help. There could be people around you that just think you need to pull up your socks and get over it, and the whole “exercise thing” might feed into that.
We need more hypothesis testing and less hypothesis generating. Ideally trials that reflect real life for people struggling with depression - and with blinded outcome assessment to reduce self-rating biases. And if the trials find a way to really reduce suffering from depression in a practical way, then both @Scicurious and I will be happier too.
Footnote: And if there is anyone wondering where I stand (or sit!) on exercise in real life – you’d only see me running if I was late for a plane.
[3] http://www.bmj.com/content/344/bmj.e2758
[4] http://www.ncbi.nlm.nih.gov/books/NBK66204/
[5] http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.0020124