Yesterday Hilda Bastian (of the Statistically funny blog which I only just discovered!) posted to the SciAm Guest Blog, with a thoughtful and useful critique of my critique (I know, so meta!) of the recent exercise and depression study.
And she's got some great points. But you know what? I've got great points, too. :) This sort of back and forth is what makes science really fun and interesting and...science! So Hilda, grab a beer, a coffee, or your drink of choice, and let's get down to business, science style!!
I will tell you...no wait there is too much. I will sum up. To recap.
1. Bristol university publishes a press release, and then a paper, about the results of the TREAD study, on exercise interventions and depression scores.
2. The internet goes wild. "Exercise won't help your depression! We've all been LIED TO!"
3. I write a "scalping" (thanks dude on Twitter!) of the study, critiquing the methodology and particularly the media coverage and press release.
4. Hilda writes a scalping of my scalping.
5. Here we are!
So I'm going to start with the excellent points Hilda has made. First, she's right, I'm not an unbiased creature. I might even be writing this post after a 5 mile run. I'm an acknowledged exercise junkie, and so I really do WANT to believe that exercise will help people with depression. I'd like to think that I would be unbiased enough to recognize compelling data showing the opposite...but I'm only human. I could be wrong. Heck, I'm wrong all the time. And it's good to have someone point that out! We're none of us entirely unbiased.
Secondly, she is right that this is a well-constructed clinical trial. They were looking at the effects of TREAD (a specific intervention aimed at increasing moderate exercise) on depression, and that is what they showed.
Finally, she's right that the humans data for exercise and depression is weaker than many people believe it to be. This is where I think my own biases came into effect, I really do want to believe that data is strong, but in the face of the lit she showed me, I am forced to agree that we really need to do more and better studies.
So these are all excellent points. But there are some places where I disagree with Hilda, and where I think this study did not go far enough, especially when it realistically could have done so.
First, the press release and paper. Hilda's right that the paper does not overstate the findings...but the press release is another matter entirely.
“Numerous studies have reported the positive effects of physical activity for people suffering with depression but our intervention was not an effective strategy for reducing symptoms."
Exercise and activity appeared to offer promise as one such treatment, but this carefully designed research study has shown that exercise does not appear to be effective in treating depression.
These are two sections of the press release which really suggest that EXERCISE, not just their intervention, is ineffective for depression. As I stated in my previous post, that's not what this study showed. This study showed that their intervention, encouraging people to exercise, did not relieve symptoms of depression. The press release does mention the specific intervention, but definitely suggests that exercise is no good, which is not what this study showed.
And while they did get "a sustained increase in activity in people who were working with our activity facilitators", it was not a stunning increase (p=0.08 for the first 4 months, for example). In fact, at any one of the time points, it was not significantly different from the control group, and only became so when the groups were all pulled in together. This increase in physical activity that they were looking for was not exactly a big one. They were trying for 150 minutes of moderate to vigorous physical activity per week, in bouts of 10 minutes or more. That includes things like walking. By this measure, I get that much physical activity in my daily commute. This is certainly a fine thing to aim for, and good in terms of cardiovascular health. But, especially given that the experimental group did not significantly increase from the control group in terms of physical activity at any given time point, I'm not at all surprised they didn't find anything. Not only that, the data was self report. And as the authors themselves acknowledge, it's very easy for people who are in the experimental arm, and encouraged to exercise, to over-report the distance they walked to and from the car, and whether they really took the stairs every day.
Secondly, I agree that they should not have separated out exercisers and non-exercisers in their initial analyses, as Hilda pointed out. This was, after all, the main outcome of their study, the effect of TREAD specifically on depressive measures.
...but why did they stop there?! They had this huge data set, they know who exercised and who didn't, and they didn't think to separate it out and see if they got any differences?! Why not?! They knew who was on what antidepressant, why not look for that too? These are simple numbers tests, and not difficult, and certainly not difficult to justify, if that's the concern. I really don't think this paper went far enough in the analyses, sticking with a primary outcome measure that showed very little, when the end result could have been a lot richer.
I think many of my critiques still stand. It's a well designed study, but I don't feel it's well analyzed. And I still certainly feel that press release overstepped the mark, especially when journalists are looking for something sensational.
But in the end, I think, Hilda and I are on the same page. We don't know the effects of exercise on depression for sure, and this paper doesn't tell us that. I think she's right to point out my biases and make me think of them, it's something I think all scientists should be forced to confront. And she's right, we need more studies, we don't know the effects of exercise on depression. There are many anecdotes, but those aren't data, and until we have a good, vigorous study comparing "doses" of exercise and depression scores as compared to antidepressant treatment (or in addition to antidepressant treatment), in a large cohort? Well, we aren't there yet.