June 4, 2013 | 15
Last week, the anchors at Fox News made headlines when they covered the recent Pew Research Center finding that 40% of all households in America have a female primary breadwinner. About 1/3 of these households consist of two-parent households where the mothers make more money than their husbands, and the remaining 2/3 consist of single mothers. Left completely in a tizzy after learning of this statistic, Fox assembled an entirely-male panel to mourn, in no uncertain terms, the downfall of society as we know it. This was quickly followed by another segment in which lead mouthpiece Erickson doubled down on his statements, and a third in which female anchor Megyn Kelly criticized the men for their backwards, offensive views. Many of the things said on these segments were hurtful to thousands of men, women, and children who were essentially being told that they are responsible for society crumbling into ruins. However, in addition to being offensive, many of the “facts” and much of the “science” on these videos were not quite correct. In a series of blog posts this week, I will be comparing what the Fox news anchors have claimed in these segments with empirical reality.
The Claim: In response to Megyn Kelly citing a 2010 study released by the American Psychological Association stating that there are no differences in positive outcomes between children whose mothers work and those whose mothers stay home, Erick Erickson says, “I think the experts can be as politically motivated as anyone else when it comes to these particular studies, because it plays into a particular current notion that it’s OK.” (Note: For an explainer on how to understand the results of a meta-analysis and more information about the meta-analysis being discussed in this post, please see my post from yesterday).
I’m never quite sure of what this type of accusation actually means, but there are a few possibilities that we can explore.
1. The Researchers Doing The Meta-Analysis Picked And Chose Which Articles To Include Because Of Politics. Maybe Erickson is trying to argue that the meta-analysts themselves are selectively choosing which articles to include in their analysis, leaving out the ones that don’t fit their idea of what’s “right.” However, that is really, truly not how a meta-analysis works. When you conduct a meta-analysis, you have to describe in excruciating detail exactly how you decided which studies to include or exclude. You are not able to decide not to include certain studies simply because you don’t want to do so. And believe me – people read meta-analyses of fields in which they have conducted research. Scientists who conduct research in a certain field certainly want to see their names in print and their work being cited in relevant meta-analyses. If a meta-analyst were to casually leave out someone’s papers because of political motives, believe me — it would NOT stay secret for very long.
So how did the researchers find all of these papers in the first place? Typically, meta-analytic researchers have to conduct a comprehensive search of all relevant databases, reporting which keywords they used in the search. In this case, the researchers searched every major database (including an entire database that consists solely of unpublished dissertation abstracts) for every single paper that had been published from 1960 through 2010 containing the following keywords in every possible combination: maternal, mother(s), parental, employment, work, labor, child care, early, infancy, children(’s), boy(s), girl(s), achievement, academic, cognitive, cognition, school, behavior problems, adjustment, external(izing), internal(izing), defiance, and compliance. They ended up with 216 total studies after searching for these keywords. So how did they end up with 69 studies if they started with 216? Again, they have to detail in the manuscript exactly why each study was excluded – nothing can be left secret. 78 studies were excluded because they did not focus on early employment (perhaps only measuring whether or not mothers were employed when the children were teens or adults), 22 were excluded because they did not measure “maternal employment” in a way that fit with the specified inclusion criteria1, 14 excluded because they did not measure “child outcomes” in a way that fit with the specified inclusion criteria2, 11 studies were excluded because even though they included measures of maternal employment and measures of child outcomes, they did not present any information on how those two things related to each other, 7 studies were excluded because they were review pieces and did not include any empirical data, 8 studies were excluded because the data were used in more than one peer-reviewed study and could not be double counted, 5 studies were not available to be downloaded, and 2 studies did not provide effect sizes that could be coded for the analysis. 216 – 78 – 22 – 14 – 11 – 7 – 8 – 5 – 2 = 69 studies, and not a single one was excluded because of any sort of vague “I don’t like what they said” criteria.
2. Publication Bias. Maybe Erickson is trying to argue that the editors and reviewers of empirical journals are suppressing any research findings that support conservative politics, and journals are therefore only publishing results that support a liberal point of view. Thus, even when researchers comprehensively search all of the relevant databases, their search will still be biased because published papers are all liberally slanted.
This is a fairly paranoid point of view, and not at all how publication/peer review really works, but let’s pretend for the sake of this argument that this is something to be concerned about. If you take a look at the sample characteristics for the meta-analysis (discussed in a previous post), 20% of the samples actually come from unpublished data (mostly in the form of unpublished doctoral dissertations or MA theses). This is quite an impressive percentage of unpublished data to have in a meta-analysis. I’m working on one where we’re thrilled we were able to get 11% of our data coming from unpublished studies. Not only that, but the way that researchers typically obtain unpublished data for meta-analyses is by e-mailing relevant listservs, faculty, universities, research centers, etc. and sending out a massive call for all unpublished studies. If you were someone who felt that your research had been suppressed by political bias, odds are fairly good that you would jump at the chance to send your data in to be part of a meta-analysis. Because of this, there’s generally an assumption that unpublished data is either more likely to have null results than published studies, or that it might actually be more likely to have controversial results (yes, it’s an issue that this might make it harder for something to get published — but that is not the point for today). So yes, 20% of the sample is pretty good representation for unpublished data. And yet, even when eyeballing the results of only the unpublished data, there’s still no evidence that maternal employment is a bad thing.
3. Social Norm Bias. Erickson mentions that researchers may want to find evidence that maternal employment is not damaging, as it “plays into a particular current notion that it’s OK.” In other words, Erickson is saying that researchers might be more likely to “find effects” that positively reflect current social norms, biases, or preferences. Extending this logic, then, you should naturally expect that the studies in this meta-analysis that were conducted in the 1960s might show a negative effect of maternal employment (as it was considerably less socially acceptable for mothers to work outside of the home in the 1960s), whereas those conducted in the 2000s would show a positive (or null) effect of maternal employment on children’s academic and behavioral outcomes.
However, once again — this is not the case. The researchers actually checked to see if the study effects differ based on when the studies were published, and they found absolutely no difference in effects based on publication year. So, studies published in the 1960s or 1980s show the same overall effects as those published in the 1990s or 2000s. Studies conducted from 2000-2009 were no more likely to find positive (or null) effects of maternal employment than those conducted in any earlier year.
Let’s take a closer look at one of the studies included in the meta-analysis from the 1960s, as an example. In a study from 1961, researchers Burchinal and Rossman sent surveys home with every single seventh- and eleventh-grade student throughout Cedar Rapids, Iowa, ending up with 1172 surveys from children and their parents. These surveys measured:
Let’s look at anxiety, one of the internalizing behavioral problems that allowed this study to qualify for inclusion in the meta-analysis, and how it correlates with Mom’s employment history. Among the 370 seventh grade boys? Correlations ranged from .01 to .08. Among the 283 eleventh grade boys? Correlations ranged from -.04 to .06. Among the 351 seventh grade girls? Correlations ranged from -.01 to .06. Among the 245 eleventh grade girls? Correlations ranged from -.05 to .03. Not a single one of those correlations was significant. Not even close. Not even marginal. Moms who worked outside of the home in the 1950s & 1960s when their children were very young were absolutely no more (or less) likely to have sons or daughters with anxiety issues than mothers who stayed at home. And sure, this is just one variable. But it seems like a pretty important one, especially since anxiety is theoretically the best candidate for a specific behavioral issue that you might expect to see in someone who struggled with maintaining high-quality parental attachment bonds throughout childhood.
We can do this for every single indicator that they measured. Obsessive, ruminating thoughts (a symptom of anxiety and/or depression)? No significant correlations. Excessive introspection? No significant correlations. Mood swings? No significant correlations. The only times when there are significant correlations, they aren’t particularly informative. Eleventh grade boys reported slightly higher rates of fatigue if their mothers worked outside of the home when they were younger, but only if this employment occurred when they were between the ages of 4 and 6. Seventh grade girls reported higher rates of head and eye complaints if their mothers worked while they were young, but eleventh grade girls didn’t show this same effect. There are five different employment indicators, four different groups of kids, and eleven dependent variables, yielding 220 correlations. Of those 220 correlations, 17 are significant and 203 are not. Given that you might reasonably expect about 11 false positives (correlations that come back as “significant” but truly aren’t) to pop out of these analyses anyway, and given that the few correlations that are significant are unpredictable, scattered, and not theoretically meaningful, this…seems like a fairly open and shut case. Maternal employment was just fine in the 1960s, too.
The study is not perfect. They only included White families and families with both biological parents living at home. But, it is a study that was done in 1961 — a year in which (as seen in the graph at left) only 3.5% of married mothers served as the primary breadwinners in their households, compared with 15% in the current day. Even in the context of a society in which social norms clearly supported the idea of a mother staying at home, research showed no negative effects of maternal employment.
4. Too Many Liberals In Science. Maybe Erickson is complaining that all studies, published or unpublished, will have a liberal bias, because all scientists are liberal. However, there are several issues with this accusation. First of all, in the studies being discussed here, they all simply measure “maternal employment” as an objective measure of how much mothers work, and then measure “child outcomes” by using commonly accepted metrics for achievement or behavioral issues, like IQ, grades, teacher evaluations, scales for depression, anxiety, or impulsivity, or teacher ratings of aggression. I’m not sure exactly how political bias could really alter their statistical analyses of these data or their choices of which metrics to use in this situation. If someone has a solid, theoretical argument that might support the idea that liberals are biased towards choosing these metrics and conservatives would clearly choose entirely different outcomes or scales that you would expect to show completely different effects, I’d be happy to hear it. Second of all, the studies span a wide range of socioeconomic and racial groups, so it doesn’t seem like results could be swayed by selective sampling.
And finally, if the complaint is just that conservatives don’t go into scientific research — I’m not sure that liberally-minded researchers can really do much to fix that problem. That one’s on conservatives to change.
Takeaway points here? Erickson might want to cry “bias” about these findings, but there doesn’t seem to be any support for that claim — at all. Meta-analyses are basically designed to “debias” entire research fields by aggregating effects across many, many different studies. The meta-analysis discussed in this post included a high percentage of unpublished data, so the analysis itself was not swayed by publication bias. The samples cover a wide range of racial and socioeconomic groups, and the metrics are as objective as they could possibly be and not clearly tied to any one political slant, limiting concerns about selective sampling or biased measures. And finally, we saw the same effects emerging in the (very different) 1960s as we see emerging in the 2000s, making it difficult to believe that researchers are only finding the effects that “society” wants them to find.
1. Maternal employment in each included paper had to be defined in terms of status as a categorical indicator (e.g., “employed,” “not employed,” “full-time employed,” or “part-time employed”) or had to provide the number of weekly work hours as a continuous indicator. This is because, for each analysis, studies were “grouped” together to compare effect sizes based on these categories. Therefore, studies were excluded if they measured “maternal employment” in ways that could not possibly have allowed the researchers to analyze these effects in this way, like characteristics of the workplace, maternal work behavior, etc.
2. Child outcomes in each included paper were limited to three specific achievement outcomes (children’s performance on formal tests of academic or intellectual development, school grades, or teacher ratings of cognitive/academic competence) and three specific behavioral outcomes (children’s externalizing problems, children’s internalizing problems, or overall behavior problems). This is, again, to make it as easy as possible to compare groups that contain as many studies as possible, to give you more power to see what the differences are. If you have too many studies where each one details a different type of outcome, that’s not as helpful. Studies were excluded if they used different kinds of achievement outcomes (like task motivation or parental perceptions of achievement other than grades) or different kinds of behavioral outcomes (like positive or prosocial behaviors).
APA Press Release: http://www.apa.org/news/press/releases/2010/10/working-mothers.aspx
Lucas-Thompson, R.G., Goldberg, W.A., & Prause, J. (2010). Maternal work early in the lives of children and its distal associations with achievement and behavior problems: A meta-analysis. Psychological Bulletin, 136 (6), 915-942 PMID: 20919797
Burchinal, L., & Rossman, J. (1961). Relations among Maternal Employment Indices and Developmental Characteristics of Children. Marriage and Family Living, 23 (4) DOI: 10.2307/347590
Image Credit: Fair and Balanced slogan via Wikipedia and available under free use, as the purpose of this post is “for identification and critical commentary on the station ID or program and its contents.”
12 Digital Issues + 4 Years of Archive Access just $19.99X