Bitch Ph.D nails the problem with a recently reported study on the relationship between day care and “disruptive behavior”: first, that “disruptive behavior” is (like “antisocial behavior” and a number of other synonymous terms) very much an eye-of-the-beholder category, but more importantly, that the effect size reported by the study is teeny-tiny.
I have seen this again and again and again with media effects research: extremely small effect sizes get reported as if they amount to tidal waves of social causality. Researchers love to blame this kind of misrepresentation on scientifically illiterate reporters, but it is as much the fault of the researchers as the reporters in many cases. Because no one wants to report, “We just found a very small contributing factor to a very complex, subtle and ambiguous social problem”. No, researchers usually say, “We just found the smoking gun that causes people to murder” in the press release and then protest apologetically when that’s the meme spread by the mainstream media.
Most of us tend to turn on this kind of skeptical parsing of research results only when the reported results offend against our own common sense or our political commitments. If the research supports our prior commitments, then we tend to act as carriers of the meme. So here’s the pledge I think we all should take. Do not endorse research about social behavior or social psychology without first looking very carefully at the methodology and the effect size. If you would disregard the study on those grounds when it contradicts your own social views, disregard it when it endorses your views.
Great post, Tim – we should view effects research as skeptically as the researchers regard the aspects of modern society they wish to blame… For a good take on this particular study, check out Judith Warner’s NYTimes blog post (if you’ve signed up for Times Select via the new free academic policy).
So say we all.
I personally love how the headline reads one way and the actual article admits the facts that this is an incredibly small percentage and that parental involvement and similar factors are still the MOST relevant in determining regular disruptions.
High-five to that. To step away from one’s ideological inclinations is the only way to separate the wheat from the chaff. And given some of the extreme wierdness that has become mainstream, it is as important as ever to actively discriminate against BS.
The spirit is willing, but the flesh is weak. A noble aspiration, but I suspect we will all continue to fail to follow it as well as we should.