A couple of people interested in my potential course, The History of Failure and Error, noted that it’s important to ask, “failure from what”, to study the kinds of thought-systems that create an expectation of success.
It seems to me that an increasing number of people are also rethinking the way that failure is a necessary part of the production of knowledge, and asking where the good failures have disappeared to. Look at Kieran Healy’s discussion of a fascinating study at Crooked Timber (also discussed by Kevin Drum.)
The key finding: studies that have barely significant results get published at a rate far outstripping those that barely fall short of significant results. The consequence of this difference is potentially very substantial. As Healy and Drum observe, what it implies first and foremost is the widespread massaging of data so as to produce just-over-the-threshold-of-significant findings. Maybe you wouldn’t call it fraud, but cumulatively it amounts to something close to that. Especially since, if I can thump the pulpit on an issue that never fails to get my blood flowing, many scholars and experts tend to call for extraordinary policy interventions based on findings about extremely small, marginally significant effects. If the only thing that justifies such policy is the technical claim of “significance”, and significance is reached through subtle manipulations of data, then that kind of policy formation is even more scandalous than it already is.
But I’d go further: I think that negative findings about commonly inferred or hypothesized relationships ought to be far more valued than they are within academia. I grant you that opening that door too widely would lead to people studying whether the consumption of gorgonzola in Burkina Faso has any relationship to neutrino emission from protostars. You need to have a standard for what makes a good initial hypothesis. But we clearly need, and should reward, publications that appear in the Journal of Spurious Correlations and The Journal of Negative Results in Biomedicine. I think most disciplines could use a similar journal–or should publish such findings in general disciplinary journals.
Even the humanities could use something of the sort. There is, even in those fields, something of a bias against claims which are modest or subtle, which I take to be one of the reasons why so many historicist literary critics and others hitched their star so strongly to claims about the political importance or efficaciousness of their work, because it is hard to say, “Well, perhaps my interpretations of this text will have some small effect on the way people think about 19th Century English society and its impact on the present, but that’s hardly the point”. In the humanities, I don’t think you’d call it negative results, exactly. More like, “The Journal of Humble Arguments”? We could use something like that, I think.
I couldn’t agree more.
My first paper, published in 1988, was essentially a negative result. Of course we didn’t write it up like that, we wrote it up as “Such and such effect is important for parameter ranges X, Y, and Z.” The negative part was that those parameter ranges were out of the region of interest to the wider community, so we were essentially saying that the effect we were studying was unimportant for all practical purposes. (Of course, this could change in the future.)
Somewhat ironically, this paper is the most cited of my career, with 60 citations to date, the most recent in April 2006. It pretty consistently gets 2-3 citations every year, by a wide range of researchers. (I don’t work in that area anymore.) Almost all the citations are of the form “As was shown in [Ref] effect blah blah is unimportant for this study.” So it apparently was a useful result, even if it was a disappointing finding at the time.
Although revisionist revisionist history does sometimes get us back to traditional history. I actually rather like the process by which the “Storm Over the Gentry” ended up by saying that there was no detectable correlation between economic status and Puritanism/Parliamentarianism.
Are you, perhaps, impatient? It’s one thing to say that the negative results never appear, and another to say that there’s a lag time because they’re not sexy. If the latter, then we get closer to the truth eventually, and it’s more a cause for eye-rolling than teeth-grinding; a venial rather than a mortal sin.
In the humanities, I think yes, they eventually appear. In the sciences and hard social sciences? The problem is that a lot of policy and a lot of money is kicked off by barely-significant findings, and that in the fullness of time, a negative result on those same findings doesn’t do much to recuperate the lost money and the institutional consequences of barely-significant findings.
One of my current hobbyhorses is the need for better analytical and descriptive work on literary texts. Think of trying to do evolutionary biology on a descriptive knowledge base cobble together from medieval beastiaries and that’s what I think we’re doing in literature these days. The sort of work I think we need is both counter to current intuitions — and therefore odd and difficult, though it’s not rocket science by any means — but modest in its accomplishments. You aren’t going to save the world by describing the structure of one literary text after another.
So, what do I mean? Consider Shakespeare. All his plays are divided into 5 acts. But those divisions were added by later editors; they aren’t Shakespeare’s work. So lets drop all those divisions and take another look at those texts. I simply want to divide them into parts according to the flow of dramatic action. Scene divisions are given by entrances and exits of characters, so that’s a level of organization given in the text. I figure there’s a level or two between that and the whole text. Let’s figure it out.
Not rocket science. But when it’s done we might begin to learn something about how the plays work.
Yeah, I’m thinking about a more general thing lately that relates to this–about how we could reconceptualize the business of literary analysis in a more holistic fashion, to enfold a technical knowledge of cultural production with a social knowledge of cultural work with an interpretative knowledge of texts. I may post something on this soon.
bbenzon has a point, and reminds me that here’s a missing third leg of this stool. We gnerally agree there’s a need to discuss finished work with positive results; and as Tim points out we should share negative results because it induces at leasst as much understanding as the positive stuff. And then there’s work in the “Hey, look over here at this!” stage. Unbegun work.
Both of the former create stacks of the latter. We are all of us, independent of field and profession, overwhelmed with open questions. Any given one might be on the scale of an easy homework problem, or could grow to be a book or Ph.D. or career.
I’ve lately been asking my colleagues: So, how many of those open questions jotted in notebooks are you going to address before you’re dead? How many of us have time to do homework problems anymore, let alone the space in our careers for supernumerary Ph.D.-scale projects? We’re like parents, accumulating and preserving houses full of obsolete furniture, who expect their adult children to gleefully accept it and do something with it all when we’re downsizing or dead. They should value it all, because we’ve kept it so nice for them….
Though those children have their own very nice, more modern furniture already.
There’s not just a need for negative results. There’s a need to pass along unaddressed questions as well. A need to consider what we are going to do with them, and whether we want the credit if they’re addressed… or if they’re just questions we’d like to hear an answer to.
(And, apropos of the Shakespeare question: Sounds like a candidate for this conference.)
Yeah, I like that idea as well: a journal of conjectures, questions and ideas?
I think this is part of jumpstarting curiosity as a motivating force in academia, most especially in the humanities, which often seem to me to be so uncurious in their spirit.