I’ve been doing a decent amount of peer review work this summer. I was struck at how my own practices had changed over time.
First, I’m curious. How many people received some instruction about how to do peer review, any kind of advice at all, from advisors or teachers while you were in graduate school? It’s one of the first things we’re called upon to do as professionals when we have our first posts, but I get a sense that almost no one has any training in it. If you get trained, it’s almost by reading other people’s peer reviews of your own work.
As I think back, I don’t second-guess any of the peer reviews I’ve done of manuscripts. Mostly I’ve recommended for publication, in a few cases, I called for extensive revision. Usually I’ve tried to do fairly detailed commentaries on manuscripts, broken into two sections: a general overview of the book with an assessment of its contributions to scholarship, its appeal to potential audiences and so on and then specific comments on individual chapters where necessary. In some cases I haven’t bothered with the second part if the manuscript seems basically well-written, tightly structured, and so on.
But I have changed a good deal the way I approach journal-length pieces. When I started out, I wrote fairly extensive reviews that sometimes also verged on being (constructively) critical replies or incipient dialogues with the author(s) of the article. This now seems to me to be completely the wrong way to go about it. My only job in reviewing a journal-length article should be: does it add to the scholarly literature? Are its claims reasonable and well-based in evidence? Is it adequately written, and are its arguments clearly laid out? That’s it. If there are things I would do differently, or if it’s not really my cup of tea if I had my druthers, that’s not my business to say as a peer reviewer. That’s up to the journal editors. I finished a review of an essay in a game studies context that I thought was based on good research, had a clear argument, and contributed to the literature. I wasn’t sure it entirely matched my understanding of the journal’s specific interests, but that’s for them to decide. It isn’t the way I would approach the particular subject that the article authors were studying, but that’s not relevant.
I know that editors (both of journals and book presses) often include a document explaining what they’d like a peer reviewer to do, but I don’t really have a sense of whether that’s what most peer reviewers end up doing. Judging from some peer reviews I’ve received over the years, I’d say that in some cases, not very much. I’m sure most practicing academics have gotten at least one of those peer reviews that consists largely of a demand that the author cite way more secondary literature in order to prove his/her scholarly credentials, and as long as the author is at it, please cite the peer reviewer’s work far more extensively thanks very much.
When I’m not thinking that maybe we ought to just abolish peer review altogether in the humanities, I’m wondering about whether we can change it–or at least have a more transparent collective discussion about how it should be done.
It’s not necessarily one of the first things one is asked to do after Grad school. Usually the load of peer review increases as one’s standing in the field does. I think I started to review articles and book manuscripts two or three years after the PhD–but I was pretty visible already in my quite small subfield so I suspect others do not begin to do this as quickly as I was asked to. Now it constitutes quite a bit of my workload.
I received absolutely no guidance or mentoring on how to do this. I modelled my peer reviews originally on the most helpful reviews I had myself received. I try to place myself in the shoes of the author of the article or book. I also look at specific questions provided by the press or journal. I will point out gaping gaps in the bibliography. If I am listed on the editorial board, and have written an article on the exact same topic, then I would normally expect that they would cite ME just because they might think that the article would be sent to me. Once or twice I have suggested that someone cite me, although I cringe while doing so.
I receive feedback of variable quality on my own work. Usually a good review will point out problems that I myself grappled with in writing the article in the first place. Good editors of journals will send more articles to those whose peer-revewing skills are be best, and stop sending them to those who perform less well.
I’ve received some as a grad student in terms of reviewing conference proposals, but my diss director – and really, my area as a whole – is very focused on professional development. And, when I am working on the diss and items that have publication potential, I’ve received advise in terms of what a reviewer might look for. This, I think, is a good thing. . Of course, I’m in composition and rhetoric, so talking about writing is a natural part of our field.
My only “training” in peer review was serving as an editor for a graduate student-run journal – this allowed me to read many more reviews beyond my own work, and develop some internal “best practices” (and more so “worst practices”).
As for the larger question of reform, I’d call people’s attention to MediaCommons, which is trying to use the norms of online communication to transform peer review into “peer-to-peer review.” Kathleen Fitzpatrick’s essay outlines these ideas – and invites open & engaged peer feedback!
Peer review is one of the few aspects of academic life I still haven’t experienced from the other side of the desk, possibly because my own publication rate’s been a bit slow.
That said, my physicist brother was doing peer reviews as a grad student, which his advisor would (usually) review and then sign off on as his own. It was considered a component of his training: learning to critique and evaluate and respond to research. Perhaps the faster publication rate of the sciences might also explain it: too much work, so it gets pushed down the food chain.
As an editor of a scholarly journal (Canadian Journal of Irish Studies) I’m with Tim on the idea that peer review in the humanities should possibly be abandoned and should at least seriously rethought.
If you’re turning an article or book back because it really doesn’t engage with a well-developed set of arguments in a given field, then that’s a legitimate way to use your scholarly expertise to do a little gatekeeping. Far more often, though, you find reviewers who work on the assumption that certain interpretive (or political) positions are simply a matter of fact (or of “responsible scholarshipâ€) and thus need to be reflected in everything that gets published. This happens a lot in Irish studies, and happens a lot in Quebec studies too, where I also do a lot of work. The fact that these positions are fought over (sometimes rather viciously) in the discourse of both fields can mean that whether an article gets a positive report really can often depend on the luck of the draw as to where the peer-reviewer sits (revisionist? Essay needs to be more “professional.†Post-colonialist? Essay needs to be more “historicizedâ€). I try to take this into account when looking for reviewers for certain articles, but I always feel annoyed that I need to do this. There are peer reviewers in these fields who can set aside their personal positions on pre-1960 Quebec as a liberal society or Ireland and colonialism or whatever and make something approximate to a dispassionate judgement on an essay’s merits, but nothing I have seen suggests that they are anything but a minority.
That said, I’m not entirely sure what would take peer review’s place. Some gatekeeping does need to occur, no question there. But axe-grinding irresponsibility on the part of peer reviewers (an irresponsibility that is greatly encouraged by the sacrosanct quality of anonymity) is far, far too common.
Reviews that boil down to “this isn’t the article I would have written” *are* annoying.
On the other hand, babies and bathwater. The process can be valuable, especially when just starting out. I owe a tremendous amount to some of my reviewers. At its best, it’s a generous act, intellectually and otherwise: anonymous work in a reputation economy.
I think Jerry describes the bad kind of peer reviewing that I feel like I’ve seen a bit too much of in my career: the kind of people who call for “historicizing”, etc. as a sort of knee-jerk form of gatekeeping. Almost as if they feel their job as a peer reviewer is to force another cycle of revisions just because nothing’s good enough on its first pass.
Tim, I’ll chime in with a slightly different take on the question of peer review: that is, coordinating it as an editor. I work on a journal with a large editorial board, and true to all the warnings I received before taking on the job, it all boils down to “chasing paper.” The process of matching up submissions with readers, querying them on their availability & willingness to referee pieces for the journal, sending out essays, tracking progress, and collating replies demands the bulk of my time and energy.
While I’ve generally enjoyed good experiences with the EB, some members have been a chore to work with. Some repeatedly refuse to read manuscripts, which I can understand; we’re all busy. But the commitment they made was for 2-3 reviews a year, so I’m placed in a position of counting refusals and then recommending the recalcitrant reader be removed from the board. More problematic are the readers who take on an essay in which they clearly have no interest. I’ve received reports from EB members saying essentially, “I suppose this piece is OK; it’s not really my area of expertise,” or worse, leaving the responses blank except for checking off a “yes” or “no” for publication. I provide abstracts in advance, so the only explanation I can think of is a lack of attention on the reader’s part. Finally, some EB members seem uninterested in providing constructive comments for the author, which seems to me to be the whole point of peer review — pooling collective expertise to (A) improve the essay and (B) educate the writer.
I don’t mean this as a bunch of sour grapes, or to disparage the complex and overcommitted lives most academics lead. But I’ve been startled by the carelessness and lack of professionalism on the part of some highly-ranked scholars, and am learning to simply cut them out of the review process rather than engage in a tiresome process of wheedling and complaint.
I have to say I’ve been bad in this respect myself from time to time. Part of it is that peer review falls into a very large category of “service” activities that you eventually discover are not only not rewarded by the profession as a whole but are quickly forgotten by even the people for whom they are done. Like any gift economy, they function only because we hope to receive at some point what we are given. I was thinking a bit about this today as I prepare a promotion dossier: I’ve put a lot of my time into things that fall under the category of service (some, unlike peer review, I’ve done more efficiently or responsively) and realizing that I haven’t even bothered to keep track of most of it because it’s not labor that anyone else would ever care to know about, *even* in a dossier-review situation. But under those circumstances, the incentives to do peer review extensively are low. That, too, makes me think that maybe we could do without it in some circumstances. Take the average humanities journal: why couldn’t an editorial board of 10-15 members make all the judgements necessary to decide what’s worth publishing? Then at least there’s the reputational credit from being on the editorial board, no need to reach beyond it to peer reviewers who are forgotten the moment they submit their review.
So here’s the question – would the problems Bob & Tim describe be eliminated or at least lessened were peer-review public and non-anonymous? In many ways, academia is a reputation system – if your response quality & timeliness were visible to others, your reputation would suffer or thrive based on review work in ways that are mostly hidden today. I don’t think that reputation would have direct impact on tenure/promotion issues, but it could help junior scholars get their name out there in positive ways, more established scholars to demonstrate their editorial prowess that might lead to opportunities, and help the blowhards and/or slackers be outed for what they are (or motivate them to improve). For me, these benefits far outweigh the illusion of “rigor” that blind anonymity allegedly provides.
Actually, I think more people would avoid service unless there were some commandment to do it under those circumstances–what incentive would people have to agree to do peer review, for example, if there was a public record of having done it (or having failed to do it).
I’m just wondering how necessary it is in the humanities. There’s a very narrow class of articles where I am extremely well-qualified to fact-check the article in the narrowest but most demanding sense from expertise that I hold readily in my own memory. Articles on Zimbabwe, for example.
As someone who had a book project derailed at the last moment (we were about to go to contract) by an incompetent peer review, I think there does need to be some thoughtful attention given to the practice and the teaching of it.
It seems to me that the sorts of skills we employ when critiquing the work of students, and when writing book reviews, are part of the skill set needed to review the work of peers. That said, it seems odd to me now (as it did then) that there was no chance to discuss your work with the reviewer, no chance of appeal, etc.
It’s a rather weird system, if you think about it, given that most editors are at least conversant in the fields they oversee, if not active participants in the research. Why would they need to go to some unknown (a literal unknown, as far as the author is concerned – it was impossible for me to determine how seriously to take his or her critiques beyond my own gut sense and understanding of my topic, not knowing his or her credentials, background, familiarity with my topic, etc.)?
I can see using a peer reviewer to help with the nitty-gritty of a given topic – no one expects editors to know all the details of myriad subfields and topics – but to assess the work overall and its merits and marketability – isn’t that the job of an editor?
(I could also add, from my new perspective on writing and researching outside the academic system, that it’s damn odd how academia keeps expecting people to do good work without credit or renumeration beyond the approval of peers. Our work is valuable, and other institutions seem willing to treat it as such – why doesn’t academia?)
Unlearning Peer Review
I’ve spent a good portion of this summer managing the peer review process for a special issue of the Journal of Research Practice, focused on interdisciplinary education,
http://jrp.icaap.org/index.php/jrp/index
which will be coming out in November. It’s been quite an education, immensely time-consuming–and I’m not sure how profitable for individual reviewers, for the special issue, or for the journal as a whole. Seems to me the reviews that are worth anyone’s time and energy are precisely those Tim says he’s not writing any more/sees now as “completely the wrong way to go about it”: “fairly extensive reviews that sometimes also verged on being (constructively) critical replies or incipient dialogues with the author(s) of the article.” I actually can’t see my way clear to any OTHER way to go about it. Just giving grades–this is good enough/meets standards/doesn’t–seems to me not useful in the same way that grading in our classes http://serendip.brynmawr.edu/exchange/courses/csem/f07/information#eval
gets in the way of education, and exploration…..
Late to the discussion, but I’ve been thinking about this a lot lately since I’m now a co-editor of a journal, and the guy who sends things out for review. I think that much of what’s wrong with the system is one that transcends this iteration of the system — namely, for lack of a better term, bad people, or at least neurotic people.
I’m not sold on the small editorial board doing all the work, since that pretends expertise that simply won’t be there. When I get a submission on Program or Practice X from Country Y using theory Z, though our board (of 41) is superb, I often can’t rely on them being experts in X, Y, and Z, so am required to farm out, lest we end up reviewing solely on faith that everything’s right, and that all we’re looking for is how convincing the argument is in and of itself. (And factual errors have been common, in my experience, btw). Apart from anything else, too, a *good* review should be more than simple gate-keeping — it could help people develop ideas, and the help of another expert is sometimes important. At both the journal and book level, some reviewer comments have been instrumental in helping me greatly develop my thinking.
So the system itself is important. Where it breaks down is with crappy reviewers. And much as some degree of instruction would help, at the end of the day, nasty and petty people are in every field (ie: outside academic too), and hamper every field. What I haven’t worked out is how to correct for this problem as much as possible, beyond blacklisting reviewers who I thought were petty and nasty (which I do).
Jason’s hope in the prospects for open peer review are somewhat shared, but now the problem would be less one of nasty and petty people and more of neurosis and the fear of neurosis. I’ve had best friends read work of mine and apologize for being critical, worried that I’d blow: if that’s how friends who know me react, the prospects for Woody Allen-esque hand-wringing and ulcers aplenty abound with reviews from those one doesn’t know. I envision emoticom filled manuscripts 🙂 Or we then risk the problems of ass-kissing (would senior academics at the unis which everyone wishes they could one day transfer to EVER get rejected, let alone accepted with minor revisions?).
Maybe we need all editors to share their blacklists — though somehow I feel many editors would be ON those lists…