Danger in Numbers

Writing and reading large numbers of letters for tenure and promotion dossiers is quite a chore. On that, I agree with Don M. Chance. On almost everything else he says about such letters in his essay for the Chronicle, we disagree.

Chance’s basic position as I read it goes like this: the sample size of outside letters in most evaluations is too small and too easily slanted consciously or unconsciously towards a favorable assessment of the candidate. Ergo, Chance believes that reading these letters is a mostly valueless sort of make-work, that at best all one can do is read them with “an entire 26-ounce box of Morton’s”, to assume that they’re inaccurate and should be read for “what is not said”. The system, he argues, has to be made rigorous and systematic, more like science, by upping the sample size to ten outside letters, at which point he feels it will be impossible to bias the sample.

As long as we’re talking rigor and science here, the notion that a larger sample size will magically change the nature of the population being sampled seems unrigorous to me as Chance argues it. Because Chance argues that the default assumption should be that letter-writers are dishonest (consciously or unconsciously) and that the default assumption should be that candidates should not be awarded tenure. Let’s just assume for a moment that Chance is correct (he’s not) that normally letters are exaggerated, self-serving or inaccurate. What’s the distribution of “normal” here in the total possible population? If this Diogenes of the dossier only finds his honest letter in 5% of the total letter-writing population, he might need more than six or even ten to appreciably improve the odds of getting even one. If letters are pervasively dishonest, if professionalism is a sham, focusing on sample size is a red herring.

But this thought concedes too much in any event, in several respects. What is Chance’s evidence that letters are largely promotional or dishonest? First, that the sample in any given promotion and tenure process is not random. It’s not clear to me what “random” in a field where there are perhaps twenty to forty qualified specialists familiar with the candidate’s work might look like. None of those people are likely to be strangers to the candidate: scholarly work incubated in social networks well before digital technology. Some may be rivals, some may be trusted collaborators, some relative strangers, but randomness won’t locate some icy, remote intellect who can appraise the candidate’s work without any previous knowledge of or feelings about the candidate. It’s also not clear to me that what Chance regards as the normal or common procedure in promotion processes is in fact normal or common–there are many institutions that try to make sure that there are writers who are not personally acquainted with or previously knowledgeable about the candidate to the degree that this is possible.

Second, Chance argues that the language of most letters is improbably positive–essentially that his colleagues can’t possibly be as good as the writers are suggesting that they are. There’s an echo here of a long-running conversation about the phenomenon of grade inflation, and as in that discussion, it is at least a possibility that the relentless pressure of competition for academic jobs, the intensification of academic professionalism, the creation of mentoring systems and so on means that yes, indeed, most of his colleagues are as good as the letters say they are.

There’s something deeper and a bit uglier lying underneath his assumption that the letters can’t be true, that it can’t be that most people deserve tenure. He uses a lot of language about journals that are B-level, institutions that are third-rate, and so on. Since he’s sharing his impressions, I’ll share one of my own. The people I’ve met (inside and outside of academia) who have the strongest, most uninhibited feelings that they are the top-rate, most elite people in their workplace and that everyone else is a third-rater are often not the most top-rate–and they are consequently the people who spend the most time trying to create systems and structures and procedures and bureaucracies that will affirm their separation from the common herd because they can receive that affirmation in no other way. A lot of the people I’d regard as truly exceptional talents just go about the business of being exceptional without worrying so much about building rules and procedures that will earn them a gold star and get everyone else wearing some dunce hats.

This in the end is the worst problem with the proposition that what we need to do is write more letters, make more metrics, indulge in a fetishistic mimicry of “science” in the evaluation of our peers. Not only would such a change dramatically amplify the uncompensated chore of writing assessments and evaluations (rules are like rabbits: require ten letters and I guarantee you in a decade, the same logic will grow the requirement to twelve, fifteen, twenty, twenty-five), it goes in entirely the wrong direction. Outside letters are already a form of outsourcing: they relieve faculty of the burden of having to become literate enough in the work of their colleagues that they can judge for themselves its value and integrity, and therefore simultaneously relieve us all of the obligation to communicate and disseminate our work in a manner that anticipates and welcomes those collegial readings.

If Chance wants to know whether to trust those adjectives, to believe in the professionalism of his profession, to know whether he’s the only person who invests time and effort in such letters or merely one in a legion, the first step is not to build a graven idol out of statistics, to believe that a bigger N is a magic path to truth and trust. The first step is a more humane gesture: to learn to read for oneself, as much as one can, the work of any colleague you’ve been asked to evaluate, and to build a culture that expects academic work to provide signposts for such readings. That’s work too, but it’s a more gratifying kind that doesn’t require putting our human sensibilities and intellectual abilities in a blind trust, yoked to a process that must be made as remote as possible from the ways we actually work and know one another.

This entry was posted in Academia. Bookmark the permalink.

4 Responses to Danger in Numbers

  1. Doug says:

    Just so.

    For a few years after college, I worked in a bookstore that did a very large number of author events; at the time (I was told) we hosted the most of any bookstore in America. The people who were at the top of their respective fields — Jimmy Carter, Ginger Rogers, Carl Sagan and Anne Rice, to name four — were all delightful to work with and for. They were personable and great with the public, too. Sagan, in particular, stands out in my memory. In private, before the event, he was entranced by a math book I had set out to catch his eye. In public, he showed great grace as a truly astonishing number of people said, “You changed my life for the better.”

    The problematic ones were in the middle: somewhat well known, with decent but not stellar work. Not everyone in that range was a pain to deal with, but the converse was nearly universal; if someone was difficult, he or she was not, as you say, top-rate.

  2. BPM says:

    I would almost guarantee you that Chance’s tenure file would never hold up to the scrutiny he’s applying to candidates who come before him.

    This is in many ways one of the most disappointing aspects of academic life: many of the old timers in the field have decided to take on the mantle of gatekeepers. I am very lucky to have great mentors at my institution, but I have many friends who do not. They have senior colleagues who have no idea how much more competitive fellowships and publications are. They have no idea what adjunct life is like. They still think that everyone’s spouse is at home taking care of children. And many have very thin publication records of their own, yet apply insanely rigorous and skeptical standards to cases that come before them.

    As you rightly point out, it wouldn’t matter if Chance got 5 or 10 or 15 letters so long as the only ones he believed were the negative ones. What a sad commentary on the failure of senior faculty to recognize their common cause with junior colleagues.

    These guys are looking down from the roof and pushing the ladder off the side of the building.

  3. PQuincy says:

    I am reminded of an outside for a senior promotion I read some years ago. The author opined firmly that only books published by university presses deserved serious consideration in this process, and that under the circumstances, he was disappointed that there were not several in the candidate’s file.

    Needless to say: the letter-writer, a person who had been promoted to full professor at a respectable institution, had published but one book with a university press, and that a version of the dissertation, and in the distant past.

    As for the scandalous bias involved in having a candidate for tenure’s original PhD mentor write about the file: I have heard from very senior and experienced colleagues at my university that the absence of such a letter in a tenure file (for which we do tend to gather about 8 letters) would be a serious lacuna. After all, if the PhD mentor is not willing to write, and not willing to explain to everyone how far the candidate has come since completing his or her dissertation, that could be a warning sign. And of course: review committees up the ladder are fully aware that a PhD mentor’s letter must be read with appropriate doses of salt — as must every letter.

    The result, it is true, requires reading attentively (and exactly for what is not there as well as for what is there), and requires experience in the fine art of making distinctions. But then again, is that not what academics do: make distinctions, unwrap seemingly subtle differences into a clearer and more comprehensive picture?

    In fact, it is Dr. Chance’s metaphor of the scientific method that seems utterly misplaced in discussing the decision whether or not to tenure a junior faculty member. Does he truly believe that what is involved is “stating a testable hypothesis, collecting data, analyzing those data, and drawing a conclusion with the admission that we could be wrong”??

    What a bizarre way of thinking about one’s colleagues’ lives!

Comments are closed.