Danger in Numbers

Writing and reading large numbers of letters for tenure and promotion dossiers is quite a chore. On that, I agree with Don M. Chance. On almost everything else he says about such letters in his essay for the Chronicle, we disagree.

Chance’s basic position as I read it goes like this: the sample size of outside letters in most evaluations is too small and too easily slanted consciously or unconsciously towards a favorable assessment of the candidate. Ergo, Chance believes that reading these letters is a mostly valueless sort of make-work, that at best all one can do is read them with “an entire 26-ounce box of Morton’s”, to assume that they’re inaccurate and should be read for “what is not said”. The system, he argues, has to be made rigorous and systematic, more like science, by upping the sample size to ten outside letters, at which point he feels it will be impossible to bias the sample.

As long as we’re talking rigor and science here, the notion that a larger sample size will magically change the nature of the population being sampled seems unrigorous to me as Chance argues it. Because Chance argues that the default assumption should be that letter-writers are dishonest (consciously or unconsciously) and that the default assumption should be that candidates should not be awarded tenure. Let’s just assume for a moment that Chance is correct (he’s not) that normally letters are exaggerated, self-serving or inaccurate. What’s the distribution of “normal” here in the total possible population? If this Diogenes of the dossier only finds his honest letter in 5% of the total letter-writing population, he might need more than six or even ten to appreciably improve the odds of getting even one. If letters are pervasively dishonest, if professionalism is a sham, focusing on sample size is a red herring.

But this thought concedes too much in any event, in several respects. What is Chance’s evidence that letters are largely promotional or dishonest? First, that the sample in any given promotion and tenure process is not random. It’s not clear to me what “random” in a field where there are perhaps twenty to forty qualified specialists familiar with the candidate’s work might look like. None of those people are likely to be strangers to the candidate: scholarly work incubated in social networks well before digital technology. Some may be rivals, some may be trusted collaborators, some relative strangers, but randomness won’t locate some icy, remote intellect who can appraise the candidate’s work without any previous knowledge of or feelings about the candidate. It’s also not clear to me that what Chance regards as the normal or common procedure in promotion processes is in fact normal or common–there are many institutions that try to make sure that there are writers who are not personally acquainted with or previously knowledgeable about the candidate to the degree that this is possible.

Second, Chance argues that the language of most letters is improbably positive–essentially that his colleagues can’t possibly be as good as the writers are suggesting that they are. There’s an echo here of a long-running conversation about the phenomenon of grade inflation, and as in that discussion, it is at least a possibility that the relentless pressure of competition for academic jobs, the intensification of academic professionalism, the creation of mentoring systems and so on means that yes, indeed, most of his colleagues are as good as the letters say they are.

There’s something deeper and a bit uglier lying underneath his assumption that the letters can’t be true, that it can’t be that most people deserve tenure. He uses a lot of language about journals that are B-level, institutions that are third-rate, and so on. Since he’s sharing his impressions, I’ll share one of my own. The people I’ve met (inside and outside of academia) who have the strongest, most uninhibited feelings that they are the top-rate, most elite people in their workplace and that everyone else is a third-rater are often not the most top-rate–and they are consequently the people who spend the most time trying to create systems and structures and procedures and bureaucracies that will affirm their separation from the common herd because they can receive that affirmation in no other way. A lot of the people I’d regard as truly exceptional talents just go about the business of being exceptional without worrying so much about building rules and procedures that will earn them a gold star and get everyone else wearing some dunce hats.

This in the end is the worst problem with the proposition that what we need to do is write more letters, make more metrics, indulge in a fetishistic mimicry of “science” in the evaluation of our peers. Not only would such a change dramatically amplify the uncompensated chore of writing assessments and evaluations (rules are like rabbits: require ten letters and I guarantee you in a decade, the same logic will grow the requirement to twelve, fifteen, twenty, twenty-five), it goes in entirely the wrong direction. Outside letters are already a form of outsourcing: they relieve faculty of the burden of having to become literate enough in the work of their colleagues that they can judge for themselves its value and integrity, and therefore simultaneously relieve us all of the obligation to communicate and disseminate our work in a manner that anticipates and welcomes those collegial readings.

If Chance wants to know whether to trust those adjectives, to believe in the professionalism of his profession, to know whether he’s the only person who invests time and effort in such letters or merely one in a legion, the first step is not to build a graven idol out of statistics, to believe that a bigger N is a magic path to truth and trust. The first step is a more humane gesture: to learn to read for oneself, as much as one can, the work of any colleague you’ve been asked to evaluate, and to build a culture that expects academic work to provide signposts for such readings. That’s work too, but it’s a more gratifying kind that doesn’t require putting our human sensibilities and intellectual abilities in a blind trust, yoked to a process that must be made as remote as possible from the ways we actually work and know one another.