Easily Distracted

I don’t think it’s a secret that I am very frustrated with prevailing trends in higher education assessment. I feel bad that this frustration often forces me to be a major annoyance to great local colleagues in the faculty and administration who have responsibilities for ensuring that Swarthmore keeps its commitments and conforms more closely to those prevailing trends.

I recognize that faculty at many institutions are sometimes overly defensive about assessment of any kind. All of us should be constantly re-evaluating what’s working and not working about our teaching. Good re-evaluations shouldn’t just be private and introspective, because it’s a bit too easy to convince yourself that everything’s fine and you’ve done enough. It’s also important that we create some kind of transcript or data or visible record that the entire world can critically examine. Our students and their families, as well as our publics in general, are owed that.

We shouldn’t be too sensitive about assessment. And we shouldn’t be against it simply because it’s more work, though it’s not unreasonable to actually subtract or remove some other part of the labor of teaching a course to compensate for producing assessment data. If it’s important, then it’s worth doing as something other than a freebie add-on to existing work.

I’m not against assessment in general. I’m against assessment as a diversionary tactic for government agencies trying to keep people from looking too closely at the failures of government. I’m against assessment as an unaccountable practice imposed upon professionals, a practice that actively contradicts what those professionals know about their own working conditions and practices and that cuts corners by using cookie-cutter bureaucratic procedures that treat all teaching institutions as if they’re doing the same thing under the same conditions. I’m against assessment when it trespasses against what my colleagues Barry Schwartz and Kenneth Sharpe describe as forms of professional and experiential “practical wisdom”.

I’m against assessment when it’s measuring the wrong things in the wrong ways. I’m against it when it’s about providing one organization the product they need in order to give another organization what it needs so that the third organization can please a fourth organization, all up and down the food chain. If that’s how meritocracies ensure their version of a full employment program, I’d just as soon have giant, clumsy, inflexible socialist bureaucracies instead, because at least more people get paid off a little bit that way.

In a recent discussion, one of my colleagues wearily suggested that we just render unto Caesar what is Caesar’s, do whatever our accreditors want so that they go away and let us get back to doing good work. In response one of my other colleagues said, “As long as you’re doing what Caesar wants, why not make it useful for you too?” My typically confusing attempt to play the metaphor further in response was this: “Convincing yourself what Caesar wants is good for you too is pretty bad if you’re a barbarian beyond the Roman frontier.” What Caesar wants in this sense is a “civilizing process”. If you have another way of doing things that you feel is better for you, for your culture, for your world, then making Caesar’s way your own is the beginning of the end.

Especially when what Caesar wants isn’t even good for Caesar.

And that’s where How College Works kicks in. Daniel Chambliss and Christopher Takacs have some explicit things to say about assessment. What they say explicitly is characteristically polite, measured, and backed up by detailed research. The most direct commentary comes in Chapter Eight, “Lessons Learned”. They first argue that some of the worst wastes of energy and resources at colleges and universities involve futile attempts to “microengineer human behavior” in strategic plans and other kinds of initiatives (here echoing Schwartz and Sharpe) and too much pursuit of “pedagogical innovation” (ok, that one leaves a bit of a mark on me personally). But they then proceed to note that after eleven years of close study of all of the major styles of educational assessment, they “came away skeptical of the entire assessment enterprise”.

Why?

1) Because assessment regularly works with the wrong units of analysis. Courses, teachers, programs and departments are the wrong units. Individual students are the right unit.

2) Because what you need to assess or understand is how students “experience your institution”. They add “Don’t assume that you know what matters.”

3) “Be open to all outcomes”. E.g., that specifying a set of learning outcomes on a syllabus and then measuring the learning outcomes just completely misses the point when it comes to understanding what is and is not working with education.

4) Because assessment practices create far more data–and far more work in chasing the data–than they need. Because assessment practices end up interfering with the work faculty are already doing to no good end. (I’ll add something to that: and because people trying to enforce assessment practices often don’t believe faculty when they say so.)

But I think there’s more said in the book that applies to assessment. Chambliss and Takacs argue throughout the book that a course or a semester or even several years of a matriculant’s experiences are not the right time frame for understanding what works and doesn’t work about assessment. That at the end of a semester, for example, students often don’t really know yet what they’ve gotten from a particular course.

The authors observe that the vagueness of many liberal arts programs about how students derive the benefits they derive from that education is empirically warranted. Meaning, that trying to break down each element of that education into measurable, atomistic units, via rubrics and standards and lists, and then tinker one-by-one with those atomized elements is missing the forest for the trees. It turns out, if you accept their research, that students get better at writing and speaking and thinking and understanding via the simultaneous, synergistic interaction between all of those activities, both in courses and outside of them. That they learn by watching others, by observing models (especially professors), by experimenting with their scholarly and personal personas in a safe environment. That efficacy in educating involves trying to nurture and support the richness and complexity of a purposeful, focused life.

Basically, I come away from How College Works thinking that the upshot of their argument, resting on their empirically-driven, carefully-designed research is basically what Geoffrey Rush’s character says repeatedly in Shakespeare in Love: that theater is naturally beset by “insurmountable obstacles on the road to disaster” but that in the end all turns out well. Why? he is asked (at first by a hostile investor who reminds me very much of an accreditor from Middle States). “I don’t know”, he says, “It’s a mystery.”

My dream is that some day accreditors and federal bureaucrats and parents and publics will learn to take that insight seriously. It’s not obfuscation or defensiveness. It’s the truth. Not a mystery beyond understanding, but a mystery in that the coming together of an education is about a great many things working together simultaneously, none of which are properly understood or measured or changed when they’re treated in isolation from one another. It’s about process and flow, not product.

One of the most extreme extreme cases of an unfair division in attention to two different books dealing with the same subject is the difference between William Deresiewicz’ Excellent Sheep and Daniel Chambliss and Christopher Takacs’ How College Works.

The one thing you can say about the intense attention to Deresiewicz’ book is that it goes a long way towards verifying some of his argument. E.g., a critique of the most elite institutions for producing coddled, soulless meritocrats touches a lot of influential people in a sensitive spot, and thus clears the way for lots of column inches in the mainstream media as well as lots of social media chatter.

But Chambliss and Takacs have done a huge amount of sensitive, interesting, qualitative work on one hundred students at Hamilton College over nearly a decade. They don’t denounce anyone or run around talking about the falling skies. But neither do they just wave everyone away and say that all is well. If you were looking for an understanding of how higher education works but also an explanation of what isn’t working, or what could work better, this book is where you should start. It’s where the national conversation should be, not the least because the findings in the book should discomfort or surprise most of the stakeholders in higher education.

The book suggests that some of what faculty fervently worry about and spend time discussing around curricular planning and pedagogy is at least a waste of their time. It suggests that administrators who focus on enrollment levels may have much more of a legitimate philosophical point about both equity and quality of education than many faculty would like to credit. It politely but insistently documents that most of the assessment practices being pushed on colleges right now either measure the wrong units or are just plain empirically wrong from start to finish. Perhaps least sensationally but most importantly of all, the book suggests that most of what colleges do, they do well, even when they don’t themselves understand what it is that they’re doing right.

The following people should treat it as a must-read book:

Prospective college applicants and their parents.
First-year college students during their orientation.
Faculty, especially at small liberal-arts colleges.
Higher education administrators, especially at small liberal-arts colleges.
Consultants, policy-makers, foundation executives, accreditation executives or any other professional whose work is focused on higher education.

Let me pull out what I found to be the most startling or interesting findings in the book, some of which affirmed some views I have developed in the course of my career and others of which challenged or reinterpreted some of those views.

1) The biggest impact a professor can have on a student is through direct, personal conversation with that student in which that professor is perceived by the student to accurately understand that student’s capabilities, aspirations, motivations and needs. This conversation can be challenging, it can be critical or intense, it can be friendly or kind. The basic thing, it seems from their research, is that a student needs to feel that a faculty member really understands them personally. Just once! That’s what is so interesting–a single such experience seems to make a big difference in satisfaction and in educational outcomes for many students. The book mentions that there’s something of a power law involved as well–that a small handful of faculty account for a large proportion of these strongly positive experiences.

2) A professor can have almost as big an impact in such a conversation in which they grossly misunderstand or mischaracterize a student’s character, interests or aspirations. Except that in this case the impact can be intensely negative–this kind of interaction is sufficient to offset much of the rest of an educational experience. So there are risks here–a faculty member who has poor emotional intelligence, who relies on stereotypes, or who is just presumptuous can all by themselves do a tremendous amount of damage to what the institution overall is trying to do.

Chambliss and Takacs argue that the effects of these interactions are sufficiently pronounced that almost everything else that faculty incessantly fret over in teaching and curricular design are comparatively unimportant in their effects on students. I think they’re right that we pay almost no conscious attention to this kind of interaction. We just assume that it will happen as an outcome of small size and intense focus on academics. The closest we get to deliberate institutional attention at Swarthmore is with programs like the Rubin Scholars that aim to connect individual students with faculty. I suppose you could argue that faculty advising as a whole is intended to promote the good kind of these conversations but it is mostly a more modest part of the overall logistics of the curriculum.

2a) Bad or indifferent teaching in introductory courses not only has a negative impact on the perception of that discipline among students, it has a disproportionately negative impact on the overall educational experience of the students exposed to such a course. It’s not such a big deal in other kinds of courses (you get the sense that their research indicates that in every student’s life, a little bit of weak teaching is inevitable) as long as it doesn’t lead to the sort of negative personal discussion described earlier.

This makes me worry a lot about whether Swarthmore and institutions like it are systematically attentive to this issue. I suspect not.

3) “Most learning happens outside the classroom” turns out to be unambiguously true in a wide variety of ways. Not only does that finding underscore the suggestion of a few faculty visionaries that “courses” might not be the best default structure for higher education, it makes the contempt that faculty like Benjamin Ginsberg show towards “deanlets”, residential life administrators, and the idea of “learning outside the classroom” look very short-sighted. What’s particularly notable about How College Works in this respect is that the researchers found that the single most important predictor of whether a student will have good educational outcomes is whether they made friends in their first six months. If Chambliss and Takacs are right, this is an aspect of the educational experience of students that faculty have virtually no influence over or interest in other than incidentally providing one possible site for friendships to develop (e.g., courses).

4) Small classes are perhaps over-exalted by faculty. They may produce better educational outcomes on average (if nothing else, giving a faculty member a much higher probability of understanding individual students well enough to have one of those powerful moments of direct connection) but they are by nature inequitable. This for me was one of the oh-my-god-of-course-why-didn’t-I-see-that-before moments in the book. Chambliss and Takacs found to their surprise that the students they were speaking with often hadn’t had a small class experience even at a small college, or that they’d had very few. And then it dawned on them: small classes are small, e.g., very few students are in them. If you have three large classes with 50 students in them and ten small classes with 5 students in them (presuming for the moment they’re not the same students in both), then the three classes have 150 students total and the ten classes have only 50 total. Only 25% of that total population is having a “small class experience”. Chambliss and Takacs gently suggest that this observation (and some similar findings elsewhere in the book) mean that at least some of the way that administration (or faculty managers) try to be mindful of enrollments and resource distribution is necessary–and that an excessive prioritization of small class experiences can produce a hidden elitism. Their discussion of this is more subtle than this summary suggests, but even at this relatively simplified level it was an uncomfortable challenge to some of my own conventional wisdom.

There are a lot of other findings and observations of great interest–say, for example, about the impact of off-campus study, about the reasons why seniors are often disengaged from their studies, etc., but these points alone struck me as having immediate, powerful implications for how faculty (and others) think about achieving better educational outcomes.

I also think the book politely but insistently undercuts contemporary fads in assessment. That’s important enough that I’m going to devote a second post to it shortly.

A campus Title IX coordinator recently made some interesting comments to Elizabeth Nolan Brown at the Dish about the problem of sexual assault on college campuses.

A lot of the national debate, such as it is, about what’s gone wrong in higher education and how to fix it, has centered on the question of whether colleges and universities should ever have judicial-style proceedings that involve accusations about serious crimes. Critics suggest that reports of sexual assault should go straight to the police and be resolved through the criminal justice system, that college administrations should be 100% uninvolved.

This is a silly argument as such, because the critics are starting from a misperception of (or sometimes a conscious, active distortion of) the overall context of campus judiciary proceedings. When reform advocates answer these critics in their own terms, the discussion sometimes tends to get trapped inside this misunderstanding. College policies need to be understood first and foremost in the context of college codes of conduct, which is what the Title IX coordinator points out at The Dish.

Both private and public universities admit students according to selective criteria and all of them reserve the right to revoke that admission to a matriculated student who violates their declared rules and procedures.

The most obvious or classic reason to expel or suspend a current student is poor academic performance. While almost no contemporary campuses will expel a student for a single semester of poor performance, a college or university could choose to adopt such a policy. An overly draconian (or lenient) policy on academic performance would likely bring heavy pressure for change from an accrediting agency, and a punitive approach would also very likely scare away prospective students. But you’re at least technically allowed to do it if you want.

Colleges and universities can also adopt very strict social policies if they like, and even strong civil libertarians and defenders of academic freedom tend to agree that a private institution is permitted to do so if it so chooses, though not to violate Constitutional rights or other statutory protections. So a strongly religious college, for example, can have a code of conduct that mandates expulsion or suspension for any consumption of alcohol, for any premarital sex, for smoking, for rejection of congregational doctrine, you name it. If a student matriculates there, they are accepting that code of conduct and the consequences of breaking it.

As the administrator quoted in The Dish points out, even secular institutions with no desire to regulate most private conduct by students in those respects routinely handle a wide variety of infractions of their own codes of conduct. They deal with plagiarism, fistfights, arrests for public drunkenness, vandalism of college property, drug dealing, theft, cheating, and so on.

One of the baseline principles in most colleges and universities is that a student who is a serious danger to the security and safety of fellow students, administrators and faculty or to the property of the institution and its community may be suspended or expelled. The most common second baseline is strictures against students who commit academic dishonesty such as plagiarism or cheating.

Academic dishonesty, for the most part, doesn’t involve criminal behavior, or even serious exposure to potential civil liability. And yet, campuses typically examine it using some kind of judicial proceeding. Higher education also has to deal with questions of conduct that might be criminal offenses in addition , and in those cases colleges and universities cannot afford to defer to and wait upon the criminal justice system. Because their concern in those instances is not with justice in that sense. It is with the safety, health and welfare of their current students, about maintaining conditions that allow those students to focus on their education while they are matriculated students. If you’re against judicial proceedings on sexual assault, are you against them on plagiarism? Ok–but surely then you are not against the notion that either of these infractions might be legitimately as violations of a code of conduct. You just believe that they should be handled differently.

If, for example, mobile devices and laptop within a dorm are being stolen regularly, that’s a threat to the welfare of the students in that dorm. If all the stolen devices are found in a dorm room of an enrolled student, the administration has to consider that student’s status whether or not they or the owners of the stolen devices have any interest in a criminal charge. If they deferred to the courts and took no action until that time, it could be months or years before the case was resolved–and in the meantime, the probable thief is still living in the dorms, where it is difficult or impossible to thoroughly monitor or supervise their actions. The college needs some way to decide what to do–and probably therefore some way to be sure that the student in question is very likely to be the person who stole the devices.

The debate should never be about whether colleges should make such decisions, or subordinate all their decisions to the criminal justice system.

I already pointed out that plagiarism and cheating couldn’t be dealt with if that were the case. There are many other situations that might fall into such a gap. You could, for example, have a student who persistently and deliberately commits extraordinary safety violations in laboratory courses. That’s likely not a criminal offense at all. The idea that a college should just stand by helplessly because there is no outside agency to consider the offense is crazy. So it needs a process, and that process has to include consideration of possible sanctions. If the student in question is a natural science major, then even if the intervention doesn’t involve something like suspension, it might involve compelling the student to change their major, which that student might likely see as a punishment.

How students see these kinds of actions is important, and is a big reason why most institutions today have some form of quasi-judicial process for dealing with some kinds of infractions of conduct codes. Not all of them: I suspect all colleges and universities have quietly suspended or expelled students in cases where they’d just as soon not talk about the infraction openly. Quasi-judicial processes on most campuses carry over historical traditions about honor codes and so on, that students should participate in judging their own and that students should be able to witness the consequences of infractions for exemplary reasons.

Higher education institutions could at their discretion rely on a purely executive decision: a faculty member could report cheating, a dean could then automatically expel. But most decision-making today in higher education is consultative and organizational hierarchies are relatively flat and decentralized, which work against that kind of action. If you’re going to have a consultative meeting between a dean and a faculty member about an accusation, why not throw in a couple more people, including the accused? The relative autonomy of tenure-track faculty, though fading in current institutions, could permit different faculty members to make accusations according to their own standards. The hearing is a chance to tie the decision to a consistent institutional policy or approach. Yet another reason is that most colleges and universities properly treat expulsion or suspension as serious penalties within their own context, and so seek procedures that are both substantively and performatively responsive to that sense of gravity. It’s not just that an expelled student is losing the value they’ve invested in their education. Even before all the current talk of “return on investment”, higher education was driven by a sense of mutual obligation between teachers and students, a belief that every generation was carrying forward important traditions.

A college or university can and should decide on standards that ensure that its students can get the education that they’re seeking. In the case of sexual assault, it can decide that any sexual behavior that could be judged to be predatory, aggressive or violating consent makes it impossible for other students to pursue their education in a safe, secure fashion. Particularly if the college or university in question believes, as most do, that at least some of the education they offer takes place outside the classroom, in the life of the community. In the pursuit of that safety, an institution could legitimately decide to decisively move students to other residences, change their course schedules, or suspend or expel the accused student. Or it could decide that it needs a thorough investigation of any such charges and a complicated, multilayered deliberative process. Or it could settle for a muddled, contradictory approach.

Under pressure from many activists, many colleges and universities are coming to the conclusion that they currently lack the expertise to assess such charges or to cope with their consequences, hence the changes unfolding at many institutions. If you question those changes, you have to come inside the frame of policies intended to deal with conduct violations, not stand outside and advocate relying on the criminal justice system as an alternative.

You can talk about what’s fair and unfair, efficient and inefficient, clear or unclear in how colleges and universities handle students who violate codes of conduct. What you can’t say is that those codes of conduct should somehow defer to criminal and civil proceedings, that colleges and universities should not have codes of conduct. They shouldn’t wait around for years to decide what to do about a student who poses a serious risk to others, or who has abused or disrupted the educational process. The only deference should be to the statutes and regulations (like Title IX) which directly address the obligations of institutions of higher learning.