Data Disconnect

Sadly, Thursday’s “Headcount” blog of the Chronicle reports on another institution misrepresenting data that is used in the US News rankings.   While there is plenty in the topic to be upset about, I found myself annoyed by this statement:

Nevertheless, replacing hard-and-fast numbers with mere estimates involves a conscious choice, and, it’s fair to assume, an intent to polish the truth.

Certainly there are situations when the intent is to “polish the truth,” and I have no idea whether this was the case at this institution, but I actually think it’s UNfair to assume the intent.   Continue reading Data Disconnect

Time for a prediction

crystal ball
photo by Cillian Storm

I don’t know, is it me?   I think it gets quieter and quieter each year after US News releases its rankings.   Has the publication that all of higher education loves to hate lost its impact?  I saw very little press yesterday, and not even much buzz on the IR listservs, in response to the release of US News’ annual rankings.   Maybe it’s all the bratty little upstart rankings that have begun to get more attention, or that we’ve just reached a point of rankings saturation and there’s nothing more to say.

I’m not big on making predictions.   In fact, whenever anyone asks me to predict what our rank will be, I make a lame joke about leaving my dice at home.   But US News depends heavily on these rankings in their business model, and I wonder if they’re missing the press they used to get.   What they need is some controversy!  I predict that it’s time for US News to “tweak” its methodology, which will result in some upsets in the rankings and presto!  More press!   They could even just update their Cost of Living Adjustment on the Faculty Salary measure – as far as I can tell, they’ve been using the same index since 2002.   That would certainly be defensible, and could have the effect of shaking things up.   But mark my word, SOMETHING will change next year!

Rank

Pond Scum
Pond Scum photo by Max F. Williams

Happiest Freshmen?!”  OK, time to get in on the action – lets start a new ranking!   First, we’ll need some data.  That’s an easy one – most institutions post their “Common Data Set” on line, and that’s a really great source.   It has data on admissions, retention, enrollments, degrees, race, gender, you name it.  This is what institutions send to publishers of other admissions guidebooks and rankings – why don’t we get in on the free data?  The top three places to find them on an institution’s website are probably the Undergraduate Admissions, Institutional Research, or About areas.

Or we can go to publicly available sources, such as the U.S. government’s National Center for Education Statistics (NCES), the National Science Foundation’s “WebCASPAR,” and others.   The advantage of that is that we can download data by institution en masse.   Also, no one can claim that the data misrepresents them – hey, they provided it to the agency, right?  So what if the data are a little outdated.  We’re not building a rocket, just a racket.

Or we could send each institution a questionnaire.  Not exactly sure what to ask for or how?  Don’t worry, those folks are experts, we’ll just send a general question and they’ll call other folks on their campus, hold meetings, and jump through all kinds of hoops to be helpful, and eventually send us something that we can then decide if we want to use.  The kids at Yale have been doing this for years with their “Insider’s Guide.”  Well, off and on for years (when they think of it).

Maybe we could start a web site, and ask people to come enter data about the institutions they attend, or attended in the past, and then use that information for each institution.  That’s what RateMyProfessor.com did, and they got covered by CBSMoneyWatch,  and others!   True, I spotted at least three Swarthmore instructors who have not been with us for some time among those ranked, and a few others I never heard of (with 175 regular faculty members, how could I possibly have heard of everyone) but that’s the beauty of it, right?  Low maintenance!  And PayScale.com has become a force to be reckoned with.  Sure, their “average income” data for Swarthmore only represents about 2% of the alumni (estimating generously), but nobody bothers to dig that deep.  It doesn’t stop well-known publications like Forbes from using it.

OK, so that’s where we can get data for our ranking, now what data should we use, and what shall we call it?   We can take a lesson from the Huffington Post story about the “Happiest Freshmen.”   Now that’s clever!  And I’ll bet it generated a ton of visits, because it sure got attention from a lot of people.  The only data used in that ranking was retention rates – brilliant!  One number, available anywhere, call it something catchy (or better yet, controversial) and let ‘er rip!  (Shhh..  as far as I can tell, it was the press that provided the label – the folks crunching the data didn’t even have to think of it!)

I propose that we pull zip codes from NCES, sort in descending order, and do a press release about the “Zippiest institutions ever!”  No that’s no good – if it’s not something that changes every year, how will we make money from new rankings?!    Any ideas?

It’s the Number 1 time for Rankings – Part II

As promised in the first part of this post, here is a description of US News’ ranking procedure, for non-IR types.

US News sends out five surveys every year – three to the IR office of every college and university, one each to Presidents, Provosts, and Admissions Deans at every college, and one to High School guidance counselors.   The surveys that go to H.S. Guidance Counselors and to college Presidents, etc. are very similar, and are called the “Reputation Survey” and the “Peer Assessment,”  respectively.  They list all of the institutions in a category (Swarthmore’s is National Liberal Arts Colleges), and ask the respondent to rate the quality of the undergraduate program at each institution on a one to five scale.  There is an option for “don’t know.”   Responses on these two surveys comprise the largest, and most controversial, component of the US News ranking, the “Academic Reputation” score.   It’s the beauty contest.

The three surveys that are sent to IR office ask questions about 1) financial aid; 2) finances; and 3) everything else.  This year, these three surveys included 713 questions.   I wish that were a typo.   We consult with other offices, crunch a lot of data, do an awful lot of checking and follow-up, and many, many hours and days later, submit our responses to US News.  Then there are several rounds of checks and verifications, in which US News flags items that seem odd based on previous years’ responses, and we must tell them “oops – please use this instead,” or “yes, it is what I said it is.”  Of those >700 items, US News uses about a dozen or two in their rankings, and the rest go into other publications and products – on which I’m sure they make oodles of money.   Here are the measures that are used for ranking our category of institution, and the weights that are assigned to the measures in computing the final, single score, on which we are ranked:

Category and Weight in Total Score Measurements and Weight in Category
22.5% Academic Reputation
67% Avg Peer Rating on “Reputation Survey”
33% Avg H.S. Counselor Rating on Rep Survey
15% Student Selectivity
10% Acceptance Rate
40% Percent in Top 10% of HS class
50% SAT / ACT
20% Faculty Resources
35% Ranked Faculty, Avg Salary+Fringe (COLA)
15% % FT Faculty with PhD or Terminal Degree
5% Percent Faculty who are Full-time
5% Student/Faculty Ratio
30% Small Classes (% < 20)
10% Big Classes (% > 50)
20% Graduation and Retention
80% 6-yr Graduation Rate
20% Freshman Retention rate
10% Financial Resources
100% Expenditures per Student
7.5% Graduation Rate Performance
100% Actual rate minus Rate predicted by formula
5% Alumni Giving Rate
100% # Alumni Giving / # Alumni of Record (Grads)

The percentages next to the individual “measurements” reflect the measure’s contribution to the category it belongs to.   So for example, the student selectivity measure is affected least by acceptance rate (only accounts for 10% of the overall category score).  The percentage next to the category reflects its weight in the overall final score.  As I mentioned, the Academic Reputation score counts the most.

The way that US News comes up with a single scores is by first converting each measure to a  z-score (remember your introductory statistics?), which is a standardized measure that reflects a score’s standing among all the scores in the distribution, expressed as a proportion of the standard deviation (z=(Score minus the  Mean)/Standard Deviation).  If an institution had a 6-year graduation rate that was one standard deviation above the average for all institutions, the z-score would be 1.0.

This transformation is VERY important.  With z-scores at the heart of this, one cannot guess whether an improvement – or drop- in a particular measure might result in an improved ranking. It is our standing on each measure that matters.   If our average SAT scores increased, but everyone else’s went up even more, our position in the distribution would actually drop.

So then they weight, combine, weight again, combine (convert to positive numbers somewhere in there, average a few years together somewhere else, an occasional log transformation, …),  and out pops a final score, which is again rescaled to a maximum value of 100.  (I always picture the Dr. Seuss star-belly sneetch machine.)  One single number.

But there are a couple of other features of the method worth mentioning.  One is that the average faculty compensation for each institution is weighted by a cost of living index, which US News doesn’t publish because it is proprietary (they purchased it from Runzheimer).  It is also very outdated (2002).  As Darren McGavin said when opening the leg lamp box in A Christmas Story, “Why, there could be anything in there!”  Another unique feature is the “Graduation Rate Performance” measure, which compares our actual graduation rate with what US News predicts that it ought to be, given our expenditures, students’ SAT scores and high school class standing, and our percentage of students who are Pell grant recipients.  Their prediction is based on a regression formula that they derive using the data submitted to them by all institutions.   Did I mention the penalty for being a private institution?  Yes, private institutions have higher graduation rates, so if you are a private institution, so should you.

Institutions are ranked within their category, based on that final single score, and with much fanfare the rankings are released.

It’s the Number 1 time for Rankings!

Runners Crossing Finish LineA number of admissions guide publishers have released rankings recently, and the Godzilla of them all, US News, will be coming out shortly.  It’s always an interesting time for Institutional Researchers.  We spend a lot of time between about November and June each year responding to thousands (I’m not kidding) of questions from these publishers, and then in late summer and early fall we get to see what amazing tricks they perform with this information, what other sources of “information” they find to spice up their product, and the many ways they slice and dice our institutions.

The time spent on their surveys is probably the most frustrating aspect of IR work.  (Not all IR offices have this responsibility, but many do.)  We are deeply committed to providing accurate information about the institution to those who need it.  But so often guidebook questions are poorly constructed or not applicable, and the way they interpret and use the data can be bizarre.  While publishers may truly believe that they are fulfilling a mission to serve the public by providing their synthesis of what admittedly is confusing data, there is no misunderstanding that selling products (guides, magazines) is their ultimate purpose.  Meanwhile, we are painfully aware of the important work that we were not able to do on behalf of our institutions because of the time we spent responding to their surveys.

So the rankings come out, alumni ask questions, administrators debate the methodology and the merit, newspapers get something juicy to write about, and then we all go back and do it all over again.   Some of my colleagues get really worked up about this, and I can understand that.  But maybe I’m just getting too old to expend energy where it does no good.   It seems to me like complaining about the weather.  It is what it is.  You do the best you can – carry an umbrella, get out your snow shovel, hibernate – and get on with life.  Don’t get me wrong – I believe we should engage in criticism, conversation, and even collaboration if appropriate.  I just don’t think we should get ulcers over it.

<Minor Rant>That said, I do think it’s especially shameful for publishers to lead prospective students to think that “measures” such as the salaries volunteered by a tiny fraction of alumni on PayScale.com will be useful in their search for a college that’s right for them.</Minor Rant>

I think we have to acknowledge that there has been some good from all this.  There was a time when some institutions spun their numbers shamelessly (I know of one that reported the average SAT of those in the top quartile), and the increased scrutiny of rankings led to some embarrassment and some re-thinking about what is right.  It also led to a collaborative effort, the Common Data Set, in which the higher education and the publishing communities agreed on a single methodology and definitions to request and report some of the most common data that admissions guidebooks present.  In the past one guidebook would ask for average SAT, another for median, another for inter-quartile range, leave athletes out, put special admits in, and worst of all – no instructions about what was wanted.  And then people wondered why there were six different numbers floating around.  Unfortunately, once this set was agreed on and came into practice, guidebooks began to ask more and more questions to differentiate themselves from each other.  (And some still don’t use it!)  So it seems that a really good idea has backfired on us in a substantial way.

Another good to come from this is that some of the measures used by the rankings really are important, and having your institution’s data lined up against everyone else’s prompts us to ask ourselves hard questions when we aren’t where we’d like to be.  Here at Swarthmore, even though we are fortunate to have excellent retention and graduation rates, we wondered why they were a few points behind some peers.  Our efforts to understand these differences have led to some positive changes for our students.  This is likely happening at many institutions.  The evil side of that coin is when institutions make artificial changes to affect numbers rather than actually improving what they do.

On balance, I think that at this moment in time the guidebooks and rankings are doing more harm than good.  The “filler” questions that use institutional resources (do prospective students really want to know the number of microform units in the library?), and the proliferation of rankings that underscore the truly commercial foundation of this whole enterprise (Newsweek/Kaplan’s “Horniest” – really??) have gotten me a bit worn this year.

But we’ll keep responding.  And we’ll keep providing information on our website and through collaborative projects such as NAICU’s UCan (University and College Accountability Network) to try to ensure that accurate information is available.  As a parent who will soon be looking at these guides from a different perspective, I will have new incentive to see some good in it all.

So in my best live and let live spirit, I will share the Reader’s Digest description of the Big One – the US News rankings-  for my non-IR  colleagues here at Swarthmore in Part II of this post.  (IR friends, look away…)