{"id":105,"date":"2011-09-08T16:43:12","date_gmt":"2011-09-08T16:43:12","guid":{"rendered":"https:\/\/blogs.swarthmore.edu\/wordpress-camp\/ir-sandbox\/?p=105"},"modified":"2023-09-13T15:41:16","modified_gmt":"2023-09-13T15:41:16","slug":"its-the-number-1-time-for-rankings-part-ii","status":"publish","type":"post","link":"https:\/\/blogs.swarthmore.edu\/institutional-research\/?p=105","title":{"rendered":"It’s the Number 1 time for Rankings – Part II"},"content":{"rendered":"

As promised in the first part of this post<\/a>, here is a description of US News’ ranking procedure, for non-IR types.<\/p>\n

US News sends out five surveys every year – three to the IR office of every college and university, one each to Presidents, Provosts, and Admissions Deans at every college, and one to High School guidance counselors.\u00a0\u00a0 The surveys that go to H.S. Guidance Counselors and to college Presidents, etc. are very similar, and are called the “Reputation Survey” and the “Peer Assessment,”\u00a0 respectively.\u00a0 They list all of the institutions in a category (Swarthmore’s is National Liberal Arts Colleges), and ask the respondent to rate the quality of the undergraduate program at each institution on a one to five scale.\u00a0 There is an option for “don’t know.”\u00a0\u00a0 Responses on these two surveys comprise the largest, and most controversial, component of the US News ranking, the “Academic Reputation” score.\u00a0\u00a0 It’s the beauty contest.<\/p>\n

The three surveys that are sent to IR office ask questions about 1) financial aid; 2) finances; and 3) everything else.\u00a0 This year, these three surveys included 713 questions.\u00a0\u00a0 I wish that were a typo.\u00a0\u00a0 We consult with other offices, crunch a lot of data, do an awful lot of checking and follow-up, and many, many hours and days later, submit our responses to US News.\u00a0 Then there are several rounds of checks and verifications, in which US News flags items that seem odd based on previous years’ responses, and we must tell them “oops – please use this instead,” or “yes, it is what I said it is.”\u00a0 Of those >700 items, US News uses about a dozen or two in their rankings, and the rest go into other publications and products – on which I’m sure they make oodles of money.\u00a0\u00a0 Here are the measures that are used for ranking our category of institution, and the weights that are assigned to the measures in computing the final, single score, on which we are ranked:<\/p>\n

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Category and Weight in Total Score<\/strong><\/td>\nMeasurements and Weight in Category<\/strong><\/td>\n<\/tr>\n
22.5%<\/td>\nAcademic Reputation<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
67%<\/td>\nAvg Peer Rating on \u201cReputation Survey\u201d<\/td>\n<\/tr>\n
33%<\/td>\nAvg H.S. Counselor Rating on Rep Survey<\/td>\n<\/tr>\n
<\/td>\n<\/td>\n<\/tr>\n
15%<\/td>\nStudent Selectivity<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
10%<\/td>\nAcceptance Rate<\/td>\n<\/tr>\n
40%<\/td>\nPercent in Top 10% of HS class<\/td>\n<\/tr>\n
50%<\/td>\nSAT \/ ACT<\/td>\n<\/tr>\n
<\/td>\n<\/td>\n<\/tr>\n
20%<\/td>\nFaculty Resources<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
35%<\/td>\nRanked Faculty, Avg Salary+Fringe (COLA)<\/td>\n<\/tr>\n
15%<\/td>\n% FT Faculty with PhD or Terminal Degree<\/td>\n<\/tr>\n
5%<\/td>\nPercent Faculty who are Full-time<\/td>\n<\/tr>\n
5%<\/td>\nStudent\/Faculty Ratio<\/td>\n<\/tr>\n
30%<\/td>\nSmall Classes (% < 20)<\/td>\n<\/tr>\n
10%<\/td>\nBig Classes (% > 50)<\/td>\n<\/tr>\n
<\/td>\n<\/td>\n<\/tr>\n
20%<\/td>\nGraduation and Retention<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
80%<\/td>\n6-yr Graduation Rate<\/td>\n<\/tr>\n
20%<\/td>\nFreshman Retention rate<\/td>\n<\/tr>\n
<\/td>\n<\/td>\n<\/tr>\n
10%<\/td>\nFinancial Resources<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
100%<\/td>\nExpenditures per Student<\/td>\n<\/tr>\n
<\/td>\n<\/td>\n<\/tr>\n
7.5%<\/td>\nGraduation Rate Performance<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
100%<\/td>\nActual rate minus Rate predicted by formula<\/td>\n<\/tr>\n
<\/td>\n<\/td>\n<\/tr>\n
5%<\/td>\nAlumni Giving Rate<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
100%<\/td>\n# Alumni Giving \/ # Alumni of Record (Grads)<\/td>\n<\/tr>\n
<\/td>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n

The percentages next to the individual “measurements” reflect the measure’s contribution to the category it belongs to.\u00a0\u00a0 So for example, the student selectivity measure is affected least by acceptance rate (only accounts for 10% of the overall category score).\u00a0 The percentage next to the category reflects its weight in the overall final score.\u00a0 As I mentioned, the Academic Reputation score counts the most.<\/p>\n

The way that US News comes up with a single scores is by first converting each measure to a\u00a0 z-score (remember your introductory statistics?), which is a standardized measure that reflects a score’s standing among all the scores in the distribution, expressed as a proportion of the standard deviation (z=(Score minus the\u00a0 Mean)\/Standard Deviation).\u00a0 If an institution had a 6-year graduation rate that was one standard deviation above the average for all institutions, the z-score would be 1.0.<\/p>\n

This transformation is VERY important.\u00a0 With z-scores at the heart of this, one cannot guess whether an improvement – or drop- in a particular measure might result in an improved ranking. It is our standing<\/em> on each measure that matters.\u00a0\u00a0 If our average SAT scores increased, but everyone else’s went up even more, our position in the distribution would actually drop.<\/p>\n

So then they weight, combine, weight again, combine (convert to positive numbers somewhere in there, average a few years together somewhere else, an occasional log transformation, …),\u00a0 and out pops a final score, which is again rescaled to a maximum value of 100.\u00a0 (I always picture the Dr. Seuss star-belly sneetch machine.)\u00a0 One single number.<\/p>\n

But there are a couple of other features of the method worth mentioning.\u00a0 One is that the average faculty compensation for each institution is weighted by a cost of living index, which US News doesn’t publish because it is proprietary (they purchased it from Runzheimer).\u00a0 It is also very outdated (2002).\u00a0 As Darren McGavin said when opening the leg lamp box in A Christmas Story, “Why, there could be anything in there!”\u00a0 Another unique feature is the “Graduation Rate Performance” measure, which compares our actual graduation rate with what US News predicts that it ought to be, given our expenditures, students’ SAT scores and high school class standing, and our percentage of students who are Pell grant recipients.\u00a0 Their prediction is based on a regression formula that they derive using the data submitted to them by all institutions.\u00a0\u00a0 Did I mention the penalty for being a private institution?\u00a0 Yes, private institutions have higher graduation rates, so if you are a private institution, so should you.<\/p>\n

Institutions are ranked within their category, based on that final single score, and with much fanfare the rankings are released.<\/p>\n","protected":false},"excerpt":{"rendered":"

As promised in the first part of this post, here is a description of US News’ ranking procedure, for non-IR types. US News sends out five surveys every year – three to the IR office of every college and university, one each to Presidents, Provosts, and Admissions Deans at every college, and one to High … Continue reading It’s the Number 1 time for Rankings – Part II<\/span><\/a><\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4,5],"tags":[35],"_links":{"self":[{"href":"https:\/\/blogs.swarthmore.edu\/institutional-research\/index.php?rest_route=\/wp\/v2\/posts\/105"}],"collection":[{"href":"https:\/\/blogs.swarthmore.edu\/institutional-research\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.swarthmore.edu\/institutional-research\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.swarthmore.edu\/institutional-research\/index.php?rest_route=\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.swarthmore.edu\/institutional-research\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=105"}],"version-history":[{"count":2,"href":"https:\/\/blogs.swarthmore.edu\/institutional-research\/index.php?rest_route=\/wp\/v2\/posts\/105\/revisions"}],"predecessor-version":[{"id":1584,"href":"https:\/\/blogs.swarthmore.edu\/institutional-research\/index.php?rest_route=\/wp\/v2\/posts\/105\/revisions\/1584"}],"wp:attachment":[{"href":"https:\/\/blogs.swarthmore.edu\/institutional-research\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=105"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.swarthmore.edu\/institutional-research\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=105"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.swarthmore.edu\/institutional-research\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=105"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}