Institutional Research Profession – Page 3

Catching our breath…

A lynx resting — photo by Tambako the Jaguar

It’s hard to believe that this semester is finally drawing to a close. The multitudes of followers to our blog may have noticed our sparse posts this spring… Shifting responsibilities, timing of projects, and just the general “stuff” of IR have left us little time to keep up.

Part of my own busy-ness has been due to an increased focus on assessment, as mentioned in an earlier post. This spring, the Associate Provost and I met with faculty members in each of our departments to talk about articulating goals and objectives for student learning. In spite of our being there to discuss what could rightly be perceived as another burden, these were wonderful meetings in which the participants inevitably ended up discussing their values as educators and their concerns for their students’ experiences at Swarthmore and beyond. In spite of the time it took to plan, attend, and follow-up on each of these meetings, it has been an inspiring few months.

Spring “reporting” is mostly finished. Our IPEDS and other external reports are filed, our Factbook is printed, and our guidebook surveys have been completed (although we are now awaiting the “assessment and verification” rounds for US News). Soon we will capture our “class file” – data reflecting this year’s graduates and their degrees, and that closes the year for freezing and most of the basic reporting of institutional data.

We also are fielding two major surveys this spring, our biennial Senior Survey (my project) and a survey of Parents (Alex’s project). Even though we are fortunate to work within a consortium that provides incredibly responsive technical support for survey administration, the projects still require a lot of preparation in the way of coordinating with others on campus, creating college-specific questions, preparing correspondence, creating population files, trouble-shooting, etc. The Senior Survey is closed, and I will soon begin to prepare feedback reports to others on campus. The Parents Survey is still live, and will keep Alex busy for quite some time.

As we turn to summer and the hope of having a quieter time in which to catch up, we anticipate focusing on our two projects that are faculty grant-funded. We don’t normally work on faculty projects – only when they are closely related to institutional research.

We are finishing our last year of work with the Hughes Medical Institute (HHMI) grant. IR supports the assessment of the peer mentor programs (focusing on the Biology and Mathematics and Statistics Departments) through analysis of institutional and program experience data, and surveys of student participants. We will be processing the final year’s surveys, and then I will be updating and finalizing a comprehensive report on these analyses that I prepared last summer.

Alex is IR’s point person for the multi-institutional Sloan-funded CUSTEMS project, which focuses on the success of underrepresented students in the sciences. Not only does he provide our own data for the project, but he will be working with the project leadership on “special studies,” conducting multi-institutional analyses beyond routine reporting to address special research needs.

I wonder if three months from now I’ll be writing… “It’s hard to believe this busy summer is finally ending!”

The Chronicle’s Recent Take on Data Mining in Higher Ed

Hello Hipsters — Photo by Andrew Coulter Enright

A recent article in The Chronicle of Higher Education titled, “A ‘Moneyball’ Approach to College” (or “Colleges Mine Data to Tailor Students’ Experience”) presents some ways that data mining is being used in higher ed. At the risk of sounding like someone overly zealous about enforcing the boundaries around obscure specializations, the article for the most part presents examples of mining instructional practice or the “Learning Analytics/Educational Data Mining” approach which is only a subset of the types of data mining or analytics being done in higher ed. Examples par excellence of this approach to higher ed data mining can be seen in The International Educational Data Mining Society’s journal and presented at their meetings.

Instead of building Amazon.com-style recommendation engines for courses or analyzing blackboard “clickstreams,” many institutional researchers have been engaged in data mining to deal with some of the perennial questions like yield, enrollment, retention, and graduation – for quite some time. For example, the professional journal New Directions for Institutional Research published an entire issue in November 2006 dedicated to data mining in enrollment management. One of the studies, conducted by Serge Herzog of University of Nevada, Reno, “Estimating Student Retention and Degree-Completion Time” found that data mining techniques such as decision trees and neural nets could be used to outperform tradition statistical inference techniques in predicting student success in certain circumstances.

The author of The Chronicle piece writes that “in education, college managers are doing something similar [to Moneyball] to forecast student success—in admissions, advising, teaching, and more”. This is true, but it has been going on for a long time and in many more ways than just learning analytics and course recommendation systems. I guess these institutional researchers who have always done data mining were Moneyball before it was cool. Does that make them hipsters?

Presently Presenting

In preparing to make a presentation at Swarthmore’s Staff Development Week next month, I thought it would be a good time to review some rules of thumb for making presentations that I’ve learned and discovered over the years. Because institutional researchers generally have such a range of people in our audiences, it can be tricky!

Power Point – This package is both a curse and a blessing. As a presenter I like having a visual reminder of key points, and to help frame for the audience where I’m going. As an audience member I know too well that slides full of text are deadly boring. Because I am a tactile learner, I have found that I like to organize my presentation by making a text-rich Power Point slide show, but then not actually showing much of it! The advantage is that I can share the full document later as the version that “includes speaker notes.” For the actual presentation, I try to use slides primarily for simple charts, illustrations, examples, and a minimal number of bullet points (with minimal associated text). I want people to engage with what I’m saying, not read ahead.

Tell what you’re going to tell – The importance of giving your audience a simple outline of the presentation was impressed on me by Alex. After a particularly boring talk we’d attended, he persuaded me how much better it could have been if we’d simply been able to follow its logic, which an outline would have provided.

Tables – Avoid all but the simplest tables of data in a presentation, and make sure, if you want them to be read, that they are indeed legible from the back of the room. If I am showing a table primarily to present a layout, I would make clear as soon as the slide is shown that it is not meant to be read. (This is not uncommon for Institutional Researchers, who share strategies and techniques – sometimes it’s not the data we want to see, but how you presented it!)

Graphs – I personally love a graph that contains a ton of information on one page. I could stare at it for hours, like someone else might stare at a painting and glean layers of meaning. Alas, I would try not to make such a graph for others! In general, the wider the audience, the simpler the chart should be. Avoid ratios, or even percentages that aren’t immediately grasped. And be sure to use colors. A simple, attractive chart that reveals an important relationship can reveal meaning to even the most staunchly anti-data.

Involvement – Whenever possible, I try to involve the audience either through humor (but DON’T overdo it – I’m an institutional researcher rather than a comedian for a very good reason) or engaging with an exercise or activity. At a faculty lunch presentation a number of years ago, before I began (during lunch), I left displayed a chart reflecting faculty opinions about their adequacy of sleep by career stage. It certainly piqued interest – people love to hear about themselves!

Certainly none of this is new, but I find it helpful to review and remind myself of them before starting a new project. As I look back through some of my past presentations I see that I haven’t always followed my own rules as well as I’d wish! But presentations can be a powerful tool for accomplishing a primary goal of Institutional Research: getting information to people who need it. And so it’s something I continue to try to learn about and work on.

“Freeze” time

It’s getting to be that time of year. I’m not talking about frost on the ground or midterms. It’s time to “freeze” the data!

Institutional Researchers report data about our students to many constituencies, and use it for our research. We must have data that is accurate, and is consistent across reports and research projects, and over time. When students are enrolling or dropping out at different points during the term, how do we keep track of it all? We don’t! Along with our Registrars, we select a single point in time early in the semester that best reflects our student body, and we essentially download a copy of relevant data about the population that is actively enrolled on that date. We call it a “snapshot” of the data, a “census,” or a “freeze.” This is what our institution looks like at that point, and we will use this data forever to reflect our students in this term. If someone drops out after the freeze date, our data will still reflect that student. If a student enrolled, but left before the freeze date, they will not be counted for general reporting or research purposes.

The date selected is typically far enough along after the start of the semester so that students have sorted themselves out. Many institutions use a particular number of days from the start of classes. The IPEDS* default language suggests October 15. Swarthmore has always used October 1 for our fall freeze of student data. (We have another date for freezing employee data.) That’s why, if you ask us for the number of students enrolled in September, we’ll ask you come back in October.

Leading up to the freeze, the Registrar’s office is busy tracking down students to make sure their status is accurate, and IR is checking with other offices (especially IT) to make sure programs are ready to run and new coding hasn’t been introduced which might affect the data extraction process. (We hate it when that happens – always give your IR shop a heads up about new codes!)

One of the interesting things about Swarthmore that is different from other institutions in which I’ve worked is that the default status for students who haven’t graduated assumes that they return each term. If they don’t return, their status must be switched to “Inactive” before the freeze date so that we don’t accidentally count them. In my other experiences, the default coding each term indicated that students were inactive, and their status must be switched to “Active” if they did return. It certainly makes sense to do it this way here, as most students continue until they graduate. It was just one of the many little things that charmed me when I first started working here.

*IPEDS stands for the Integrated Postsecondary Education Data System, the reporting system used by the National Center for Education Statistics (NCES) of the U.S. Department of Education. All institutions in the country that participate in any kind of Title IV funding programs (federal student financial aid) must participate in this reporting.

It’s the Number 1 time for Rankings – Part II

As promised in the first part of this post, here is a description of US News’ ranking procedure, for non-IR types.

US News sends out five surveys every year – three to the IR office of every college and university, one each to Presidents, Provosts, and Admissions Deans at every college, and one to High School guidance counselors. The surveys that go to H.S. Guidance Counselors and to college Presidents, etc. are very similar, and are called the “Reputation Survey” and the “Peer Assessment,” respectively. They list all of the institutions in a category (Swarthmore’s is National Liberal Arts Colleges), and ask the respondent to rate the quality of the undergraduate program at each institution on a one to five scale. There is an option for “don’t know.” Responses on these two surveys comprise the largest, and most controversial, component of the US News ranking, the “Academic Reputation” score. It’s the beauty contest.

The three surveys that are sent to IR office ask questions about 1) financial aid; 2) finances; and 3) everything else. This year, these three surveys included 713 questions. I wish that were a typo. We consult with other offices, crunch a lot of data, do an awful lot of checking and follow-up, and many, many hours and days later, submit our responses to US News. Then there are several rounds of checks and verifications, in which US News flags items that seem odd based on previous years’ responses, and we must tell them “oops – please use this instead,” or “yes, it is what I said it is.” Of those >700 items, US News uses about a dozen or two in their rankings, and the rest go into other publications and products – on which I’m sure they make oodles of money. Here are the measures that are used for ranking our category of institution, and the weights that are assigned to the measures in computing the final, single score, on which we are ranked:

Category and Weight in Total Score		Measurements and Weight in Category
22.5%	Academic Reputation
		67%	Avg Peer Rating on “Reputation Survey”
		33%	Avg H.S. Counselor Rating on Rep Survey

15%	Student Selectivity
		10%	Acceptance Rate
		40%	Percent in Top 10% of HS class
		50%	SAT / ACT

20%	Faculty Resources
		35%	Ranked Faculty, Avg Salary+Fringe (COLA)
		15%	% FT Faculty with PhD or Terminal Degree
		5%	Percent Faculty who are Full-time
		5%	Student/Faculty Ratio
		30%	Small Classes (% < 20)
		10%	Big Classes (% > 50)

20%	Graduation and Retention
		80%	6-yr Graduation Rate
		20%	Freshman Retention rate

10%	Financial Resources
		100%	Expenditures per Student

7.5%	Graduation Rate Performance
		100%	Actual rate minus Rate predicted by formula

5%	Alumni Giving Rate
		100%	# Alumni Giving / # Alumni of Record (Grads)

The percentages next to the individual “measurements” reflect the measure’s contribution to the category it belongs to. So for example, the student selectivity measure is affected least by acceptance rate (only accounts for 10% of the overall category score). The percentage next to the category reflects its weight in the overall final score. As I mentioned, the Academic Reputation score counts the most.

The way that US News comes up with a single scores is by first converting each measure to a z-score (remember your introductory statistics?), which is a standardized measure that reflects a score’s standing among all the scores in the distribution, expressed as a proportion of the standard deviation (z=(Score minus the Mean)/Standard Deviation). If an institution had a 6-year graduation rate that was one standard deviation above the average for all institutions, the z-score would be 1.0.

This transformation is VERY important. With z-scores at the heart of this, one cannot guess whether an improvement – or drop- in a particular measure might result in an improved ranking. It is our standing on each measure that matters. If our average SAT scores increased, but everyone else’s went up even more, our position in the distribution would actually drop.

So then they weight, combine, weight again, combine (convert to positive numbers somewhere in there, average a few years together somewhere else, an occasional log transformation, …), and out pops a final score, which is again rescaled to a maximum value of 100. (I always picture the Dr. Seuss star-belly sneetch machine.) One single number.

But there are a couple of other features of the method worth mentioning. One is that the average faculty compensation for each institution is weighted by a cost of living index, which US News doesn’t publish because it is proprietary (they purchased it from Runzheimer). It is also very outdated (2002). As Darren McGavin said when opening the leg lamp box in A Christmas Story, “Why, there could be anything in there!” Another unique feature is the “Graduation Rate Performance” measure, which compares our actual graduation rate with what US News predicts that it ought to be, given our expenditures, students’ SAT scores and high school class standing, and our percentage of students who are Pell grant recipients. Their prediction is based on a regression formula that they derive using the data submitted to them by all institutions. Did I mention the penalty for being a private institution? Yes, private institutions have higher graduation rates, so if you are a private institution, so should you.

Institutions are ranked within their category, based on that final single score, and with much fanfare the rankings are released.

It’s the Number 1 time for Rankings!

A number of admissions guide publishers have released rankings recently, and the Godzilla of them all, US News, will be coming out shortly. It’s always an interesting time for Institutional Researchers. We spend a lot of time between about November and June each year responding to thousands (I’m not kidding) of questions from these publishers, and then in late summer and early fall we get to see what amazing tricks they perform with this information, what other sources of “information” they find to spice up their product, and the many ways they slice and dice our institutions.

The time spent on their surveys is probably the most frustrating aspect of IR work. (Not all IR offices have this responsibility, but many do.) We are deeply committed to providing accurate information about the institution to those who need it. But so often guidebook questions are poorly constructed or not applicable, and the way they interpret and use the data can be bizarre. While publishers may truly believe that they are fulfilling a mission to serve the public by providing their synthesis of what admittedly is confusing data, there is no misunderstanding that selling products (guides, magazines) is their ultimate purpose. Meanwhile, we are painfully aware of the important work that we were not able to do on behalf of our institutions because of the time we spent responding to their surveys.

So the rankings come out, alumni ask questions, administrators debate the methodology and the merit, newspapers get something juicy to write about, and then we all go back and do it all over again. Some of my colleagues get really worked up about this, and I can understand that. But maybe I’m just getting too old to expend energy where it does no good. It seems to me like complaining about the weather. It is what it is. You do the best you can – carry an umbrella, get out your snow shovel, hibernate – and get on with life. Don’t get me wrong – I believe we should engage in criticism, conversation, and even collaboration if appropriate. I just don’t think we should get ulcers over it.

<Minor Rant>That said, I do think it’s especially shameful for publishers to lead prospective students to think that “measures” such as the salaries volunteered by a tiny fraction of alumni on PayScale.com will be useful in their search for a college that’s right for them.</Minor Rant>

I think we have to acknowledge that there has been some good from all this. There was a time when some institutions spun their numbers shamelessly (I know of one that reported the average SAT of those in the top quartile), and the increased scrutiny of rankings led to some embarrassment and some re-thinking about what is right. It also led to a collaborative effort, the Common Data Set, in which the higher education and the publishing communities agreed on a single methodology and definitions to request and report some of the most common data that admissions guidebooks present. In the past one guidebook would ask for average SAT, another for median, another for inter-quartile range, leave athletes out, put special admits in, and worst of all – no instructions about what was wanted. And then people wondered why there were six different numbers floating around. Unfortunately, once this set was agreed on and came into practice, guidebooks began to ask more and more questions to differentiate themselves from each other. (And some still don’t use it!) So it seems that a really good idea has backfired on us in a substantial way.

Another good to come from this is that some of the measures used by the rankings really are important, and having your institution’s data lined up against everyone else’s prompts us to ask ourselves hard questions when we aren’t where we’d like to be. Here at Swarthmore, even though we are fortunate to have excellent retention and graduation rates, we wondered why they were a few points behind some peers. Our efforts to understand these differences have led to some positive changes for our students. This is likely happening at many institutions. The evil side of that coin is when institutions make artificial changes to affect numbers rather than actually improving what they do.

On balance, I think that at this moment in time the guidebooks and rankings are doing more harm than good. The “filler” questions that use institutional resources (do prospective students really want to know the number of microform units in the library?), and the proliferation of rankings that underscore the truly commercial foundation of this whole enterprise (Newsweek/Kaplan’s “Horniest” – really??) have gotten me a bit worn this year.

But we’ll keep responding. And we’ll keep providing information on our website and through collaborative projects such as NAICU’s UCan (University and College Accountability Network) to try to ensure that accurate information is available. As a parent who will soon be looking at these guides from a different perspective, I will have new incentive to see some good in it all.

So in my best live and let live spirit, I will share the Reader’s Digest description of the Big One – the US News rankings- for my non-IR colleagues here at Swarthmore in Part II of this post. (IR friends, look away…)

On NOT reinventing the wheel

Stone wheel A couple of recent projects have reminded me of what a sharing profession Institutional Research is. We often share the results of our efforts when it will help others avoid needlessly repeating that effort. I’m not sure if it comes from the empathy that develops from working in small offices where resources are stretched so thin, or just the kind of people attracted to the field, but I have yet to meet a stingy IR person! (Although I have encountered plenty of people outside the field trying to make some money by selling us the stuff we’d otherwise “reinvent” ourselves…)

One of my earlier experiences with this kind of generosity was the data on faculty achievements collected by Carol Berthold, of the University System of Maryland. Carol would troll press releases and websites to maintain her database by institution of faculty members’ prestigious memberships (e.g. Institute of Medicine, National Academies, etc.) and awards (e.g. NSF New Faculty Awards, Guggenheims, etc.) by institution. And then she freely opened up her database to share with IR offices! This was data that we all found useful in touting our faculties’ accomplishments, providing contextual peer data, etc. Very cool!

Some of my wonderful colleagues distribute their SPSS syntax files for creating routine reports from the surveys in which a number of our institutions participate . Inspired by this, Alex and I are trying to make an effort to share some of our SAS syntax for these same surveys. (SPSS, SAS, and R are statistical analysis software. Probably the majority of IR offices use SPSS, but an increasing number use SAS, with use of R starting to pick up as well.)

Collecting and summarizing publicly available peer data is another area for collaboration and sharing. The data may be publicly available, but it can take some work to put it into a user-friendly format. A colleague recently shared a dataset he built of Fulbright Scholars. This effort was facilitated by staff at HEDS, and made available to HEDS members.

Having overlapping peer groups presents another opportunity to share. My good colleague at a nearby college has given me data that I needed from a peer summary that included Swarthmore. Another colleague at a peer institution would routinely share her fascinating anthropological/institutional research work on the CIRP survey using peer data that included Swarthmore.

Like many professional associations, ours offers “Tips and Tricks” from members through its newsletter and website. One of the things that Alex is doing with his blog is discussing some of the technical work we do, in an effort to encourage learning about tools and shortcuts from each other.

This kind of sharing provides the gifts of convenience, insights, and time. In the instances where we are doing or would benefit from similar projects, it just makes sense for us to spread the load.