By Watson Scott Swail, President & Senior Research Scientist, Educational Policy Institute

Many colleges and universities continue to turn towards predictive modeling to complement their admissions and enrollment processes. A recent report from the Higher Education Quality Council of Ontario found that 36 percent of respondents from Canadian universities and colleges were using predictive modeling as a way to “improving student retention” and 40 percent said they were considering doing so. Respondents said they used the process to inform their strategies for retention, including support services and individual advising.

Predictive modeling, simply put, uses available student variables to determine, via some form of multiple regression statistical test, whether a student will be successful at an institution, typically measured by semester-to-semester and year-to-year retention, courses completed, course grades, and graduation rates.

And while institutions may use predictive modeling to help outline their retention practices, most use the practice as a part of their strategic enrollment management practice. That is, they use it to choose their freshman class, which is, in a major way, pre-determining some of their retention issues (or non issues) in the future by sorting people out in front rather than during. Of course, it sounds a lot cleaner than it is.

Predictive modeling can be a useful tool to find out which variables most predict the future progress of a student at a particular institution. On the positive side, a prudent institution could use this information to inform advisors and others on the needs of a student and how to help him or her. Of course, the dark, underbelly of predictive modeling is that it is often used as a gatekeeper for students.

The challenges to effective predictive modeling are based largely on the availability of reliable and valid data. After decades of study on predicting the behavior of students in higher education, the reality remains that only a few variables effectively predict the future. These typically include high school GPA, SAT/ACT scores, class rank, as well as demographic indicators such as SES, Race/Ethnicity, and gender.[1] There are other variables that come from a variety of assessments and inventories that measure what we call “non-cognitive” indicators.[2] Variables that may help describe learning preferences, study issues, self-concept, social safety nets, financial barriers, and others can help describe a student beyond the numbers. The problem here is that these indicators do not hold together, statistically, when compared to the academic variables previously described. Non-cogs are useful, but institutions have to be wary of how much they lean on these relatively weak indicators. I’ve always thought non-cognitive variables were excellent data for advising students, but beyond that, not altogether useful for admissions and retention personnel (an upcoming Swail Letter will talk about non-cognitive data at length). In the end, these data are interesting, but interesting does not always connote utility.

Thus, decades into the study of predictive modeling, we end up with about the same, simple data to predict the future. What we know is what we have known for some time: the combination of HSGPA and SAT/ACT is the best predictor of future academic success for students. I have consistently argued that income is the best predictor, but it isn’t one we really want to use for admissions processes far equity and fairness. When I worked at the College Board back in the 1990s, I looked at the intersect of income and SAT score. Plotted out on a graph, the two produce a perfect staircase, whereas every $10,000 increment in income correlated perfectly with a relative increase in SAT score: all the way from low-income to high income bands. You can walk that staircase of data. Thus, income tells us a lot about educational measures. If you’ve read my Swail Letters before, you know that I say income trumps (sorry) every other variable, because income influences everything in life, especially access to quality educational opportunities, such as good schools, teachers, and resources, let alone the critical “college knowledge” piece.

Exhibit 1. SAT scores by family income, 2014 (Source: Washington Post)[3]

Still, of all variables that can be banded together, beyond that of income and SES, we know that high school GPA and SAT/ACT are the highest predictors of freshman college GPA. A recent college board analysis of the predictive validity of the new SAT test found that the tests had an approximate correlation of about 0.50 as compared to a high school GPA correlation of 0.48.[4] (See Table 1 below). This is what we might call a moderate correlation. However, when HSGPA and SAT were used together, the correlation jumped to 0.58. That is the highest predictor I have seen, and it is why the College Board, ACT, and other organizations overtly advise that institutions should use multiple points of reference in the admissions process, not just their tests. While the whole is greater than the sum of the parts, sometimes we have to resort to the sum of the parts. In this case, it strengthens the predictive model.

Exhibit 1. College Board analysis of SAT predictive validity for first year GPA[5]

The challenge remains that even a good correlation of 0.50, let alone 0.58, only accounts for about a quarter of the relationship between a variable and the outcomes (the infamous r² value)[6]. That is, three-quarters of the dependent variable—in this case freshman GPA—is unaccounted and explained by “other” variables.

If these are our strongest predictors, how much stronger are the various predictive models being pedaled in the marketplace? And what worth do they have for our admissions and retention officers? Given that I have yet to see models that provide higher predicative values, we must be careful in interpreting what the findings mean and how we use them to inform practice.

Statistical analysis is a creaky business based not only on the rigor of data but on the judicious interpretation of statistical output. Great data and poor interpretation can be a dangerous thing; poor data with poor analysis is even more troublesome. As we say in the numbers business: garbage in, garbage out.

But what if we don’t know what is garbage and what is meaningful?

That, my friends, is the crunch. In the black boxes of predictive models in the marketplace, all bandied about as the solution to our institutional woes, most clients have little ability to judge the difference between good and bad. It is simply the output they are given. And because they have paid handsomely for it, it must be good.

This is a cautionary tale. Predictive modelling can be an important tool in the toolbox of institutional professionals in determining their best freshman class while also providing important data to inform retention programming. But it is only good when made specific to the institution with appropriate dependent variables in play (e.g., first-year GPA; fall-to-fall retention; DFW rates; graduation). When the institution takes the time to fully understand the black box, it is useful. However, when the same tool is used as a gate, then we start the slippery slope. What we know is this: whatever information we produce to tell us what students will succeed will ultimately tell us who will not.

So… how do you use your predictive models?

 

[1] https://www.uccs.edu/Documents/retention/2003%20Student%20Variables%20that%20Predict%20Retention.pdf.

[2] http://williamsedlacek.info/publications/articles/employing1.html.

[3] https://www.washingtonpost.com/news/wonk/wp/2014/03/05/these-four-charts-show-how-the-sat-favors-the-rich-educated-families/?utm_term=.0e3aa3d93823.

[4] https://collegereadiness.collegeboard.org/pdf/redesigned-sat-pilot-predictive-validity-study-first-look.pdf.

[5] https://collegereadiness.collegeboard.org/pdf/redesigned-sat-pilot-predictive-validity-study-first-look.pdf.

[6] Cliff Adelman would be proud that I threw this in.