[IPAC-List] Differential prediction generalization: a new concept

Sun Oct 2 06:02:03 EDT 2016

Dear Joel and Winfred,

Thank you for noticing our recently published article in APA’s Journal of Educational Psychology, which we believe has important implications for testing practices.  Also, thanks to Winfred for bringing up an excellent point—we need to understand whether variability in differential prediction is real or artifactual and therefore this precise issue was a central part of our analyses. Results showed a lack of differential prediction generalization because variability remained after accounting for methodological and statistical artifacts including sample size, range restriction, proportion of students across ethnicity- and gender-based subgroups, subgroup mean differences on the predictors (i.e., HSGPA, SAT-Critical Reading, SAT-Math, and SAT-Writing), and SDs for the predictors. For those interested, the abstract is below and the article is available at http://hermanaguinis.com/pubs.html:

*       Aguinis, H., Culpepper, S.A., & Pierce, C.A. (2016). Differential prediction generalization in college admissions testing. Journal of Educational Psychology, 108, 1045-1059. Media coverage by:  <http://www.bizedmagazine.com/archives/2016/3/research/standardized-tests-dont-predict-student-success> BizEd,   <https://www.insidehighered.com/news/2016/01/26/new-research-suggests-sat-under-or-overpredicts-first-year-grades-hundreds-thousands> Inside Higher Ed,  <http://www.singaporenews.sg/science/professor-herman-aguinis-and-co-authors-believe-sats-are-unfair-and-bias/> Singapore News (Singapore),  <http://www.educationnews.org/higher-education/as-sat-revamps-researcher-says-results-may-not-be-useful/> Education News,  <http://www.dailymail.co.uk/sciencetech/article-3417880/The-SATS-BIASED-Researchers-new-evidence-tests-not-marked-way-country.html> Daily Mail (UK),  <http://www.educationdive.com/news/new-research-raises-issue-with-sat-predictability/412830/> Education Dive,  <http://www.onenewspage.co.uk/n/Science/759ghp38v/Professor-Herman-Aguinis-and-co-authors-believe-SATs.htm> One News Page (UK),  <http://www.sciencedaily.com/releases/2016/01/160125091404.htm> Science Daily,  <http://www.eurekalert.org/pub_releases/2016-01/iu-nru012516.php> EurekAlert!,  <http://www.idsnews.com/article/2016/01/professor-reaffirms-bias-in-standardized-testing> Indiana Daily Student,  <http://trendyissues.com/as-sat-revamps-researcher-says-results-may-not-be-useful/> Trendy Issues Global News, etc.

Abstract

We introduce the concept of differential prediction generalization in the context of college admissions testing. Specifically, we assess the extent to which predicted first-year college grade-point average (GPA) based on high-school grade point average (HSGPA) and SAT scores depends on a student’s ethnicity and gender and whether this difference varies across samples. We compared 257,336 female and 220,433 male students across 339 samples, 29,734 Black and 304,372 White students across 247 samples, and 35,681 Hispanic and 308,818 White students across 264 samples collected from 176 colleges and universities between the years 2006 and 2008. Overall, results show a lack of differential prediction generalization because variability remains after accounting for methodological and statistical artifacts including sample size, range restriction, proportion of students across ethnicity- and gender-based subgroups, subgroup mean differences on the predictors (i.e., HSGPA, SAT-Critical Reading, SAT-Math, and SAT-Writing), and standard deviations for the predictors. We offer an agenda for future research aimed at understanding several contextual reasons for a lack of differential prediction generalization based on ethnicity and gender. Results from such research will likely lead to a better understanding of the reasons for differential prediction and interventions aimed at reducing or eliminating it when it exists. 

We look forward to reactions/feedback/comments!

All the best,

--Herman.

Herman Aguinis, Ph.D.

Avram Tucker Distinguished Scholar and Professor of Management

George Washington University School of Business

2201 G Street, NW 

Washington, DC 20052

 <http://hermanaguinis.com/> http://hermanaguinis.com/ 

-----Original Message-----
From: IPAC-List [mailto:ipac-list-bounces at ipacweb.org] On Behalf Of Winfred Arthur, Jr.
Sent: Friday, September 30, 2016 12:14 PM
To: ipac-list at ipacweb.org; Joel Wiesen <jwiesen at appliedpersonnelresearch.com>
Subject: Re: [IPAC-List] Differential prediction generalization: a new concept

so Joel, a contrarian view  :)   of course the question is "in science are we trying to describe, explain, and predict phenomena at the level of individual cases (and by inference individual studies) or as they broadly exist in the population?"  if the former, then we shld give strong deference to case studies [and primary studies] and that shld be our primary methodology; if the latter, then we are going to have to use and rely on population parameters (based on aggregates) from which(some) individual cases [studies] by definition will vary.  so variation in the primary studies that are included in a meta-analysis is nothing new; indeed that is the very issue meta-analysis is trying to address . . . is there any variation, is it real or artifactual, and if it is real, what is the cause (moderators)?  so in a domain in which there is as much overprediction as underprediction, the question is, is this random or systematic error (variation)?  that is the question  :)

- winfred

On 9/30/2016 9:35 AM, Joel Wiesen wrote:

FYI-

A new Aguinis et al. study again shows that the absence of differential prediction seen in aggregational analyses of criterion related validity studies hides actual differential prediction in both directions (under prediction and over prediction) seen in individual studies. The authors summarize with this analogy: “The British writer and politician Benjamin Disraeli (1804–1881) stated ... 'A man eats a loaf of bread, and another man eats nothing; statistics is the science that tells us that each of these men ate half a loaf of bread.”

Aguinis, H., Culpepper, S. A. & Pierce, C. A. (2016) Differential prediction generalization in college admissions testing. Journal of Educational Psychology, 108, 1045-1059.

Joel P. Wiesen, Ph.D., Director

Applied Personnel Research

62 Candlewood Road

Scarsdale, NY 10583-6040

http://www.linkedin.com/in/joelwiesen 

(617) 244-8859

http://appliedpersonnelresearch.com 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist9.pair.net/pipermail/ipac-list/attachments/20161002/f631d228/attachment.html>