[IPAC-List] Michael McDaniel's Reference to the so-called Validity-Diversity Dilemma

[Aamodt, Mike]

Pat,

Interesting question.  Based on some limited research done by our students, I would guess that there would be little difference.  I base this thought on two of their findings:

One study of 216 students taking PSY 121 exams showed a minimal difference (d = .06) in test taking times between whites and minorities.

A meta-analysis conducted in a summer research class showed that  across 13 studies (n = 14,557), the score difference between timed and untimed tests was minimal (d = -.10)

If these results can be generalized, which is a big IF, it seems that adding a time limit doesn't make a huge difference in test scores and there isn't much of a difference in how much time minorities and whites take to finish tests.  Of course, the length of the time limit itself and the number and types of items could make a difference.

Sounds like a great research project for a student looking for a SIOP presentation :)

_________________________________________

Michael G. Aamodt, Ph.D. (Mike)
Professor Emeritus
Department of Psychology
Radford University
Radford, VA  24142-6946
(202) 280-2172
maamodt at radford.edu<mailto:maamodt at radford.edu>


________________________________
From: IPAC-List [ipac-list-bounces at ipacweb.org] on behalf of Patrick McCoy [Patrick.McCoy at cfp-psc.gc.ca]
Sent: Friday, June 03, 2016 10:09 AM
To: List, IPAC; Putka, Dan
Subject: Re: [IPAC-List] Michael McDaniel's Reference to the so-called Validity-Diversity Dilemma

Interesting discussion!

A question, if I might. Has anyone seen convincing research that differences in group test performance are reduced when a test has minimal time press (i.e., when the time to do the test is generous). That assumption is often made, but is it warranted?

Pat McCoy
Ottawa, Canada

>>> Dan Putka <dputka at humrro.org> 2016/06/03 9:04 AM >>>
Hi Richard,

I think decisions regarding this matter will vary from organizations, and hinge on what they value and the resources they can offer to deal with potential inefficiencies in their selection process.

For example, let’s say an organization in question places high value on diversity.  The organization may be willing to sacrifice some level of criterion-related validity with their selection measure if it results in bringing a more diverse workforce in the door.  A critic may rightly argue that using a test with lower criterion-related validity would lower the expected mean performance of individuals hired.  While that would be true, the organization could potentially offset the loss of validity through more rigorous training and performance management (i.e., a post selection intervention designed to ensure a greater proportion of the workforce meet performance standards). Of course adding that rigor may require more investment on the part of the organization, which they may or may not see as worth it.

The other thing to consider here is the magnitude of the difference.  For example, using Mike’s quote as an example, if increasing the time limit of the test has only a small negative impact on validity, but makes a more sizable reduction in subgroup differences, then it would arguably be harder to justify not sacrificing validity in that case.  In that situation, one might construe the situation as the organization having a reasonable alternative to their assumed current approach (i.e., the reduced time limit) that has minimal impact on validity, but is better from a subgroup difference perspective. Of course, a challenge in practice is drawing the line between how much of a drop in validity is too much, and how much of a drop in subgroup differences is enough to warrant deviating from the “maximize validity” philosophy.  Though statistical inference can help here (e.g., non-sig. change in validity, sig. change in subgroup diffs), I think this is a judgment that will also vary from organization to organization.

Personally, I think one of our key roles as scientist-practitioners in the selection arena is to help the organizations we work with understand the implications of potential decisions they make regarding their selection processes.  We can do our best to explain potential options and likely outcomes based on data, theory, literature, past precedent, experience, and offer a reasoned, logical opinion, but the final decision may often not rest in our hands.

Dan

Dan J. Putka, Ph.D.
Principal Staff Scientist
Office: 703.706.5640
dputka at humrro.org | www.linkedin.com/in/dputka<redir.aspx?REF=2s2SKeUosblnevQ-FuNhxlr-wuCH_LhPlbSIg2N5W2ijEIi1wovTCAFodHRwOi8vd3d3LmxpbmtlZGluLmNvbS9pbi9kcHV0a2E.>

66 Canal Center Plaza, Suite 700
Alexandria, Virginia 22314-1578
www.humrro.org<UrlBlockedError.aspx>




From:        "Richard Joines" <mpscorp at value.net>
To:        "IPAC List" <ipac-list at ipacweb.org>
Date:        06/02/2016 08:15 PM
Subject:        [IPAC-List] Michael McDaniel's Reference to the so-called        Validity-Diversity Dilemma
Sent by:        "IPAC-List" <ipac-list-bounces at ipacweb.org>
________________________________



Mike,

You make the statement that "if job-related reading speed has undesirable consequences such as group differences, one may wish to sacrifice merit hiring for diversity hiring and increase the time limit of the exam."

I guess the question for those who think I/O Psychology is a science is... how does one reach the decision to throw the science out and go another route?  If the result is lowering validity, I'm certainly not about to increase the time limit of any of my empirically validated tests.  There would be no scientific basis for doing that.

I would be interested in what people think about this and how they view their role and what limitations they think they should observe, but my view has always been to try to maximize validity while ensuring compliance with federal guidelines.  Since the 1978 Uniform Guidelines we've been compelled to look for alternative selection methods, the idea being that if we can find or develop a test that has the same or higher validity but lower adverse impact, we should do that.

However, the idea that we should sacrifice validity in order to increase diversity strikes me as going too far.  Who are we to make such decisions?  We're supposed to be scientists, not social engineers, yes?

Thoughts anyone?

Rich Joines
Mgt & Personnel Systems, Inc.
www.mps-corp.com<redir.aspx?REF=IqPlzWo5Ju-A7cNn-hhKFpvIns_uxouNoQsTiCX66BYFcoq1wovTCAFodHRwOi8vd3d3Lm1wcy1jb3JwLmNvbS8.>
925-932-0203

 _______________________________________________________
IPAC-List
IPAC-List at ipacweb.org
https://pairlist9.pair.net/mailman/listinfo/ipac-list<redir.aspx?REF=0enhU3NQqgmASs73EOawA-8AD4aCzyDoGIdo_x_jZuQFcoq1wovTCAFodHRwczovL3BhaXJsaXN0OS5wYWlyLm5ldC9tYWlsbWFuL2xpc3RpbmZvL2lwYWMtbGlzdA..>





Ce courriel est destiné exclusivement au destinataire mentionné en titre et peut contenir de l'information privilégiée, confidentielle ou soustraite à la communication aux termes des lois applicables. Toute divulgation non autorisée, toute reproduction ou réacheminement est interdit. Si vous n'êtes pas le destinataire de ce courriel, ou n'êtes pas autorisé par le destinataire visé, ou encore, si vous l'avez reçu par erreur, veuillez le mentionner immédiatement à l'expéditeur et supprimer le courriel et les copies.

This e-mail message is intended for the named recipient(s) and may contain information that is privileged, confidential and/or exempt from disclosure under applicable law. Unauthorized disclosure, copying or re-transmission is prohibited. If you are not a named recipient or not authorized by the named recipient(s), or if you have received this e-mail in error, then please notify the sender immediately and delete the message and any copies.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist9.pair.net/pipermail/ipac-list/attachments/20160603/37c1694f/attachment.html>