[IPAC-List] Michael McDaniel's Reference to the so-called Validity-Diversity Dilemma

[Winfred Arthur, Jr.]

we tend to use very general time limits too on our knowledge and similar 
tests.  but it has *not* been under the premise or expectation that it 
will reduce subgroup differences but instead (besides the usual 
administrative constraints) there has been little reason to justify a 
time pressure.  hence, Mike's results are aligned w/ what i would expect.

- winfred

On 6/3/2016 10:32 AM, Aamodt, Mike wrote:
> Pat,
>
> Interesting question.  Based on some limited research done by our 
> students, I would guess that there would be little difference.  I base 
> this thought on two of their findings:
>
> One study of 216 students taking PSY 121 exams showed a minimal 
> difference (d = .06) in test taking times between whites and minorities.
>
> A meta-analysis conducted in a summer research class showed that 
>  across 13 studies (n = 14,557), the score difference between timed 
> and untimed tests was minimal (d = -.10)
>
> If these results can be generalized, which is a big IF, it seems that 
> adding a time limit doesn't make a huge difference in test scores and 
> there isn't much of a difference in how much time minorities and 
> whites take to finish tests.  Of course, the length of the time limit 
> itself and the number and types of items could make a difference.
>
> Sounds like a great research project for a student looking for a SIOP 
> presentation :)
>
> _________________________________________
> Michael G. Aamodt, Ph.D. (Mike)
> Professor Emeritus
> Department of Psychology
> Radford University
> Radford, VA  24142-6946
> (202) 280-2172
> maamodt at radford.edu <mailto:maamodt at radford.edu>
> ------------------------------------------------------------------------
> *From:* IPAC-List [ipac-list-bounces at ipacweb.org] on behalf of Patrick 
> McCoy [Patrick.McCoy at cfp-psc.gc.ca]
> *Sent:* Friday, June 03, 2016 10:09 AM
> *To:* List, IPAC; Putka, Dan
> *Subject:* Re: [IPAC-List] Michael McDaniel's Reference to the 
> so-called Validity-Diversity Dilemma
>
> Interesting discussion!
> A question, if I might. Has anyone seen convincing research that 
> differences in group test performance are reduced when a test has 
> minimal time press (i.e., when the time to do the test is generous). 
> That assumption is often made, but is it warranted?
> Pat McCoy
> Ottawa, Canada
>
> >>> Dan Putka <dputka at humrro.org> 2016/06/03 9:04 AM >>>
> Hi Richard,
>
> I think decisions regarding this matter will vary from organizations, 
> and hinge on what they value and the resources they can offer to deal 
> with potential inefficiencies in their selection process.
>
> For example, let’s say an organization in question places high value 
> on diversity.  The organization may be willing to sacrifice some level 
> of criterion-related validity with their selection measure if it 
> results in bringing a more diverse workforce in the door.  A critic 
> may rightly argue that using a test with lower criterion-related 
> validity would lower the expected mean performance of individuals 
> hired.  While that would be true, the organization could potentially 
> offset the loss of validity through more rigorous training and 
> performance management (i.e., a post selection intervention designed 
> to ensure a greater proportion of the workforce meet performance 
> standards). Of course adding that rigor may require more investment on 
> the part of the organization, which they may or may not see as worth it.
>
> The other thing to consider here is the magnitude of the difference. 
>  For example, using Mike’s quote as an example, if increasing the time 
> limit of the test has only a small negative impact on validity, but 
> makes a more sizable reduction in subgroup differences, then it would 
> arguably be harder to justify not sacrificing validity in that case. 
>  In that situation, one might construe the situation as the 
> organization having a reasonable alternative to their assumed current 
> approach (i.e., the reduced time limit) that has minimal impact on 
> validity, but is better from a subgroup difference perspective. Of 
> course, a challenge in practice is drawing the line between how much 
> of a drop in validity is too much, and how much of a drop in subgroup 
> differences is enough to warrant deviating from the “maximize 
> validity” philosophy.  Though statistical inference can help here 
> (e.g., non-sig. change in validity, sig. change in subgroup diffs), I 
> think this is a judgment that will also vary from organization to 
> organization.
>
> Personally, I think one of our key roles as scientist-practitioners in 
> the selection arena is to help the organizations we work with 
> understand the implications of potential decisions they make regarding 
> their selection processes.  We can do our best to explain potential 
> options and likely outcomes based on data, theory, literature, past 
> precedent, experience, and offer a reasoned, logical opinion, but the 
> final decision may often not rest in our hands.
>
> Dan
>
> *Dan J. Putka, Ph.D.*
> Principal Staff Scientist
> Office: 703.706.5640
> dputka at humrro.org | www.linkedin.com/in/dputka 
> <redir.aspx?REF=2s2SKeUosblnevQ-FuNhxlr-wuCH_LhPlbSIg2N5W2ijEIi1wovTCAFodHRwOi8vd3d3LmxpbmtlZGluLmNvbS9pbi9kcHV0a2E.> 
>
>
> 66 Canal Center Plaza, Suite 700
> Alexandria, Virginia 22314-1578
> www.humrro.org <UrlBlockedError.aspx>
>
>
>
>
> From: "Richard Joines" <mpscorp at value.net>
> To: "IPAC List" <ipac-list at ipacweb.org>
> Date: 06/02/2016 08:15 PM
> Subject: [IPAC-List] Michael McDaniel's Reference to the so-called 
>  Validity-Diversity Dilemma
> Sent by: "IPAC-List" <ipac-list-bounces at ipacweb.org>
> ------------------------------------------------------------------------
>
>
>
> Mike,
>
> You make the statement that "if job-related reading speed has 
> undesirable consequences such as group differences, one may wish to 
> sacrifice merit hiring for diversity hiring and increase the time 
> limit of the exam."
>
> I guess the question for those who think I/O Psychology is a science 
> is... how does one reach the decision to throw the science out and go 
> another route?  If the result is lowering validity, I'm certainly not 
> about to increase the time limit of any of my empirically validated 
> tests.  There would be no scientific basis for doing that.
>
> I would be interested in what people think about this and how they 
> view their role and what limitations they think they should observe, 
> but my view has always been to try to maximize validity while ensuring 
> compliance with federal guidelines.  Since the 1978 Uniform Guidelines 
> we've been compelled to look for alternative selection methods, the 
> idea being that if we can find or develop a test that has the same or 
> higher validity but lower adverse impact, we should do that.
>
> *However*, the idea that we should sacrifice validity in order to 
> increase diversity strikes me as going too far.  Who are we to make 
> such decisions?  We're supposed to be scientists, not social 
> engineers, yes?
>
> Thoughts anyone?
>
> Rich Joines
> Mgt & Personnel Systems, Inc.
> _www.mps-corp.com_ 
> <redir.aspx?REF=IqPlzWo5Ju-A7cNn-hhKFpvIns_uxouNoQsTiCX66BYFcoq1wovTCAFodHRwOi8vd3d3Lm1wcy1jb3JwLmNvbS8.> 
>
> 925-932-0203
>
> _______________________________________________________
> IPAC-List
> IPAC-List at ipacweb.org
> https://pairlist9.pair.net/mailman/listinfo/ipac-list 
> <redir.aspx?REF=0enhU3NQqgmASs73EOawA-8AD4aCzyDoGIdo_x_jZuQFcoq1wovTCAFodHRwczovL3BhaXJsaXN0OS5wYWlyLm5ldC9tYWlsbWFuL2xpc3RpbmZvL2lwYWMtbGlzdA..>
>
>
>
>
>
> Ce courriel est destiné exclusivement au destinataire mentionné en 
> titre et peut contenir de l'information privilégiée, confidentielle ou 
> soustraite à la communication aux termes des lois applicables. Toute 
> divulgation non autorisée, toute reproduction ou réacheminement est 
> interdit. Si vous n'êtes pas le destinataire de ce courriel, ou n'êtes 
> pas autorisé par le destinataire visé, ou encore, si vous l'avez reçu 
> par erreur, veuillez le mentionner immédiatement à l'expéditeur et 
> supprimer le courriel et les copies.
>
> This e-mail message is intended for the named recipient(s) and may 
> contain information that is privileged, confidential and/or exempt 
> from disclosure under applicable law. Unauthorized disclosure, copying 
> or re-transmission is prohibited. If you are not a named recipient or 
> not authorized by the named recipient(s), or if you have received this 
> e-mail in error, then please notify the sender immediately and delete 
> the message and any copies.
>
>
> _______________________________________________________
> IPAC-List
> IPAC-List at ipacweb.org
> https://pairlist9.pair.net/mailman/listinfo/ipac-list

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist9.pair.net/pipermail/ipac-list/attachments/20160603/78a1a8d8/attachment-0001.html>