[IPAC-List] Fwd: Changes in assessment research and practice

Fri Aug 10 13:50:25 EDT 2018

good set of comments Dennis and Scott  :)

for the structured  vs. unstructured interview issue, their chpt (see 
ref below) speaks more directly to that.  (let me know if you are 
interested in a copy.)

Oh, I.-S., Postlethwaite, B. E., & Schmidt, F. L. (2013). Rethinking the 
validity of interviews for employment decision making: Implications of 
recent developments in meta-analysis (Chapter 12, pp. 297-329). In D. J. 
Svyantek & K. Mahoney (Eds.), Received wisdom, kernels of truth, and 
boundary conditions in organizational studies. Charlotte, NC: 
Information Age Publishing.

their treatise for questioning the prevailing empirical work and 
thinking on the superiority of structured over unstructured interviews 
is predicated on "new evidence based on more accurate calibrations of 
range restriction and publication bias".  as Dennis and i have mused 
over in the past, whether these are more accurate corrections or over 
corrections that reflect "a world that does not exist", is the 
question.  however, since this is work is a reanalysis of Mike's 1994 
piece, i believe he is in a good position to comment on this.

furthermore, i think this paragraph at the end of the chpt is informative:

"Lastly, we again note that the purpose of the current study is not to 
update McDaniel et al. (1994) but to illustrate the potential influence 
of methodological advancements and refinements on meta-analytic results 
and conclusions using the data used in McDaniel et al. (1994). Given 
this, our re-analyses were conducted by focusing on a broad difference 
in validity between structured and unstructured interviews across 
conditions. Thus, we cannot rule out the possibility that publication 
bias might be more (or less) severe under some specific conditions 
(e.g., in predictive validation studies, among less experienced 
interviewers). In addition, our findings may not be generalizable beyond 
McDaniel et al.’ (1994) data set. It is rather obvious that many primary 
studies have been conducted since the publication of McDaniel et al. 
(1994). Accordingly, future meta-analytic updates of McDaniel et al. 
(1994) should employ the most updated range restriction correction and 
publication bias detection/adjustment methods and also examine all 
important moderators to be more informative."

- winfred

On 8/10/2018 9:11 AM, Scott Edward Highhouse wrote:
>
>
> Sent from my iPhone
>
> Begin forwarded message:
>
>> *From:* Scott Edward Highhouse <shighho at bgsu.edu 
>> <mailto:shighho at bgsu.edu>>
>> *Date:* August 10, 2018 at 10:08:09 AM EDT
>> *To:* "Tsugawa, James" <James.Tsugawa at mspb.gov 
>> <mailto:James.Tsugawa at mspb.gov>>
>> *Subject:* *Re: [IPAC-List] Changes in assessment research and practice*
>>
>> James
>> As I recall, they used only studies in which the data were for 
>> research purposes only (not operational).
>>
>> It is also important to keep in mind that the vast majority of UIs 
>> are not scored. You must have a scored UI to include it in a 
>> meta-analysis. As such, those UIs are likely on the high end of rigor.
>>
>> Scott
>>
>> Sent from my iPhone
>>
>> On Aug 10, 2018, at 9:17 AM, Tsugawa, James via IPAC-List 
>> <ipac-list at ipacweb.org <mailto:ipac-list at ipacweb.org>> wrote:
>>
>>> Good morning -
>>>
>>>
>>> A year or two ago, I saw an update of the much-cited 1998 Schmidt 
>>> and Hunter metaanalysis of assessment methods.
>>>
>>> One striking result was that unstructured interviews (UIs) fared 
>>> surprisingly well.  The paper at the link below contains the 
>>> findings of interest.
>>>
>>>
>>> https://home.ubalt.edu/tmitch/645/articles/2016-100%20Yrs%20Working%20Paper%20for%20Research%20Gate%2010-17.pdf 
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__secure-2Dweb.cisco.com_1RYcTtBusitkpwm1ukXhE5I9L6wtoMFU1rZ-2D9lmdXi2z3MrX-5F9uHzPAC6sN4KgWNk1gsCiiZf9vpDlzZhg27BHSm5dIcFpBupsXaDEQf06x97jxWXArU1bel3cZ0htjdcu-5Fi88fQv3XRRbFAjmBY26V46IKTz-2DNu-5FrqMQHGFFzBRiI9NPKOwFFc2-5FrnmiED999g7oAwDYY8QwkgpWrbIZHiILvnElQefF4MotrkKz6qjrIeF-5F2xXQVSjjmvjW-5FpNelLZqGW9BjqUZ13Qfm4IsAV5JGSlzzz2mkTcLO0Sw3KK7Q3CFpQ148nTZwRWDPoDZQT2Cg0rxM1kPfhhIz46yyBcc2gVpkMTC7USNaI-5FPvYHQ3qOtjeyAI96Duj5slnlZvOe5biUm2MeiZfiSVC9bt2sedTKiRfUx1hgZGWeQGh-5FmhqeuWqH4Zz18x7K1OEKcr2eOIZsWOAUvjXERDGpJViAttt8tgH1E5qs2Ubz3yRPKdQLvyvB578OxJZgFVWdzeNinH-2DwXvHQ-2D9V2C9hPwDA_https-253A-252F-252Fhome.ubalt.edu-252Ftmitch-252F645-252Farticles-252F2016-2D100-252520Yrs-252520Working-252520Paper-252520for-252520Research-252520Gate-25252010-2D17.pdf&d=DwMGaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=QPXvfD7OlbB-bn0RogVh0PVRDyppaZvhXQLzH1IbdZ8&m=06sWW8eLBrSQrZWnHHzaVs1jIBNDQ8NlpCAQ7I3l-ZE&s=bt2uxb7cyJpumWwHBrJcsTEKJL51conTOwxWFtpG6PY&e=>
>>>
>>> So, two questions:
>>>
>>> (1) Was that result further explored or confirmed in other studies?  
>>> For example, do UIs have previously unseen or underappreciated 
>>> measurement properties?  Or might there be a selection effect at 
>>> work?  (Given standards for metaanalysis and studies, the 
>>> UIs probably weren't the ones of our nightmares.)
>>>
>>> (2) Has it influenced subsequent research--or your own practice, as 
>>> a developer/recommender/user of assessments?
>>>
>>>
>>> James Tsugawa / U.S. MSPB
>>>
>>>
>>> _______________________________________________________
>>> IPAC-List
>>> IPAC-List at ipacweb.org <mailto:IPAC-List at ipacweb.org>
>>> https://secure-web.cisco.com/1M3fIt49rHa8IRF83netM3v2mPQ6sBG3YquAw3u_uQxYN3FGNmSU5plnUDxW4JblTqw1w7r_rMNJgfb7Xq8P-dHSfx2lbtBwDoKDHWmIUkhMWd2P6NmNscbQZhrg1SgPp891Ya85DL1TMF2zOgWc58CoeSFZ5Sc_WRNNxqX3OTPfDQBXlqJIVvP0CPtQ95dN-G9D0tOSY8VrXGP4xpxC4PYTxoSwVY_lPWz2ZIGzooi2fvbaDn2M0zKceTNpgGRb2GLVaJE_4rwxyEw2lsKERSwqukoEAu-56JeH0EPpYhoOyDZ3eRYzCszJL6flr5vG6UP0Xqan_D7TNe8_qy-mYF3aSakWg_FKDcr8U7oQg5h0Gqz6HKt9XVJBEFklLodW6GSCPwnLTln9IlA6IEly9xUzlysiYVi-kUay7vuzGJaTPh4AkFF3L-OYXshiEmr-3UdVq8ropC3JZgGiAoNrykNS-PzqEAZ3DkIXXOccIpUecQL3SsPwvV9MqgngKVe_apfymNmRQOvkbZLfrwq8xpQ/https%3A%2F%2Fpairlist9.pair.net%2Fmailman%2Flistinfo%2Fipac-list 
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__secure-2Dweb.cisco.com_1M3fIt49rHa8IRF83netM3v2mPQ6sBG3YquAw3u-5FuQxYN3FGNmSU5plnUDxW4JblTqw1w7r-5FrMNJgfb7Xq8P-2DdHSfx2lbtBwDoKDHWmIUkhMWd2P6NmNscbQZhrg1SgPp891Ya85DL1TMF2zOgWc58CoeSFZ5Sc-5FWRNNxqX3OTPfDQBXlqJIVvP0CPtQ95dN-2DG9D0tOSY8VrXGP4xpxC4PYTxoSwVY-5FlPWz2ZIGzooi2fvbaDn2M0zKceTNpgGRb2GLVaJE-5F4rwxyEw2lsKERSwqukoEAu-2D56JeH0EPpYhoOyDZ3eRYzCszJL6flr5vG6UP0Xqan-5FD7TNe8-5Fqy-2DmYF3aSakWg-5FFKDcr8U7oQg5h0Gqz6HKt9XVJBEFklLodW6GSCPwnLTln9IlA6IEly9xUzlysiYVi-2DkUay7vuzGJaTPh4AkFF3L-2DOYXshiEmr-2D3UdVq8ropC3JZgGiAoNrykNS-2DPzqEAZ3DkIXXOccIpUecQL3SsPwvV9MqgngKVe-5FapfymNmRQOvkbZLfrwq8xpQ_https-253A-252F-252Fpairlist9.pair.net-252Fmailman-252Flistinfo-252Fipac-2Dlist&d=DwMGaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=QPXvfD7OlbB-bn0RogVh0PVRDyppaZvhXQLzH1IbdZ8&m=06sWW8eLBrSQrZWnHHzaVs1jIBNDQ8NlpCAQ7I3l-ZE&s=S4lk-us1hDbfLr8ujMTnML3tEIszkalbydK_4XM8kpU&e=>
>
>
> _______________________________________________________
> IPAC-List
> IPAC-List at ipacweb.org
> https://pairlist9.pair.net/mailman/listinfo/ipac-list

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist9.pair.net/pipermail/ipac-list/attachments/20180810/597222c9/attachment.html>