[IPAC-List] Fwd: Changes in assessment research and practice

[Scott Edward Highhouse]

Sent from my iPhone

Begin forwarded message:

From: Scott Edward Highhouse <shighho at bgsu.edu<mailto:shighho at bgsu.edu>>
Date: August 10, 2018 at 10:08:09 AM EDT
To: "Tsugawa, James" <James.Tsugawa at mspb.gov<mailto:James.Tsugawa at mspb.gov>>
Subject: Re: [IPAC-List] Changes in assessment research and practice

James
As I recall, they used only studies in which the data were for research purposes only (not operational).

It is also important to keep in mind that the vast majority of UIs are not scored. You must have a scored UI to include it in a meta-analysis. As such, those UIs are likely on the high end of rigor.

Scott

Sent from my iPhone

On Aug 10, 2018, at 9:17 AM, Tsugawa, James via IPAC-List <ipac-list at ipacweb.org<mailto:ipac-list at ipacweb.org>> wrote:


Good morning -


A year or two ago, I saw an update of the much-cited 1998 Schmidt and Hunter metaanalysis of assessment methods.

One striking result was that unstructured interviews (UIs) fared surprisingly well.  The paper at the link below contains the findings of interest.


https://home.ubalt.edu/tmitch/645/articles/2016-100%20Yrs%20Working%20Paper%20for%20Research%20Gate%2010-17.pdf<https://secure-web.cisco.com/1RYcTtBusitkpwm1ukXhE5I9L6wtoMFU1rZ-9lmdXi2z3MrX_9uHzPAC6sN4KgWNk1gsCiiZf9vpDlzZhg27BHSm5dIcFpBupsXaDEQf06x97jxWXArU1bel3cZ0htjdcu_i88fQv3XRRbFAjmBY26V46IKTz-Nu_rqMQHGFFzBRiI9NPKOwFFc2_rnmiED999g7oAwDYY8QwkgpWrbIZHiILvnElQefF4MotrkKz6qjrIeF_2xXQVSjjmvjW_pNelLZqGW9BjqUZ13Qfm4IsAV5JGSlzzz2mkTcLO0Sw3KK7Q3CFpQ148nTZwRWDPoDZQT2Cg0rxM1kPfhhIz46yyBcc2gVpkMTC7USNaI_PvYHQ3qOtjeyAI96Duj5slnlZvOe5biUm2MeiZfiSVC9bt2sedTKiRfUx1hgZGWeQGh_mhqeuWqH4Zz18x7K1OEKcr2eOIZsWOAUvjXERDGpJViAttt8tgH1E5qs2Ubz3yRPKdQLvyvB578OxJZgFVWdzeNinH-wXvHQ-9V2C9hPwDA/https%3A%2F%2Fhome.ubalt.edu%2Ftmitch%2F645%2Farticles%2F2016-100%2520Yrs%2520Working%2520Paper%2520for%2520Research%2520Gate%252010-17.pdf>


So, two questions:

(1) Was that result further explored or confirmed in other studies?  For example, do UIs have previously unseen or underappreciated measurement properties?  Or might there be a selection effect at work?  (Given standards for metaanalysis and studies, the UIs probably weren't the ones of our nightmares.)

(2) Has it influenced subsequent research--or your own practice, as a developer/recommender/user of assessments?


James Tsugawa / U.S. MSPB


_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org<mailto:IPAC-List at ipacweb.org>
https://secure-web.cisco.com/1M3fIt49rHa8IRF83netM3v2mPQ6sBG3YquAw3u_uQxYN3FGNmSU5plnUDxW4JblTqw1w7r_rMNJgfb7Xq8P-dHSfx2lbtBwDoKDHWmIUkhMWd2P6NmNscbQZhrg1SgPp891Ya85DL1TMF2zOgWc58CoeSFZ5Sc_WRNNxqX3OTPfDQBXlqJIVvP0CPtQ95dN-G9D0tOSY8VrXGP4xpxC4PYTxoSwVY_lPWz2ZIGzooi2fvbaDn2M0zKceTNpgGRb2GLVaJE_4rwxyEw2lsKERSwqukoEAu-56JeH0EPpYhoOyDZ3eRYzCszJL6flr5vG6UP0Xqan_D7TNe8_qy-mYF3aSakWg_FKDcr8U7oQg5h0Gqz6HKt9XVJBEFklLodW6GSCPwnLTln9IlA6IEly9xUzlysiYVi-kUay7vuzGJaTPh4AkFF3L-OYXshiEmr-3UdVq8ropC3JZgGiAoNrykNS-PzqEAZ3DkIXXOccIpUecQL3SsPwvV9MqgngKVe_apfymNmRQOvkbZLfrwq8xpQ/https%3A%2F%2Fpairlist9.pair.net%2Fmailman%2Flistinfo%2Fipac-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist9.pair.net/pipermail/ipac-list/attachments/20180810/941748ed/attachment-0001.html>