[IPAC-List] Asking SMEs to evaluate the representativeness of a test, as a whole?

Thu Aug 26 22:20:26 EDT 2021

This sounds like what I am looking for, Michael.

Do you have any other recollection that might help me track down such cases?

Thanks.

Joel

On 8/26/21 3:01 PM, Blair, Michael wrote:
>
> In addition to the appeal process Lance describes, there have been 
> instances in which SMEs were reconvened after the test administration 
> to review content validity in association with legal challenges to the 
> assessment tool and/or process. I do not have a listing of these, but 
> those that ended up in court would be documented in legal briefings.
>
> *Michael D. Blair*
>
> *Lead Personnel Research Psychologist*
>
> *United States Office of Personnel Management*
>
> P: 202-957-5427 | M: 202-957-5427
>
> Michael.Blair2 at opm.gov <mailto:Michael.Blair2 at opm.gov>| 
> www.opm.gov/HRS <http://www.opm.gov/HRS>
>
> *From:* IPAC-List <ipac-list-bounces at ipacweb.org> *On Behalf Of *Lance 
> Seberhagen
> *Sent:* Thursday, August 26, 2021 1:49 PM
> *To:* 'Joel Wiesen' <jwiesen at appliedpersonnelresearch.com>; 
> 'IPAC-List' <IPAC-List at ipacweb.org>
> *Subject:* Re: [IPAC-List] Asking SMEs to evaluate the 
> representativeness of a test, as a whole?
>
> Some employers permit test-takers to review the scoring of promotional 
> exams.  If test-takers appeal the scoring of certain promotional exam 
> questions, the employer might have SMEs take another look at the 
> questions and scoring procedures to see if test-takers have a 
> legitimate complaint.  If so, the scoring of these questions might be 
> revised (e.g., 2 correct answers for a given question), resulting in a 
> new rank order of test-takers based on test scores.
>
> Lance
>
> ---
> Lance Seberhagen, Ph.D.
> Seberhagen & Associates
> 9021 Trailridge Ct
> Vienna, VA 22182
> Tel 703-790-0796
> www.seberhagen.com <http://www.seberhagen.com/>
>
> *From:* Joel Wiesen <jwiesen at appliedpersonnelresearch.com 
> <mailto:jwiesen at appliedpersonnelresearch.com>>
> *Sent:* Thursday, August 26, 2021 2:27 PM
> *To:* sebe at rcn.com <mailto:sebe at rcn.com>; 'IPAC-List' 
> <IPAC-List at ipacweb.org <mailto:IPAC-List at ipacweb.org>>
> *Subject:* Re: [IPAC-List] Asking SMEs to evaluate the 
> representativeness of a test, as a whole?
>
> Lance,
>
> Thank you for your reply.
>
> I agree, we do our best to develop a test that is a representative 
> sample of the job.
>
> I also agree that the concern about test security limits what SMEs can 
> do before the exam is held, for the reason you describe.
>
> My inquiry concerns SME input after the exam has been held. (My 
> question should have been worded more clearly.)
>
> After the test is held, test security is less of an issue.
>
> I am wondering about the use of SME's, after the test has been held, 
> to evaluate whether the test is a representative sample of the job.
>
> Has anyone done that and, if so, how has it worked out?
>
> Thank you.
>
> Joel
>
> On 8/26/21 1:51 PM, Lance Seberhagen wrote:
>
>     SMEs normally rate each test item.  Test security is major
>     concern.  Must prevent SMEs from leaking exam questions to
>     candidates.  This has been a problem in police depts.  One
>     solution is to use SMEs from other police depts in same state.  If
>     SMEs from same dept are used, they must be high-ranking officials
>     who sign nondisclosure agreement, with severe penalty for
>     violation.  Dept should also have general order that prohibits
>     cheating on promotional exams, with severe penalty for violation.
>      I have had no problem with promotional exams I have developed
>     following these procedures.
>
>     Lance
>
>     ---
>     Lance Seberhagen, Ph.D.
>     Seberhagen & Associates
>     9021 Trailridge Ct
>     Vienna, VA 22182
>     Tel 703-790-0796
>     www.seberhagen.com <http://www.seberhagen.com/>
>
>     *From:* IPAC-List <ipac-list-bounces at ipacweb.org>
>     <mailto:ipac-list-bounces at ipacweb.org> *On Behalf Of *Joel Wiesen
>     *Sent:* Friday, August 20, 2021 3:26 PM
>     *To:* IPAC-List <IPAC-List at ipacweb.org> <mailto:IPAC-List at ipacweb.org>
>     *Subject:* [IPAC-List] Asking SMEs to evaluate the
>     representativeness of a test, as a whole?
>
>     Has anyone asked SMEs to evaluate or rate the representativeness
>     of the content of a promotional exam?
>
>     I would be interested in learning how that was done and how it
>     worked out.
>
>     Thank you.
>
>     Joel
>
>     -- 
>
>     "Injustice anywhere is a threat to justice everywhere."
>
>     Dr. Martin Luther King, Jr.; Letter from a Birmingham Jail, April 16, 1963.
>
>       
>
>     - -
>
>     Joel P. Wiesen, Ph.D., Director
>
>     Applied Personnel Research
>
>     62 Candlewood Road
>
>     Scarsdale, NY 10583-6040
>
>     http://www.linkedin.com/in/joelwiesen  <http://www.linkedin.com/in/joelwiesen>
>
>     (617) 244-8859
>
>     http://appliedpersonnelresearch.com  <http://appliedpersonnelresearch.com>
>
>       
>
>     Continuing Education website (home study of recent journal articles):https://www.aprtestingservice.com/  <https://www.aprtestingservice.com/>
>
>       
>
>     Note: This e-mail and any attachments may contain confidential and/or legally privileged information. Please do not forward any contents without permission. If you have received this message in error please destroy all copies, completely remove it from your computer, and notify the sender. Thank you.
>
>       
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist9.pair.net/pipermail/ipac-list/attachments/20210826/816522e4/attachment.htm>