[IPAC-List] Asking SMEs to evaluate the representativeness of a test, as a whole?

Fri Aug 27 14:59:17 EDT 2021

Thank you for your thoughtful comments, Michael.

I agree that often we strive to make test content representative of job 
content, but there is unavoidable subjectivity in deciding which exact 
questions to ask.  It is possible that this subjectivity could undermine 
the procedures we follow that are intended to assure that the test 
content is a representative sample of job content.  I would like SMEs to 
review the final product (the test) to see whether, or to what extent, 
the test development efforts were successful in producing test content 
that is representative of job content.

If others have conducted such reviews, I would like to learn how they 
went about it.

FYI, this is not being done in connection with a legal challenge. 
Rather, I am trying to identify any weaknesses in representativeness to 
inform future test development.  Of course, if the SMEs give the exam a 
perfect bill of health, I'll use that in support of future exams, too.

Take care,
Joel

On 8/27/21 10:37 AM, Blair, Michael wrote:

> Joel –
>
> I would argue that the discussion and confirmation (i.e., #4) should 
> occur prior to finalizing and administering the test. 
> Representativeness of the job is a foundational aspect that I/O 
> psychologists build into the job analysis, test development, and 
> content validity processes. Assessment experts want to ensure that 
> test content is linked to and represents important aspects of the job 
> before finalizing the assessment. If established professional 
> practices aligning with appropriate legal guidelines are followed, 
> there would be no need to do it after the fact, short of what Lance 
> shared in terms of the appeal process that many organizations, 
> especially those in the public sector, have in place. Organizations 
> that are doing this after the fact, most likely in reaction to a 
> challenge and lacking appropriate documentation/records, could follow 
> a similar process to establish content validity.
>
> If your question is focused on how representative a particular 
> assessment is of the job as a whole, this starts with the test plan. 
> To ensure the test content accurately reflects the test plan, a 
> strategy would be to establish linkage between test content and 
> competencies or KSAs (depending on the approach) in the test plan. The 
> extent to which test content links to aspects of the job identified in 
> the test plan, representativeness of the assessment is established.
>
> Not sure if this helps or not. If you’re in the awkward position of 
> trying to defend an assessment that is lacking in technical 
> documentation, I wish you well.
>
> *Michael D. Blair*
>
> *Lead Personnel Research Psychologist*
>
> *United States Office of Personnel Management*
>
> P: 202-957-5427 | M: 202-957-5427
>
> Michael.Blair2 at opm.gov <mailto:Michael.Blair2 at opm.gov>| 
> www.opm.gov/HRS <http://www.opm.gov/HRS>
>
> *From:* IPAC-List <ipac-list-bounces at ipacweb.org> *On Behalf Of *Joel 
> Wiesen
> *Sent:* Thursday, August 26, 2021 9:22 PM
> *To:* Mudd, Jordan <JMudd at jcsoky.org>; IPAC-List <IPAC-List at ipacweb.org>
> *Subject:* Re: [IPAC-List] Asking SMEs to evaluate the 
> representativeness of a test, as a whole?
>
> Hello Jordan,
>
> Thank you for your reply.
>
> It is #4 on your list that I am most interested in now: 
> representativeness of content.
>
> We can try hard to create tests that mirror the job or are 
> representative of the job. But, at the end of the day, do SMEs think 
> our tests reflect the content of the job (e.g., breath, depth, level 
> of detail)
>
> Has anyone used SME's, after the test has been held, to evaluate 
> whether the test is a representative sample of the job?
>
> Thanks.
>
> Joel
>
> On 8/26/21 4:54 PM, Mudd, Jordan wrote:
>
>     A strategy I have used for SME reviews of potential written test questions:
>
>     1) Write 120-150 potential questions for a 100 question exam.
>
>     2) Divide questions into groups of 20 or so.
>
>     3) Have at least one SME review each group of potential questions on the following:
>
>     A) Is the knowledge needed required for successful performance?
>
>     B) Is the needed before promotion?
>
>     C) Does the content of the question match agency practice? (Policy vs practice)
>
>     4) Eliminate questions based on material and representatives of content.
>
>     This procedure will allow SMEs to individually only know a small percentage of potential questions to be on the test. Same can be done for scenarios where 1 SME helps develop/evaluate portion of total set of scenarios. Selection of SMEs is a whole other process to consider and probably an entirely different thread.
>
>     Jordan Mudd
>
>     Chief Examiner
>
>     Jefferson County Sheriff’s Office
>
>     Louisville, KY
>
>     ________________________________
>
>     From: IPAC-List<ipac-list-bounces at ipacweb.org>  <mailto:ipac-list-bounces at ipacweb.org>  on behalf of Blair, Michael via IPAC-List<ipac-list at ipacweb.org>  <mailto:ipac-list at ipacweb.org>
>
>     Sent: Thursday, August 26, 2021 3:01:54 PM
>
>     To:sebe at rcn.com  <mailto:sebe at rcn.com>; 'Joel Wiesen'; 'IPAC-List'
>
>     Subject: Re: [IPAC-List] Asking SMEs to evaluate the representativeness of a test, as a whole?
>
>     In addition to the appeal process Lance describes, there have been instances in which SMEs were reconvened after the test administration to review content validity in association with legal challenges to the assessment tool and/or process. I do not have a listing of these, but those that ended up in court would be documented in legal briefings.
>
>     Michael D. Blair
>
>     Lead Personnel Research Psychologist
>
>     United States Office of Personnel Management
>
>     P: 202-957-5427 | M: 202-957-5427
>
>     Michael.Blair2 at opm.gov  <mailto:Michael.Blair2 at opm.gov><mailto:Michael.Blair2 at opm.gov>  <mailto:Michael.Blair2 at opm.gov>  |www.opm.gov/HRS  <http://www.opm.gov/HRS><http://www.opm.gov/HRS>  <http://www.opm.gov/HRS>
>
>     From: IPAC-List<ipac-list-bounces at ipacweb.org>  <mailto:ipac-list-bounces at ipacweb.org>  On Behalf Of Lance Seberhagen
>
>     Sent: Thursday, August 26, 2021 1:49 PM
>
>     To: 'Joel Wiesen'<jwiesen at appliedpersonnelresearch.com>  <mailto:jwiesen at appliedpersonnelresearch.com>; 'IPAC-List'<IPAC-List at ipacweb.org>  <mailto:IPAC-List at ipacweb.org>
>
>     Subject: Re: [IPAC-List] Asking SMEs to evaluate the representativeness of a test, as a whole?
>
>     Some employers permit test-takers to review the scoring of promotional exams.  If test-takers appeal the scoring of certain promotional exam questions, the employer might have SMEs take another look at the questions and scoring procedures to see if test-takers have a legitimate complaint.  If so, the scoring of these questions might be revised (e.g., 2 correct answers for a given question), resulting in a new rank order of test-takers based on test scores.
>
>     Lance
>
>     ---
>
>     Lance Seberhagen, Ph.D.
>
>     Seberhagen & Associates
>
>     9021 Trailridge Ct
>
>     Vienna, VA 22182
>
>     Tel 703-790-0796
>
>     www.seberhagen.com  <http://www.seberhagen.com><http://www.seberhagen.com/>  <http://www.seberhagen.com/>
>
>     From: Joel Wiesen <jwiesen at appliedpersonnelresearch.com  <mailto:jwiesen at appliedpersonnelresearch.com><mailto:jwiesen at appliedpersonnelresearch.com>  <mailto:jwiesen at appliedpersonnelresearch.com>>
>
>     Sent: Thursday, August 26, 2021 2:27 PM
>
>     To:sebe at rcn.com  <mailto:sebe at rcn.com><mailto:sebe at rcn.com>  <mailto:sebe at rcn.com>; 'IPAC-List' <IPAC-List at ipacweb.org  <mailto:IPAC-List at ipacweb.org><mailto:IPAC-List at ipacweb.org>  <mailto:IPAC-List at ipacweb.org>>
>
>     Subject: Re: [IPAC-List] Asking SMEs to evaluate the representativeness of a test, as a whole?
>
>     Lance,
>
>     Thank you for your reply.
>
>     I agree, we do our best to develop a test that is a representative sample of the job.
>
>     I also agree that the concern about test security limits what SMEs can do before the exam is held, for the reason you describe.
>
>     My inquiry concerns SME input after the exam has been held.  (My question should have been worded more clearly.)
>
>     After the test is held, test security is less of an issue.
>
>     I am wondering about the use of SME's, after the test has been held, to evaluate whether the test is a representative sample of the job.
>
>     Has anyone done that and, if so, how has it worked out?
>
>     Thank you.
>
>     Joel
>
>     On 8/26/21 1:51 PM, Lance Seberhagen wrote:
>
>     SMEs normally rate each test item.  Test security is major concern.  Must prevent SMEs from leaking exam questions to candidates.  This has been a problem in police depts.  One solution is to use SMEs from other police depts in same state.  If SMEs from same dept are used, they must be high-ranking officials who sign nondisclosure agreement, with severe penalty for violation.  Dept should also have general order that prohibits cheating on promotional exams, with severe penalty for violation.  I have had no problem with promotional exams I have developed following these procedures.
>
>     Lance
>
>     ---
>
>     Lance Seberhagen, Ph.D.
>
>     Seberhagen & Associates
>
>     9021 Trailridge Ct
>
>     Vienna, VA 22182
>
>     Tel 703-790-0796
>
>     www.seberhagen.com  <http://www.seberhagen.com><http://www.seberhagen.com/>  <http://www.seberhagen.com/>
>
>     From: IPAC-List<ipac-list-bounces at ipacweb.org>  <mailto:ipac-list-bounces at ipacweb.org><mailto:ipac-list-bounces at ipacweb.org>  <mailto:ipac-list-bounces at ipacweb.org>  On Behalf Of Joel Wiesen
>
>     Sent: Friday, August 20, 2021 3:26 PM
>
>     To: IPAC-List<IPAC-List at ipacweb.org>  <mailto:IPAC-List at ipacweb.org><mailto:IPAC-List at ipacweb.org>  <mailto:IPAC-List at ipacweb.org>
>
>     Subject: [IPAC-List] Asking SMEs to evaluate the representativeness of a test, as a whole?
>
>     Has anyone asked SMEs to evaluate or rate the representativeness of the content of a promotional exam?
>
>     I would be interested in learning how that was done and how it worked out.
>
>     Thank you.
>
>     Joel
>
>     --
>
>     "Injustice anywhere is a threat to justice everywhere."
>
>     Dr. Martin Luther King, Jr.; Letter from a Birmingham Jail, April 16, 1963.
>
>     - -
>
>     Joel P. Wiesen, Ph.D., Director
>
>     Applied Personnel Research
>
>     62 Candlewood Road
>
>     Scarsdale, NY 10583-6040
>
>     http://www.linkedin.com/in/joelwiesen  <http://www.linkedin.com/in/joelwiesen>
>
>     (617) 244-8859
>
>     http://appliedpersonnelresearch.com  <http://appliedpersonnelresearch.com>
>
>     Continuing Education website (home study of recent journal articles):https://www.aprtestingservice.com/  <https://www.aprtestingservice.com/>
>
>     Note: This e-mail and any attachments may contain confidential and/or legally privileged information. Please do not forward any contents without permission. If you have received this message in error please destroy all copies, completely remove it from your computer, and notify the sender. Thank you.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist9.pair.net/pipermail/ipac-list/attachments/20210827/86575804/attachment-0001.htm>