[IPAC-List] promotional exam strategy

[Winfred Arthur, Jr.]

just echoing Mark and Harry's comments.  i currently use a similar 
approach in my undergraduate rsch methods, graduate personnel selection, 
and graduate advanced personnel selection and placement courses.  
however, i provide a concepts and principles list (about 120) but not 
item stems.

the obvious advantages are as articulated by Mark and Harry; it defines 
the content domain and although i cannot test them on everything, it 
gives me some assurance that they would have made an effort to master 
all the material on the list.

i have not used this approach in an applied context but as Harry notes, 
it is really just a variant of providing a reading list; albeit somewhat 
more specific and narrow.

like Harry, i don't see any reason why structurally this approach shld 
result in exam psychometric properties that are inherently different 
from other approaches to defining the exam content domain.  likewise, i 
don't think the resulting exam will be any more g-loaded than other 
knowledge-based exams.

curious -- you did not explicitly state so but i assume the service is 
generally pleased w/ the approach?  so, what is the push-back or 
concern?  and form what source?

thanks.

- winfred


On 6/28/2011 2:56 PM, Mark Hammer wrote:

> I used to use something like that for teaching as well.  I'd list all

> the concepts I wanted students to know, and terms I wanted them to be

> familiar with, and inform them that this was the master list, from which

> I would draw questions, hopefully packing as much of that into a

> predefined number of questions as I could.  (In some instances, the

> distractors also require you to understand the concept, just so you can

> classify it as a distractor and invalid response, such that one question

> can actually address much more than just the material in the correct

> response.)

>

> The advantage it provides to testees is that it defines what they

> *don't* have to worry about when preparing, and they appreciate that.

> Assuming the range of information covered off by the list of stems is of

> an appropriate size compared to the final list of questions (which in

> this case it seems to be, 5:1), it will be perceived as fair by testees,

> and also gives you some confidence that they've spent some time learning

> about things that maybe you'd like them to know, but don't have

> time/space to test them on.

>

> The caveat is that you'd want some assurances that any given subset of

> 100 was roughly equal in both difficulty and predictive value.  Y

> certainly don't want to find yourself with mean scores that fluctuate

> from session/class to session/class.  And that entails a bit of work on

> your part.

>

> Mark Hammer

> Ottawa

>

>>>> <Harry.Brull at pdininthhouse.com>  2011/06/28 2:44 PM>>>

> This would seem to me as a variant on providing a reading list as a

> study guide. In this case, the jurisdiction is directing the

> candidates

> to the issues (items) that are considered important.

> I have used a comparable technique in my collegiate teaching.

> As for what to call it - beyond an inspired idea - I'd suggest

> "item-based exam preparation/study guide".

>

> As for the psychometric properties, I see no reason to expect anything

> different than ant content-based multiple choice test - high

> coefficient

> alpha.

> As for what you are really measuring, some combination of job

> knowledge

> and a healthy does of "G".

>

> Harry Brull

> H.A.R.

> PDI Ninth House

>

>

> ------------------------------------------------

> Harry Brull |  Senior Vice-President

> PDI Ninth House

> Global Leadership Solutions

>

> 1.612.337.8233  office

> 1.612.414.8998  mobile

> 1.612.337.3695  fax

> Harry.Brull at PDINinthHouse.com

>

> 33 South Sixth Street

> Suite 4900

> Minneapolis, MN 55402

>

> www.pdininthhouse.com

>

> -----Original Message-----

> From: ipac-list-bounces at ipacweb.org

> [mailto:ipac-list-bounces at ipacweb.org] On Behalf Of Morris, Ramona

> (JUS)

> Sent: Tuesday, June 28, 2011 1:32 PM

> To: ipac-list at ipacweb.org

> Subject: [IPAC-List] promotional exam strategy

>

> Situation:

>

> A police service is using a test on their local policy and procedures

> as

> part of the promotional process.  Candidates are provided with the

> pool

> of items (say 500)...just the stems without the distractors. The test

> consists of a subset of multiple choice items(say 100).

>

> The argument is that they want candidates to know all of the p and p,

> and it is up to the candidate to prepare themselves. And, exam

> security

> becomes less of a headache.

>

> Questions:

>

> 1.    What is this strategy called?

>

> 2.    Do you have any advice about its use?

>

> 3.    What do we know about the strategy (e.g. validity, reliability

> efficiency etc.)

>

>

>

> Ramona Morris

>

> Ontario Police College

>

> Aylmer, Ontario

>

>

> _______________________________________________________

> IPAC-List

> IPAC-List at ipacweb.org

> http://www.ipacweb.org/mailman/listinfo/ipac-list

>

> Confidentiality Notice: All information in this communication,

> including any files or attachments, is intended for the sole use of the

> individual or entity to which it is addressed and may contain

> information that is confidential, proprietary and/or trade secret

> information entitled to protection and/or exempt from disclosure under

> applicable law. If you are not the intended recipient, you are hereby

> notified that any dissemination, distribution, or copying of this

> communication is strictly prohibited. If you have received this

> communication in error, please notify the sender by return email and

> delete this communication from your system. Thank you for your

> cooperation.

> _______________________________________________________

> IPAC-List

> IPAC-List at ipacweb.org

> http://www.ipacweb.org/mailman/listinfo/ipac-list

>

> This e-mail message is intended for the named recipient(s) and may

> contain information that is privileged, confidential and/or exempt from

> disclosure under applicable law. Unauthorized disclosure, copying or

> re-transmission is prohibited. If you are not a named recipient or not

> authorized by the named recipient(s), or if you have received this

> e-mail in error, then please notify the sender immediately and delete

> the message and any copies.

>   >

> Ce courriel est destiné exclusivement au destinataire mentionné en titre

> et peut contenir de l'information privilégiée, confidentielle ou

> soustraite à la communication aux termes des lois applicables. Toute

> divulgation non autorisée, toute reproduction ou réacheminement est

> interdit. Si vous n'êtes pas le destinataire de ce courriel, ou n'êtes

> pas autorisé par le destinataire visé, ou encore, si vous l'avez reçu

> par erreur, veuillez le mentionner immédiatement à l'expéditeur et

> supprimer le courriel et les copies.

>

> _______________________________________________________

> IPAC-List

> IPAC-List at ipacweb.org

> http://www.ipacweb.org/mailman/listinfo/ipac-list