[IPAC-List] promotional exam strategy

Winfred Arthur, Jr. w-arthur at neo.tamu.edu
Tue Jun 28 16:20:59 EDT 2011

just echoing Mark and Harry's comments. i currently use a similar
approach in my undergraduate rsch methods, graduate personnel selection,
and graduate advanced personnel selection and placement courses.
however, i provide a concepts and principles list (about 120) but not
item stems.

the obvious advantages are as articulated by Mark and Harry; it defines
the content domain and although i cannot test them on everything, it
gives me some assurance that they would have made an effort to master
all the material on the list.

i have not used this approach in an applied context but as Harry notes,
it is really just a variant of providing a reading list; albeit somewhat
more specific and narrow.

like Harry, i don't see any reason why structurally this approach shld
result in exam psychometric properties that are inherently different
from other approaches to defining the exam content domain. likewise, i
don't think the resulting exam will be any more g-loaded than other
knowledge-based exams.

curious -- you did not explicitly state so but i assume the service is
generally pleased w/ the approach? so, what is the push-back or
concern? and form what source?


- winfred

On 6/28/2011 2:56 PM, Mark Hammer wrote:

> I used to use something like that for teaching as well. I'd list all

> the concepts I wanted students to know, and terms I wanted them to be

> familiar with, and inform them that this was the master list, from which

> I would draw questions, hopefully packing as much of that into a

> predefined number of questions as I could. (In some instances, the

> distractors also require you to understand the concept, just so you can

> classify it as a distractor and invalid response, such that one question

> can actually address much more than just the material in the correct

> response.)


> The advantage it provides to testees is that it defines what they

> *don't* have to worry about when preparing, and they appreciate that.

> Assuming the range of information covered off by the list of stems is of

> an appropriate size compared to the final list of questions (which in

> this case it seems to be, 5:1), it will be perceived as fair by testees,

> and also gives you some confidence that they've spent some time learning

> about things that maybe you'd like them to know, but don't have

> time/space to test them on.


> The caveat is that you'd want some assurances that any given subset of

> 100 was roughly equal in both difficulty and predictive value. Y

> certainly don't want to find yourself with mean scores that fluctuate

> from session/class to session/class. And that entails a bit of work on

> your part.


> Mark Hammer

> Ottawa


>>>> <Harry.Brull at pdininthhouse.com> 2011/06/28 2:44 PM>>>

> This would seem to me as a variant on providing a reading list as a

> study guide. In this case, the jurisdiction is directing the

> candidates

> to the issues (items) that are considered important.

> I have used a comparable technique in my collegiate teaching.

> As for what to call it - beyond an inspired idea - I'd suggest

> "item-based exam preparation/study guide".


> As for the psychometric properties, I see no reason to expect anything

> different than ant content-based multiple choice test - high

> coefficient

> alpha.

> As for what you are really measuring, some combination of job

> knowledge

> and a healthy does of "G".


> Harry Brull

> H.A.R.

> PDI Ninth House



> ------------------------------------------------

> Harry Brull | Senior Vice-President

> PDI Ninth House

> Global Leadership Solutions


> 1.612.337.8233 office

> 1.612.414.8998 mobile

> 1.612.337.3695 fax

> Harry.Brull at PDINinthHouse.com


> 33 South Sixth Street

> Suite 4900

> Minneapolis, MN 55402


> www.pdininthhouse.com


> -----Original Message-----

> From: ipac-list-bounces at ipacweb.org

> [mailto:ipac-list-bounces at ipacweb.org] On Behalf Of Morris, Ramona

> (JUS)

> Sent: Tuesday, June 28, 2011 1:32 PM

> To: ipac-list at ipacweb.org

> Subject: [IPAC-List] promotional exam strategy


> Situation:


> A police service is using a test on their local policy and procedures

> as

> part of the promotional process. Candidates are provided with the

> pool

> of items (say 500)...just the stems without the distractors. The test

> consists of a subset of multiple choice items(say 100).


> The argument is that they want candidates to know all of the p and p,

> and it is up to the candidate to prepare themselves. And, exam

> security

> becomes less of a headache.


> Questions:


> 1. What is this strategy called?


> 2. Do you have any advice about its use?


> 3. What do we know about the strategy (e.g. validity, reliability

> efficiency etc.)




> Ramona Morris


> Ontario Police College


> Aylmer, Ontario



> _______________________________________________________

> IPAC-List

> IPAC-List at ipacweb.org

> http://www.ipacweb.org/mailman/listinfo/ipac-list


> Confidentiality Notice: All information in this communication,

> including any files or attachments, is intended for the sole use of the

> individual or entity to which it is addressed and may contain

> information that is confidential, proprietary and/or trade secret

> information entitled to protection and/or exempt from disclosure under

> applicable law. If you are not the intended recipient, you are hereby

> notified that any dissemination, distribution, or copying of this

> communication is strictly prohibited. If you have received this

> communication in error, please notify the sender by return email and

> delete this communication from your system. Thank you for your

> cooperation.

> _______________________________________________________

> IPAC-List

> IPAC-List at ipacweb.org

> http://www.ipacweb.org/mailman/listinfo/ipac-list


> This e-mail message is intended for the named recipient(s) and may

> contain information that is privileged, confidential and/or exempt from

> disclosure under applicable law. Unauthorized disclosure, copying or

> re-transmission is prohibited. If you are not a named recipient or not

> authorized by the named recipient(s), or if you have received this

> e-mail in error, then please notify the sender immediately and delete

> the message and any copies.

> >

> Ce courriel est destiné exclusivement au destinataire mentionné en titre

> et peut contenir de l'information privilégiée, confidentielle ou

> soustraite à la communication aux termes des lois applicables. Toute

> divulgation non autorisée, toute reproduction ou réacheminement est

> interdit. Si vous n'êtes pas le destinataire de ce courriel, ou n'êtes

> pas autorisé par le destinataire visé, ou encore, si vous l'avez reçu

> par erreur, veuillez le mentionner immédiatement à l'expéditeur et

> supprimer le courriel et les copies.


> _______________________________________________________

> IPAC-List

> IPAC-List at ipacweb.org

> http://www.ipacweb.org/mailman/listinfo/ipac-list

More information about the IPAC-List mailing list