[IPAC-List] promotional exam strategy

[Morris, Ramona (JUS)]

The service is pleased so far.  We conduct a provincial promotional test on provincial and federal law, management, leadership, community policing etc. 
We try to balance definitional, procedural and problem solving questions.  We have a fairly detailed study guide, but perceived relevance to their day to day jobs is not as high as we might like. 

Perhaps this would also help with our ongoing cat and mouse game with exam security by alleviating anxiety????

Ramona
x4417

-----Original Message-----
From: ipac-list-bounces at ipacweb.org [mailto:ipac-list-bounces at ipacweb.org] On Behalf Of Winfred Arthur, Jr.
Sent: June 28, 2011 4:21 PM
To: ipac-list at ipacweb.org
Subject: Re: [IPAC-List] promotional exam strategy

just echoing Mark and Harry's comments.  i currently use a similar 
approach in my undergraduate rsch methods, graduate personnel selection, 
and graduate advanced personnel selection and placement courses.  
however, i provide a concepts and principles list (about 120) but not 
item stems.

the obvious advantages are as articulated by Mark and Harry; it defines 
the content domain and although i cannot test them on everything, it 
gives me some assurance that they would have made an effort to master 
all the material on the list.

i have not used this approach in an applied context but as Harry notes, 
it is really just a variant of providing a reading list; albeit somewhat 
more specific and narrow.

like Harry, i don't see any reason why structurally this approach shld 
result in exam psychometric properties that are inherently different 
from other approaches to defining the exam content domain.  likewise, i 
don't think the resulting exam will be any more g-loaded than other 
knowledge-based exams.

curious -- you did not explicitly state so but i assume the service is 
generally pleased w/ the approach?  so, what is the push-back or 
concern?  and form what source?

thanks.

- winfred


On 6/28/2011 2:56 PM, Mark Hammer wrote:

> I used to use something like that for teaching as well.  I'd list all

> the concepts I wanted students to know, and terms I wanted them to be

> familiar with, and inform them that this was the master list, from which

> I would draw questions, hopefully packing as much of that into a

> predefined number of questions as I could.  (In some instances, the

> distractors also require you to understand the concept, just so you can

> classify it as a distractor and invalid response, such that one question

> can actually address much more than just the material in the correct

> response.)

>

> The advantage it provides to testees is that it defines what they

> *don't* have to worry about when preparing, and they appreciate that.

> Assuming the range of information covered off by the list of stems is of

> an appropriate size compared to the final list of questions (which in

> this case it seems to be, 5:1), it will be perceived as fair by testees,

> and also gives you some confidence that they've spent some time learning

> about things that maybe you'd like them to know, but don't have

> time/space to test them on.

>

> The caveat is that you'd want some assurances that any given subset of

> 100 was roughly equal in both difficulty and predictive value.  Y

> certainly don't want to find yourself with mean scores that fluctuate

> from session/class to session/class.  And that entails a bit of work on

> your part.

>

> Mark Hammer

> Ottawa

>

>>>> <Harry.Brull at pdininthhouse.com>  2011/06/28 2:44 PM>>>

> This would seem to me as a variant on providing a reading list as a

> study guide. In this case, the jurisdiction is directing the

> candidates

> to the issues (items) that are considered important.

> I have used a comparable technique in my collegiate teaching.

> As for what to call it - beyond an inspired idea - I'd suggest

> "item-based exam preparation/study guide".

>

> As for the psychometric properties, I see no reason to expect anything

> different than ant content-based multiple choice test - high

> coefficient

> alpha.

> As for what you are really measuring, some combination of job

> knowledge

> and a healthy does of "G".

>

> Harry Brull

> H.A.R.

> PDI Ninth House

>

>

> ------------------------------------------------

> Harry Brull |  Senior Vice-President

> PDI Ninth House

> Global Leadership Solutions

>

> 1.612.337.8233  office

> 1.612.414.8998  mobile

> 1.612.337.3695  fax

> Harry.Brull at PDINinthHouse.com

>

> 33 South Sixth Street

> Suite 4900

> Minneapolis, MN 55402

>

> www.pdininthhouse.com

>

> -----Original Message-----

> From: ipac-list-bounces at ipacweb.org

> [mailto:ipac-list-bounces at ipacweb.org] On Behalf Of Morris, Ramona

> (JUS)

> Sent: Tuesday, June 28, 2011 1:32 PM

> To: ipac-list at ipacweb.org

> Subject: [IPAC-List] promotional exam strategy

>

> Situation:

>

> A police service is using a test on their local policy and procedures

> as

> part of the promotional process.  Candidates are provided with the

> pool

> of items (say 500)...just the stems without the distractors. The test

> consists of a subset of multiple choice items(say 100).

>

> The argument is that they want candidates to know all of the p and p,

> and it is up to the candidate to prepare themselves. And, exam

> security

> becomes less of a headache.

>

> Questions:

>

> 1.    What is this strategy called?

>

> 2.    Do you have any advice about its use?

>

> 3.    What do we know about the strategy (e.g. validity, reliability

> efficiency etc.)

>

>

>

> Ramona Morris

>

> Ontario Police College

>

> Aylmer, Ontario

>

>

> _______________________________________________________

> IPAC-List

> IPAC-List at ipacweb.org

> http://www.ipacweb.org/mailman/listinfo/ipac-list

>

> Confidentiality Notice: All information in this communication,

> including any files or attachments, is intended for the sole use of the

> individual or entity to which it is addressed and may contain

> information that is confidential, proprietary and/or trade secret

> information entitled to protection and/or exempt from disclosure under

> applicable law. If you are not the intended recipient, you are hereby

> notified that any dissemination, distribution, or copying of this

> communication is strictly prohibited. If you have received this

> communication in error, please notify the sender by return email and

> delete this communication from your system. Thank you for your

> cooperation.

> _______________________________________________________

> IPAC-List

> IPAC-List at ipacweb.org

> http://www.ipacweb.org/mailman/listinfo/ipac-list

>

> This e-mail message is intended for the named recipient(s) and may

> contain information that is privileged, confidential and/or exempt from

> disclosure under applicable law. Unauthorized disclosure, copying or

> re-transmission is prohibited. If you are not a named recipient or not

> authorized by the named recipient(s), or if you have received this

> e-mail in error, then please notify the sender immediately and delete

> the message and any copies.

>   >

> Ce courriel est destiné exclusivement au destinataire mentionné en titre

> et peut contenir de l'information privilégiée, confidentielle ou

> soustraite à la communication aux termes des lois applicables. Toute

> divulgation non autorisée, toute reproduction ou réacheminement est

> interdit. Si vous n'êtes pas le destinataire de ce courriel, ou n'êtes

> pas autorisé par le destinataire visé, ou encore, si vous l'avez reçu

> par erreur, veuillez le mentionner immédiatement à l'expéditeur et

> supprimer le courriel et les copies.

>

> _______________________________________________________

> IPAC-List

> IPAC-List at ipacweb.org

> http://www.ipacweb.org/mailman/listinfo/ipac-list

_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org
http://www.ipacweb.org/mailman/listinfo/ipac-list