[IPAC-List] Threatening a Penalty for Guessing

[Lance Seberhagen]

Here's a quick reference that's pretty good.

Frary, Bob (1988). Formula scoring of multiple-choice tests (correction 
for guessing).  NCME, Instructional Topics in Educational Measurement, 
Module 4.  http://www.ncme.org/pubs/items/ITEMS_Mod_4.pdf

Lance Seberhagen, Ph.D.
Seberhagen&  Associates
9021 Trailridge Ct
Vienna, VA 22182
Tel 703-790-0796
www.seberhagen.com


On 4/27/2010 10:43 PM, Winfred Arthur, Jr. wrote:

> no, Joel, sorry for being unclear but i did not mean to suggest or 

> imply that it is standard practice to correct both speeded and 

> non-speeded tests for guessing/accuracy.  however, to echo Dennis' 

> point, we do not correct our knowledge tests for guessing -- this is 

> primarily b/c there is not a strong conceptual case or basis for doing 

> so.  [although, i can envisage some domains in which errors and lack 

> of knowledge might be so dangerous that you might want to penalize 

> test takers for getting an answer wrong!!  :)  ]

>

> anyway, if you are interested in some reading on this, you might want 

> to look at the "scoring of information-processing tests" section (pp. 

> 60-61) of:

>

> Arthur, W. Jr., Doverspike, D., & Bell, S. T. (2004).  Information 

> processing tests.  In M. Hersen, & J. C. Thomas (Eds.), /Comprehensive 

> Handbook of Psychological Assessment: Volume 4, Industrial and 

> Organizational Assessment/ (pp. 56-74).  NY: John Wiley & Sons, Inc.

>

> it discusses the different ways in which tests (like your clerical 

> speed test?) could be scored, namely, (1) # of correct responses 

> [accuracy], (2) response time [speed], or (3) some weighted composite 

> of accuracy and speed using specified algorithms.

>

> - winfred

>

> On 4/27/2010 11:13 AM, Joel Wiesen wrote:

>> Thanks for your thoughtful comments, Winfred.

>>

>> Do I understand you correctly as saying it is standard practice to 

>> correct both speeded and non-speeded test for guessing, but that you 

>> do not do so for your (non-speeded) knowledge tests?

>>

>> If so, why do you not correct for guessing?

>>

>> Thx.

>>

>> Joel

>>

>>

>>

>> - -

>> Joel P. Wiesen, Ph.D., Director

>> Applied Personnel Research

>> 62 Candlewood Road

>> Scarsdale, NY 10583-6040

>> (617) 244-8859

>> http://appliedpersonnelresearch.com

>>

>>

>>

>>

>> Note: This e-mail and any attachments may contain confidential and/or 

>> legally privileged information. Please do not forward any contents 

>> without permission. If you have received this message in error please 

>> destroy all copies, completely remove it from your computer, and 

>> notify the sender. Thank you.

>>

>>

>>

>> Winfred Arthur, Jr. wrote:

>>> Joel, some general thoughts are embedded below:

>>>

>>> On 4/27/2010 8:19 AM, Joel Wiesen wrote:

>>>> If a test's instructions say "you may be penalized for questions you

>>>> mark incorrectly" and then the grading does not correct for guessing,

>>>> what might the effect be?

>>> well, my first thought is that the use of "may" makes this 

>>> problematic.  it shld state explicitly whether one is going to do so 

>>> or not.  "may" without specifying the conditions under which this 

>>> will or will not be invoked seems to me to be a recipe for . . . 

>>> well, problems!

>>>

>>> and whereas i have not seen any empirical rsch or data on this, the 

>>> college board uses this instruction set for some sections of the SAT 

>>> [they do not use "may"; they use "will"] and it is my impression 

>>> that students are more likely to leave these items blank than guess 

>>> when they do not know the answer.

>>>

>>>>

>>>> Has anyone had practical experience with such instructions? Do test 

>>>> takers pay attention to such instructions?

>>> not personally.  indeed to the contrary, i use an instruction set 

>>> that states that there is no penalty for guessing and so it is in 

>>> one's best interest to guess if one does not know the answer.  

>>> subsequently, i rarely get any non-responses.  of course, these are 

>>> knowledge tests.

>>>>

>>>> Is there research on this type of ambiguous ("may be penalized") 

>>>> test instruction?

>>>>

>>> not that i am aware of; but then i have not done a lit search either.

>>>> (This particular instruction was used on a speeded (clerical speed)

>>>> portion of a longer test for a craft type job.)

>>> for what it is worth, it is common, if not stand practice to correct 

>>> these types of tests for accuracy as well.

>>>

>>> hope this is somewhat useful.

>>>

>>> thanks.

>>>

>>> - winfred

> _______________________________________________________

> IPAC-List

> IPAC-List at ipacweb.org

> http://www.ipacweb.org/mailman/listinfo/ipac-list

>

>

>