[IPAC-List] Threatening a Penalty for Guessing

[Doverspike,Dennis]

And Mark is correct, you do need people to guess to accurately look at item bias and other item level psychometric phenomena. This does lead to the problem of what to do with items at the end of the test, since people often do not reach those items and you get a lot of nonresponders or nonguessers. And guessing itself may represent a form of bias, especially with penalty instructions, since it may be related to gender and ethnicity.

Dennis D

________________________________________
From: ipac-list-bounces at ipacweb.org [ipac-list-bounces at ipacweb.org] On Behalf Of Mark Hammer [Mark.Hammer at psc-cfp.gc.ca]
Sent: Thursday, April 29, 2010 2:09 PM
To: IPAC-List at ipacweb.org
Subject: Re: [IPAC-List] Threatening a Penalty for Guessing

To respond to Chris's comments.  In fact, the original research I drew from (stumbled onto in the journal "Teaching of Psychology") noted that students did not necessarily do better when the answer justification option was offered, but they did think of the exam as significantly fairer.  My experience in using it was no different.  Of course, in the employment testing or entrance-exam situation, one is not obliged to earn and maintain the buy-in of the unsuccessful the way one is in courses that have several exams distributed over the semester.  "If I do it for you, then I would have to do it for everyone" is a tremendously unsatisfying argument to students, or any other individual on the receiving end of a less than stellar result.  The advantage it offered me was that by allowing people an opportunity to provide evidence at the time, rather than post hoc. of what they believed the question or response alternative to be, I actually *could* do it for them without having to do
  it for everyone since I had incontrovertible evidence of the exam that they believed they were taking.  I can imagine circumstances where use of this option imposes undue hardship on the instructor, but in my instance it worked as well as promised.

But back to penalizing for guessing.  Ruminating over Chris' remarks got me to thinking that in the evaluation of test bias, and other aspects of item performance, we actually NEED people to take honest wild stabs at items, rather than hold off cautiously, since that very response forms the datum of our item analysis, doesn't it?  Moreover, if there are new items being piloted for whether they function as intended, you need every single response to that item you can lay your hands on.  If the phrasing of the item has enough of a hint of ambiguity about it that leads people to skip it for fear of penalization, you can cheat yourself out of acquiring sufficient data to assess the item.

And finally, you're only penalized for guessing if you guess wrong.  If you happen, for reasons of dumb luck rather than deep insight, to select the correct answer while guessing. you DON'T get penalized.  Hence the use of the phrase "may be penalized", in the absence of its companion phrase "...although who knows, you might get lucky".

Mark Hammer
Ottawa



_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org
http://www.ipacweb.org/mailman/listinfo/ipac-list