[IPAC-List] Threatening a Penalty for Guessing

[Mark Hammer]

Here's a different angle that may (or may not) shed some light.

I used "answer justifications" on student exams for a number of years.  Under this rubric, students can provide a 1-2 sentence written justification/explanation for why they selected the response they did.  If the scoring key declares them wrong but they show some thought in what they selected (and some students read FAR too much into questions/answers sometimes), they can get partial or full credit for the item, depending on whether what they wrote provides evidence of understanding of the subject matter.  It adds a couple of hours to the grading of a large class, but then so does arguing about grades with keeners who feel cheated because they read too much into the question and picked the "wrong" answer for reasons they feel were legitimate.

My experience with this was that maybe 10% of students used this option for maybe 10% of the questions.  Much of the time, they were correct anyway (i.e., the justification was a complete waste of their time, and merely a reflection of their anxiety or indecision).  Much of the time, they were WAAAAAAYYYYYY off, such that credit was only garnered occasionally.

Why do I mention this?  Chiefly because many folks cannot tell the difference between when they are guessing blindly, and when they are hopelessly off but hold a sincere belief they are at least close.  So if you TELL people they are going to be "penalized for guessing", will they be in any position to consistently *know* when they are guessing?  Notwithstanding all the research which indicates that "feeling of knowing" is often reasonably correlated with correct recognition/recall, my guess is rarely.

I think it also bears considering that the effects of such instructions may have differential impact on those test-takers who are more, and less confident and prepared.  If I'm not a strong contender, and feel that way, then the benefits of guessing may well be seen as outweighing the potential risks.  If I'm competent and feel that way (i.e., I have something to actually risk - my success), then I may be more risk averse and decline any questions that feel like a guess.  There is also the metacognitive argument that those who are more competent are generally better judges of the accuracy of their knowledge, such that people at the bottom half of the distribution are more likely to be affected by such penalties.

I think an elegant solution to this conundrum was offered in an article I posted about recently, regarding use of 5 choice questions, where 2 correct choices per item were required for the item to be correct.  The authors contend that this reduces the guessing component.  Requiring 2 of 5 available choices reduces the correct-by-chance element from 1-in-4 to 1-in-20. (Remember that this is choice without replacement, so there are 5 choices initially, and then only 4 available once you pick your first choice.)

Having said all of that, I think Joel makes an excellent point in distinguishing between the advice given, and whether or not there is any quantitative adjustment for guessing by the tester.

Mark Hammer
Ottawa