[IPAC-List] Threatening a Penalty for Guessing

[Chris Hornick]

I have found this to be a very interesting discussion, as my 15 year old son
is in the process of being scarred for life by taking the SAT, ACT and other
placement exams.  I have tried to help him understand how guessing can
affect his score, and I suspect it has about as much value to him as a lot
of the life experience help I try to offer.  I am sure many of you know what
I am saying there!  My real comment here is that I think we should be clear
on what we are trying to accomplish with correcting for guessing on
employment exams (speeded or not speeded).  I don't think it is all that
critical or helpful in the employment arena.  In my estimation, correction
for guessing on a speeded test is not necessary or helpful.  As Dennis
pointed out, most speeded tests include calculation of both accuracy and
speed, thus penalizing applicants further by correcting for guessing does
not make sense to me.

I understand the statistical reasons for correction for guessing on such
college placement and "education qualification" exams, but I am struggling
with why we are concerned with this issue in regard to employment testing.

My confusion is trying to understand why people are concerned with this
issue of accuracy in true test score in an employment setting.  Most written
exams used for selection in my experience (most frequently in hiring and
promotion tests for public safety positions) are used as only a portion of
the overall hiring decision.  They are not used as a point estimate of
ability or knowledge as in the case of the ACT and SAT.  I was struck by
Mark's comments about students "justifying" their answer choices in his
classes.  I found it humorous and consistent with much of what I have seen
in promotional exams where candidates "challenge" or "appeal" items on their
job knowledge exams.  The same behaviors described by Mark are replete in
the challenges we see in public safety promotional exams.  In many cases in
my experience, their "appeal" is an attempt to justify their choice, which
is very frequently the correct selection according to the original key
anyway (or it is an amusing demonstration of their lack of command of the
reading material).  I think the value in allowing "appeals" in promotional
exams is much more psychological than real (as Mark described about his
students).  Candidates feel like they are being heard and considered (which
they are) and are more accepting of the final decision.  The "appeals" do
offer some humor to those of us who read and investigate their efficacy.
There is some fun in our work!

In all the exams each of us has taken, we have guessed when we were unsure
about our choice, so we all have personal experience in this arena.  I have
been more selective in making guesses when there is a penalty for guessing,
based upon my understanding of how guessing affected my final score.
Telling me it "may" versus "will" affect my score is dishonest, in my
opinion.  I need to know if it will or won't affect my score, so that I can
adjust my approach.  

I don't believe in an employment setting that the impact of this concern is
that critical.  Yes, it can impact whether a candidate or applicant "passes"
at the cutoff score, but it has relatively little usefulness or impact
anywhere else in the distribution of scores.  A difference of a point or 2
correction for guessing for most candidates or applicants does little to
help us make a hiring or promotion decision.  And it has a very small impact
in most cases on the final ranking of candidates.  Given the error in our
measurements in employment tests for a host of other reasons, I would not
recommend penalties for guessing.




Chris W. Hornick, Ph.D.
President
CWH Research, Inc.
9360 Teddy Lane, Suite 203
Lone Tree, CO 80124
 
Office: (303) 617-3433
Cell:    (303) 810-3645
-----Original Message-----
From: ipac-list-bounces at ipacweb.org [mailto:ipac-list-bounces at ipacweb.org]
On Behalf Of Mark Hammer
Sent: Tuesday, April 27, 2010 10:15 AM
To: IPAC-List at ipacweb.org
Subject: Re: [IPAC-List] Threatening a Penalty for Guessing

Here's a different angle that may (or may not) shed some light.

I used "answer justifications" on student exams for a number of years.
Under this rubric, students can provide a 1-2 sentence written
justification/explanation for why they selected the response they did.  If
the scoring key declares them wrong but they show some thought in what they
selected (and some students read FAR too much into questions/answers
sometimes), they can get partial or full credit for the item, depending on
whether what they wrote provides evidence of understanding of the subject
matter.  It adds a couple of hours to the grading of a large class, but then
so does arguing about grades with keeners who feel cheated because they read
too much into the question and picked the "wrong" answer for reasons they
feel were legitimate.

My experience with this was that maybe 10% of students used this option for
maybe 10% of the questions.  Much of the time, they were correct anyway
(i.e., the justification was a complete waste of their time, and merely a
reflection of their anxiety or indecision).  Much of the time, they were
WAAAAAAYYYYYY off, such that credit was only garnered occasionally.

Why do I mention this?  Chiefly because many folks cannot tell the
difference between when they are guessing blindly, and when they are
hopelessly off but hold a sincere belief they are at least close.  So if you
TELL people they are going to be "penalized for guessing", will they be in
any position to consistently *know* when they are guessing?  Notwithstanding
all the research which indicates that "feeling of knowing" is often
reasonably correlated with correct recognition/recall, my guess is rarely.

I think it also bears considering that the effects of such instructions may
have differential impact on those test-takers who are more, and less
confident and prepared.  If I'm not a strong contender, and feel that way,
then the benefits of guessing may well be seen as outweighing the potential
risks.  If I'm competent and feel that way (i.e., I have something to
actually risk - my success), then I may be more risk averse and decline any
questions that feel like a guess.  There is also the metacognitive argument
that those who are more competent are generally better judges of the
accuracy of their knowledge, such that people at the bottom half of the
distribution are more likely to be affected by such penalties.

I think an elegant solution to this conundrum was offered in an article I
posted about recently, regarding use of 5 choice questions, where 2 correct
choices per item were required for the item to be correct.  The authors
contend that this reduces the guessing component.  Requiring 2 of 5
available choices reduces the correct-by-chance element from 1-in-4 to
1-in-20. (Remember that this is choice without replacement, so there are 5
choices initially, and then only 4 available once you pick your first
choice.)

Having said all of that, I think Joel makes an excellent point in
distinguishing between the advice given, and whether or not there is any
quantitative adjustment for guessing by the tester.

Mark Hammer
Ottawa

_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org
http://www.ipacweb.org/mailman/listinfo/ipac-list

__________ Information from ESET NOD32 Antivirus, version of virus signature
database 5065 (20100427) __________

The message was checked by ESET NOD32 Antivirus.

http://www.eset.com


 

__________ Information from ESET NOD32 Antivirus, version of virus signature
database 5072 (20100429) __________

The message was checked by ESET NOD32 Antivirus.

http://www.eset.com