[IPAC-List] Guessing on the SAT/ACT

[Oscar Spurlin]

Another angle on the guessing issue is the constructs being measured.  If
you are measuring math or have questions where there is only one possible
answer with no shades of gray, then it is reasonable to conclude that
guessing is a compensation for actually ability or knowledge, and a penalty
might compensate.  However with so many employment or promotional tests we
are asking questions about human relations or situational judgment in which
there are definite shades of gray in responses.  Having subjects skip
questions almost always would mean losing meaningful variance with an
expected impact on reliability and validity.

Oscar Spurlin
Ergometrics

-----Original Message-----
From: ipac-list-bounces at ipacweb.org [mailto:ipac-list-bounces at ipacweb.org]
On Behalf Of Shekerjian, Rene
Sent: Wednesday, May 05, 2010 11:46 AM
To: IPAC-List at ipacweb.org
Subject: Re: [IPAC-List] Guessing on the SAT/ACT

If the better candidates can eliminate two choices, then they are down to  a
50-50 chance of guessing right. If I understand correction for guessing
correctly, then they should lose as many points as they make incorrect
choices (if the scoring is one point per question). So CFG actually could
underpenalize the better candidates.

But it seems that CFG was challenged on the possibility that candidates who
get many items wrong did not guess on any of them and their correct
responses were all earned, as their incorrect responses were earned, so to
speak. In which case the correction is more of an undeserved penalty.

René

René Shekerjian | Testing Services Division | NYS Department of Civil
Service | 518-474-3778
===========================================================================

-----Original Message-----
From: David Friedland [mailto:david at FriedlandHR.com] 
Sent: Monday, May 03, 2010 3:55 PM
To: IPAC-List at ipacweb.org
Subject: Re: [IPAC-List] Guessing on the SAT/ACT

Kelly raises important points. In the case of highly speeded tests, in the
absence of a correction for guessing it would be a good strategy to blindly
guess on remaining questions when time is up on the test. In that case the
individual is guessing randomly. The correction in that case accomplishes
what it is designed to do. However, in the case of multiple choice questions
the correction for guessing overcorrects, and it is to the advantage of the
test taker to guess when at least one of the distracters can be eliminated.
Since the correction over-corrects it is likely to result in some slight
reduction of validity by removing some true variance as well as error
variance. 

The issue of guessing and correction for guessing may be more complicated
than is recognized by simple corrections for guessing. As Kelly points out,
more sophisticated test-takers may have a better idea of when it is best to
guess. I published an article some years ago exploring a strategy for giving
credit for answers based on the degree of confidence, by having test-takers
divide a set number of points among the choices to reflect their degree of
confidence in the correctness of each of the choices. Scores were calculated
in several ways to determine whether validity could be increased by using
this technique. One of the scores used in the study was arrived at by
partialing out level of knowledge to arrive at a score reflecting the
tendency to express certainty in one's answers regardless of knowledge.
Interestingly I found that the tendency to express certainty predicted
supervisory ratings for police officers. The tendency to express certainty
may reflect personality variance that influences the tendency to guess. This
result made some intuitive sense for the police officer job because of the
need for police to project confidence and authority. It might work
differently for other jobs. However, I believe this result suggests that
guessing behavior may be influenced by personality. 


David Friedland


Friedland & Associates, Inc.
Phone: (310) 204-0045
Fax:   (310) 836-1823
Email: David at FriedlandHR.com
Web:   www.FriedlandHR.com
 
This email message is for the sole use of the intended recipient(s) and may
contain confidential and privileged information. If you are not the intended
recipient, please contact the sender by reply email and destroy all copies
of the original message.

   

-----Original Message-----
From: ipac-list-bounces at ipacweb.org [mailto:ipac-list-bounces at ipacweb.org]
On Behalf Of Kelly Sorensen
Sent: Monday, May 03, 2010 6:53 AM
To: IPAC-List at ipacweb.org
Subject: Re: [IPAC-List] Guessing on the SAT/ACT

I want to chime in here because there is a lot of conflicting information
regarding guessing on the SAT, with advice ranging from "never guess" to
"always guess."

First off, people talk about there being a "penalty for guessing" on the
SAT, which isn't exactly true. Basically, ETS is simply trying to prevent
folks from getting more points than they deserve based simply upon guessing,
so it's more of a "correction for guessing" rather than a "penalty." And
maybe I'm splitting hairs, here, but the term "penalty" is much more
intimidating than the term "correction."

On the SAT students generally have 5 multiple choice answers to choose from.
ETS takes off 1/4 a point for each wrong answer. Yet for each correct
answer, the student earns one full point. And, of course, -1/4 + -1/4 + -1/4
+ -1/4 = -1.
And -1 + 1 = 0. And 0 points are what the student would have received had
they not guessed. This is my objection to calling it a guessing "penalty."
This is important for students to know, but often test prep books and even
high school English teacher will tell students not to guess, to guess only
if they can narrow the answer choices down to two, etc. The reason this is
important to know is because students rarely guess at random. As I'm sure
you all know, it is difficult to come up with attractive distractor items.
Students generally can eliminate an answer or two quite easily. There are
also strategies that can be taught to help help eliminate answers (e.g., ETS
doesn't like extreme answers, they aren't going to have an answer choice the
would be insulting to woman or minorities be the correct answer, if the word
is extremely difficult and the student is on problem one in the sentence
completion section then it isn't going to be the answer, or if the word is
extremely easy and it's the last question in that section it isn't going to
be the answer, etc.), so it's rare when a student cannot eliminate any
answer possibilities, even if they don't know why they think an answer
choice should be eliminated.

And of course the SAT is controversial because minorities as a group tend to
score less well...

The ACT does not correct for guessing, so there is no debate on guessing
there, or shouldn't be. One should always guess on the ACT, though of
course, in my opinion, one should always guess on the SAT too.

On Thu, Apr 29, 2010 at 1:54 PM, Chris Hornick <cwhornick at cwhms.com> wrote:


> I have found this to be a very interesting discussion, as my 15 year old

> son

> is in the process of being scarred for life by taking the SAT, ACT and

> other

> placement exams.  I have tried to help him understand how guessing can

> affect his score, and I suspect it has about as much value to him as a lot

> of the life experience help I try to offer.  I am sure many of you know

> what

> I am saying there!  My real comment here is that I think we should be

clear

> on what we are trying to accomplish with correcting for guessing on

> employment exams (speeded or not speeded).  I don't think it is all that

> critical or helpful in the employment arena.  In my estimation, correction

> for guessing on a speeded test is not necessary or helpful.  As Dennis

> pointed out, most speeded tests include calculation of both accuracy and

> speed, thus penalizing applicants further by correcting for guessing does

> not make sense to me.

_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org
http://www.ipacweb.org/mailman/listinfo/ipac-list


_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org
http://www.ipacweb.org/mailman/listinfo/ipac-list
No virus found in this incoming message.
Checked by AVG - www.avg.com 
Version: 9.0.814 / Virus Database: 271.1.1/2855 - Release Date: 05/04/10
23:26:00