[IPAC-List] Threatening a Penalty for Guessing

[Jeff Feuquay]

I'm late getting in on this, but the perception of procedural justice  
to which Richard referred struck a nerve. Kristine Smith put together  
a great session for WRIPAC & PTC-NC on the subject. I have been  
preaching (fear mongering) for years on lawsuit avoidance - don't  
surprise people and remember you're always playing to an audience,  
i.e., if the candidate sees the process as fair, you're in good shape.  
If not, you're next audience is their family & friends, then a  
plaintiff's atty, then a judge & jury. Perceived justice is a big deal.

Jeff

On Apr 29, 2010, at 1:36 PM, "Richard Arwood" <richard.arwood at comcast.net 
 > wrote:


> I concur with the comments made by Chris (and Mark), and I will give  

> an

> additional reason for "testing appeals" that goes beyond the realms of

> psychological and practical; it also has a "legal claim avoidance"  

> aspect

> (i.e. due process) in which the employer hopes to dissuade a  

> candidate from

> filing a lawsuit due to a perception that the test was "unfair."   

> When a

> perception of "unfairness" exists within a diverse set of employee

> candidates, then "Katie bar the door", and get ready for those  

> fascinating

> depositions!

>

> I know such comments make us all sick to our stomach, but the testing

> appeals process in Memphis was almost entirely for legal rather than  

> for

> other "real" reasons.  I love being retired now.

>

> ..................RBA

>

> =======================================

> Richard Arwood, Fire Chief (retired -Memphis, TN)

> Collierville, TN

>

> Join with us at:         http://www.iracing.com/

>

> =======================================

>

> -----Original Message-----

> From: ipac-list-bounces at ipacweb.org [mailto:ipac-list- 

> bounces at ipacweb.org]

> On Behalf Of Chris Hornick

> Sent: Thursday, April 29, 2010 12:55 PM

> To: 'Mark Hammer'; IPAC-List at ipacweb.org

> Subject: Re: [IPAC-List] Threatening a Penalty for Guessing

>

> I have found this to be a very interesting discussion, as my 15 year  

> old son

> is in the process of being scarred for life by taking the SAT, ACT  

> and other

> placement exams.  I have tried to help him understand how guessing can

> affect his score, and I suspect it has about as much value to him as  

> a lot

> of the life experience help I try to offer.  I am sure many of you  

> know what

> I am saying there!  My real comment here is that I think we should  

> be clear

> on what we are trying to accomplish with correcting for guessing on

> employment exams (speeded or not speeded).  I don't think it is all  

> that

> critical or helpful in the employment arena.  In my estimation,  

> correction

> for guessing on a speeded test is not necessary or helpful.  As Dennis

> pointed out, most speeded tests include calculation of both accuracy  

> and

> speed, thus penalizing applicants further by correcting for guessing  

> does

> not make sense to me.

>

> I understand the statistical reasons for correction for guessing on  

> such

> college placement and "education qualification" exams, but I am  

> struggling

> with why we are concerned with this issue in regard to employment  

> testing.

>

> My confusion is trying to understand why people are concerned with  

> this

> issue of accuracy in true test score in an employment setting.  Most  

> written

> exams used for selection in my experience (most frequently in hiring  

> and

> promotion tests for public safety positions) are used as only a  

> portion of

> the overall hiring decision.  They are not used as a point estimate of

> ability or knowledge as in the case of the ACT and SAT.  I was  

> struck by

> Mark's comments about students "justifying" their answer choices in  

> his

> classes.  I found it humorous and consistent with much of what I  

> have seen

> in promotional exams where candidates "challenge" or "appeal" items  

> on their

> job knowledge exams.  The same behaviors described by Mark are  

> replete in

> the challenges we see in public safety promotional exams.  In many  

> cases in

> my experience, their "appeal" is an attempt to justify their choice,  

> which

> is very frequently the correct selection according to the original key

> anyway (or it is an amusing demonstration of their lack of command  

> of the

> reading material).  I think the value in allowing "appeals" in  

> promotional

> exams is much more psychological than real (as Mark described about  

> his

> students).  Candidates feel like they are being heard and considered  

> (which

> they are) and are more accepting of the final decision.  The  

> "appeals" do

> offer some humor to those of us who read and investigate their  

> efficacy.

> There is some fun in our work!

>

> In all the exams each of us has taken, we have guessed when we were  

> unsure

> about our choice, so we all have personal experience in this arena.   

> I have

> been more selective in making guesses when there is a penalty for  

> guessing,

> based upon my understanding of how guessing affected my final score.

> Telling me it "may" versus "will" affect my score is dishonest, in my

> opinion.  I need to know if it will or won't affect my score, so  

> that I can

> adjust my approach.

>

> I don't believe in an employment setting that the impact of this  

> concern is

> that critical.  Yes, it can impact whether a candidate or applicant  

> "passes"

> at the cutoff score, but it has relatively little usefulness or impact

> anywhere else in the distribution of scores.  A difference of a  

> point or 2

> correction for guessing for most candidates or applicants does  

> little to

> help us make a hiring or promotion decision.  And it has a very  

> small impact

> in most cases on the final ranking of candidates.  Given the error  

> in our

> measurements in employment tests for a host of other reasons, I  

> would not

> recommend penalties for guessing.

>

>

>

>

> Chris W. Hornick, Ph.D.

> President

> CWH Research, Inc.

> 9360 Teddy Lane, Suite 203

> Lone Tree, CO 80124

>

> Office: (303) 617-3433

> Cell:    (303) 810-3645

> -----Original Message-----

> From: ipac-list-bounces at ipacweb.org [mailto:ipac-list- 

> bounces at ipacweb.org]

> On Behalf Of Mark Hammer

> Sent: Tuesday, April 27, 2010 10:15 AM

> To: IPAC-List at ipacweb.org

> Subject: Re: [IPAC-List] Threatening a Penalty for Guessing

>

> Here's a different angle that may (or may not) shed some light.

>

> I used "answer justifications" on student exams for a number of years.

> Under this rubric, students can provide a 1-2 sentence written

> justification/explanation for why they selected the response they  

> did.  If

> the scoring key declares them wrong but they show some thought in  

> what they

> selected (and some students read FAR too much into questions/answers

> sometimes), they can get partial or full credit for the item,  

> depending on

> whether what they wrote provides evidence of understanding of the  

> subject

> matter.  It adds a couple of hours to the grading of a large class,  

> but then

> so does arguing about grades with keeners who feel cheated because  

> they read

> too much into the question and picked the "wrong" answer for reasons  

> they

> feel were legitimate.

>

> My experience with this was that maybe 10% of students used this  

> option for

> maybe 10% of the questions.  Much of the time, they were correct  

> anyway

> (i.e., the justification was a complete waste of their time, and  

> merely a

> reflection of their anxiety or indecision).  Much of the time, they  

> were

> WAAAAAAYYYYYY off, such that credit was only garnered occasionally.

>

> Why do I mention this?  Chiefly because many folks cannot tell the

> difference between when they are guessing blindly, and when they are

> hopelessly off but hold a sincere belief they are at least close.   

> So if you

> TELL people they are going to be "penalized for guessing", will they  

> be in

> any position to consistently *know* when they are guessing?   

> Notwithstanding

> all the research which indicates that "feeling of knowing" is often

> reasonably correlated with correct recognition/recall, my guess is  

> rarely.

>

> I think it also bears considering that the effects of such  

> instructions may

> have differential impact on those test-takers who are more, and less

> confident and prepared.  If I'm not a strong contender, and feel  

> that way,

> then the benefits of guessing may well be seen as outweighing the  

> potential

> risks.  If I'm competent and feel that way (i.e., I have something to

> actually risk - my success), then I may be more risk averse and  

> decline any

> questions that feel like a guess.  There is also the metacognitive  

> argument

> that those who are more competent are generally better judges of the

> accuracy of their knowledge, such that people at the bottom half of  

> the

> distribution are more likely to be affected by such penalties.

>

> I think an elegant solution to this conundrum was offered in an  

> article I

> posted about recently, regarding use of 5 choice questions, where 2  

> correct

> choices per item were required for the item to be correct.  The  

> authors

> contend that this reduces the guessing component.  Requiring 2 of 5

> available choices reduces the correct-by-chance element from 1-in-4 to

> 1-in-20. (Remember that this is choice without replacement, so there  

> are 5

> choices initially, and then only 4 available once you pick your first

> choice.)

>

> Having said all of that, I think Joel makes an excellent point in

> distinguishing between the advice given, and whether or not there is  

> any

> quantitative adjustment for guessing by the tester.

>

> Mark Hammer

> Ottawa

>

> _______________________________________________________

> IPAC-List

> IPAC-List at ipacweb.org

> http://www.ipacweb.org/mailman/listinfo/ipac-list

>

> __________ Information from ESET NOD32 Antivirus, version of virus  

> signature

> database 5065 (20100427) __________

>

> The message was checked by ESET NOD32 Antivirus.

>

> http://www.eset.com

>

>

>

>

> __________ Information from ESET NOD32 Antivirus, version of virus  

> signature

> database 5072 (20100429) __________

>

> The message was checked by ESET NOD32 Antivirus.

>

> http://www.eset.com

>

>

> _______________________________________________________

> IPAC-List

> IPAC-List at ipacweb.org

> http://www.ipacweb.org/mailman/listinfo/ipac-list

>

> _______________________________________________________

> IPAC-List

> IPAC-List at ipacweb.org

> http://www.ipacweb.org/mailman/listinfo/ipac-list