[IPAC-List] Angoff method of setting pass points - how many minimally competent candidates pass at a given pass point?

[Shekerjian, Rene]

Thank you Dennis. I was there at your MAPAC session. It was very interesting and exciting. Work and life have been busy and I forgot all about it. 

Just as an aside, according to my notes, you suggested rephrasing the Angoff question to be "What percent of the people who get this right will be minimally competent?" Certainly food for thought.


René

René Shekerjian  |  Testing Services Division  |  NYS Department of Civil Service  |  518-473-9937
===================================================================================




-----Original Message-----
From: Dennis Doverspike [mailto:dd1 at uakron.edu] 
Sent: Monday, July 20, 2009 4:38 PM
To: Shekerjian, Rene; ipac-list at ipacweb.org
Subject: RE: [IPAC-List] Angoff method of setting pass points - how many minimally competent candidates pass at a given pass point?

Rene,

Based on classical psychometrics or test theory, the correlation between
items for the minimally competent group will be zero. I know that does not
change the main point of your example, but wanted to offer that
clarification.

Second, and as pointed out by Lance, any procedure for setting cutoffs will
result in correct decisions and also incorrect decisions. This has long been
recognized and the impact of such decisions is given by the Taylor-Russell
tables. 

To show why this is critical, you ask "Is that considered acceptable? Or are
we obligated to pass all minimally competent candidates? Or at least avoid
knowingly failing minimally competent candidates?" Those are good questions,
however an equally valid question would be - what percent of applicants who
pass the test are not competent? Certainly, if I am a patient going to a
physician, I may care very little as to whether minimally competent persons
did not pass their certification exams (unless there are a shortage of
physicians). I am much more concerned with what percentage of doctors
passing the certification exam might not be competent.

Anyway, it is for this reason at the recent SIOP and MAPAC we did
presentations arguing that organizations should consider along with Angoff
data traditional Taylor Russell information. In addition, an excellent book
is Standard Setting authored by Cizek and Bunch. They discuss a number of
relevant cutoff setting procedures including the Hofstee and Beuk methods.

Dennis Doverspike, Ph.D., ABPP
Professor of Psychology
Director, Center for Organizational Research
Senior Fellow of the Institute for Life-Span Development and Gerontology
Psychology Department
University of Akron
Akron, Ohio  44325-4301
330-972-8372 (Office)
330-972-5174 (Office Fax)
ddoverspike at uakron.edu

The information is intended only for the person or entity to which it is
addressed and may contain confidential, privileged and/or a work product for
the sole use of the intended recipient.  No confidentiality or privilege is
waived or lost by any errant transmission. If you receive this message in
error, please destroy all copies of it and notify the sender. If the reader
of this message is not the intended recipient, you are hereby notified that
any dissemination, distribution or copying of this communication is strictly
prohibited. In the case of E-mail or electronic transmission, immediately
delete it and all copies of it from your system and notify the sender.
E-mail and fax transmission cannot be guaranteed to be secure or error-free
as information could be intercepted, corrupted, lost, destroyed, arrive late
or incomplete, or contain viruses. 


-----Original Message-----
From: ipac-list-bounces at ipacweb.org [mailto:ipac-list-bounces at ipacweb.org]
On Behalf Of Shekerjian, Rene
Sent: Thursday, July 16, 2009 4:40 PM
To: ipac-list at ipacweb.org
Subject: [IPAC-List] Angoff method of setting pass points - how
manyminimally competent candidates pass at a given pass point?

I am trying to understand how the Angoff method of setting passpoints
affects passing rates. I am not sure if my reasoning is correct, and I was
hoping for to get some help from this august body.

Roughly, my understanding is that the Angoff method has SMEs indicate the
probability that a minimally competent person would get each item correct.
SMEs' item-probability ratings are averaged, and the averages are summed
across the items in the test to arrive at the passpoint.

Suppose that for all items in a ten-item test, the average rating is .5 This
would mean that a minimally competent candidate would have a 50% chance of
getting each item right. If this is true, and all 10 items are perfectly
inter-correlated, then half the minimally competent candidates should have a
score of 0 and half of them should have a score of 10. As the item
intercorrelations decrease, the scores distribution (of minimally competent
candidates) should have more candidates at scores in between 0 and 10. 

It seems to me that if you set the pass point at 5 in this example, which
the Angoff method would lead you to do, typically about half the minimally
competent candidate would pass and about half would fail. 

Is that considered acceptable? Or are we obligated to pass all minimally
competent candidates? Or at least avoid knowingly failing minimally
competent candidates?

I will appreciate any help anyone offers.

Thanks, 
René





_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org
http://www.ipacweb.org/mailman/listinfo/ipac-list