[IPAC-List] Angoff method of setting pass points - how many minimally competent candidates pass at a given pass point?

[Shekerjian, Rene]

Thank you Lance. I was out last week on vacation.  Perhaps some of my trouble comes from not understanding the definition of "minimally qualified." I was interpreting this similarly to minimum qualifications, meaning that the person is qualified to do the job. But it sounds as if that is "clearly qualified."

 

What I find fascinating about this is trying to interpret how this would be manifested in actuality. The idea of minimally qualified that you have shared with me requires a new look at this.

 

Thanks again. If you have any other insights to share with me, I would love to hear them.

René 

René Shekerjian  |  Testing Services Division  |  NYS Department of Civil Service  |  518-473-9937 
=================================================================================== 

 

-----Original Message-----
From: Lance Seberhagen [mailto:sebe at erols.com] 
Sent: Monday, July 20, 2009 12:13 PM
To: Shekerjian, Rene
Cc: ipac-list at ipacweb.org
Subject: Re: [IPAC-List] Angoff method of setting pass points - how manyminimally competent candidates pass at a given pass point?

 

Rene, you are essentially correct, but there is more to the story.  There will always be some error around the cutoff score, no matter what method you use to set the cutoff score, depending on the skill distribution and the reliability and validity of the test.  For example, if you used a 100-item test, rather than a 10-item test, you would obtain a more reliable and precise measure of each applicant, including those around the cutoff score.  Also, don't forget that the Angoff method focuses on "minimally qualified" applicants who are borderline by definition, as opposed to "clearly unqualified" and "clearly qualified" applicants.  The Angoff method isn't perfect but is a professionally accepted way to set cutoff scores on the basis of expert judgment, consistent with content validation methodology.  You could do a lot worse with other methods.

Lance Seberhagen, Ph.D.
Seberhagen & Associates
9021 Trailridge Ct
Vienna, VA 22182
Tel 703-790-0796
www.seberhagen.com 



Shekerjian, Rene wrote:



I am trying to understand how the Angoff method of setting passpoints affects passing rates. I am not sure if my reasoning is correct, and I was hoping for to get some help from this august body.
 
Roughly, my understanding is that the Angoff method has SMEs indicate the probability that a minimally competent person would get each item correct. SMEs' item-probability ratings are averaged, and the averages are summed across the items in the test to arrive at the passpoint.
 
Suppose that for all items in a ten-item test, the average rating is .5 This would mean that a minimally competent candidate would have a 50% chance of getting each item right. If this is true, and all 10 items are perfectly inter-correlated, then half the minimally competent candidates should have a score of 0 and half of them should have a score of 10. As the item intercorrelations decrease, the scores distribution (of minimally competent candidates) should have more candidates at scores in between 0 and 10. 
 
It seems to me that if you set the pass point at 5 in this example, which the Angoff method would lead you to do, typically about half the minimally competent candidate would pass and about half would fail. 
 
Is that considered acceptable? Or are we obligated to pass all minimally competent candidates? Or at least avoid knowingly failing minimally competent candidates?
 
I will appreciate any help anyone offers.
 
Thanks, 
René