[IPAC-List] Measurement education

[Dennis Doverspike]

Steven,

I realize this will be of absolutely no help, but I find that even many
expert measurement and selection people way overestimate the efficiency of
assessment. That is not to say that tests do not have utility, they clearly
do, or that a good test is not better than a bad test, it clearly is, but
the real value of tests comes in making lots of decisions about lots of
people and at the organizational level (I have an article I wrote up for
IPAC a number of years ago on this topic, although it helps to make my
point and not yours).

In the artificial, but instructive case of a dichotomous predictor and a
dichotomous criterion, the efficiency of tests is given by the
Taylor-Russell tables. I believe it is very useful to review those tables,
because they reveal that in many real world cases the use of assessments
will result in many correct decisions, but still many wrong decisions.  For
example, with a correlation coefficient of .50, and a base rate and
selection ratio of.50, the percentage of successful people is only 67%;
that is good, but probably much lower than most people would estimate.
 With a more typical correlation of .25, the percentage of successful
people would be .58, which would mean there are still a lot of unsuccessful
people.

My point, although tests are better than the alternative, they are not
perfect and do result in a great many more incorrect decisions than we
would probably expect, especially from the perspective of the individual
test taker.

Two additional caveats from many years of experience with promotional
testing.  Many times the "best" people do not want the promotion.  Thus,
especially if you are talking about failing the test, they fail because
they do not study, do not prepare, or are not prepared.  They fail because
they are not motivated to obtain a good score or the promotion. Other
times, the best people may simply be too busy, with life or with their job,
to prepare for the exam.  Not sure that argument will help you either, but
I believe it is a reality of promotional testing.

Dennis

On Wed, Oct 24, 2012 at 7:26 PM, Partain, Steven C. <Steven.Partain at tvfr.com

> wrote:



> Folks, I am facing a bit of a crisis of understanding related to

> measurement in promotional exams.  Every-so-often we have a promotional

> exam in which the names and ranking of eligibles on the list don't match

> what our folks know about those candidates.  We've had several recently

> with the "best" people failing the exams.  As you might imagine, HR is to

> blame, and we are under great pressure to change our approach to exams to

> ensure the "best" people pass and are appropriately ranked.  I won't go

> through all the practices we use to ensure validity, reliability,

> standardization, etc.  We certainly are always looking at those factors and

> have room to improve.  But the underlying message is to make our exams

> "more successful," which means that the resulting eligible list matches the

> perceptions of our workforce about their true ability.

>

> So, here's my question.  I feel pretty well-versed in the folly of

> holistic assessments, the relatively low validity of others "sizing up"

> candidates intuitively, etc.  I have attempted-and obviously failed-to

> convey some of the science underlying this.  How have others successfully

> overcome this challenge?  Are there metaphors that have worked?  A written

> piece published that captures the issue in laymen's terms?

>

> Any help is appreciated.  Otherwise, I fear we will head down the road of

> having the workforce rank candidates-kind of a popularity contest.

>

> Thanks,

>

> Steven Partain

> HR Manager

> Human Resources

> Tualatin Valley Fire & Rescue

> 11945 SW 70th Avenue, Tigard, Oregon 97223

> www.tvfr.com<http://www.tvfr.com/>

> Ph. 503-259-1292

>

> _______________________________________________________

> IPAC-List

> IPAC-List at ipacweb.org

> http://www.ipacweb.org/mailman/listinfo/ipac-list

>




-- 
Dennis Doverspike, PhD., ABPP
Licensed Psychologist, #3539 (OHIO)
Independent Consultant
Professor of Psychology, University of Akron
dennisdoverspike at gmail.com

The information is intended only for the person or entity to which it is
addressed and may contain confidential, privileged and/or a work product
for the sole use of the intended recipient. No confidentiality or privilege
is waived or lost by any errant transmission. If you receive this message
in error, please destroy all copies of it and notify the sender. If the
reader of this message is not the intended recipient, you are hereby
notified that any dissemination, distribution or copying of this
communication is strictly prohibited. In the case of E-mail or electronic
transmission, immediately delete it and all copies of it from your system
and notify the sender. E-mail and fax transmission cannot be guaranteed to
be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or contain viruses.