[IPAC-List] Measurement education

[Shekerjian, Rene]

I think it is very easy to spot the obvious failures of tests. And as Dennis pointed out, failures there will be. But that doesn't make the process a bad investment. And it doesn't make the people who spot the obvious failures testing experts. I can spot some obvious failures in my car, but I am unable to successfully design a car. (If I designed a car, it would correct the one or two deficiencies that really bug me and have hundreds of others that I couldn't anticipate.)

It is fairly easy to explain the smart people who do well on tests but are unproductive or uncooperative; intelligence is not the same thing as conscientiousness, integrity, organizational ability, or the ability to interact effectively with others.

It is harder to explain the effective people who would make great hires but do poorly on our tests. But I did gain some insight from a manager the other day. He noted that one such person would have made a great manager because he knew how to delegate and keep a group humming along. Unfortunately, the test did not tap that ability sufficiently to award that person the points he needed to be among the top scorers. Apparently this person was weak in the other areas or had a bad day.

Your example may be tougher to explain, but it may be that the person who believes your assessment failed gets along easily and well with the candidate and others do not have the same experience.

René 

René Shekerjian | Testing Services Division | NYS Department of Civil Service | 
===============================================================================


-----Original Message-----
From: Partain, Steven C. [mailto:Steven.Partain at tvfr.com] 
Sent: Thursday, October 25, 2012 10:27 AM
To: ipac-list at ipacweb.org
Subject: Re: [IPAC-List] Measurement education

Hi Gwen.  I feel fortunate to work for a department that emphasizes soft skills like problem analysis over knowledge components.  We rarely use MC tests anymore, especially for officer exams.  We use pretty involved assessment centers with multiple simulations that have high fidelity with actual work.  Still, the critics blame the test when someone they believe is effective in interpersonal communication but does poorly on an exam component in which that competency is measured.  It begs the question, in my eyes, of why we're wasting our time and resources on these testing components if folks are so certain about what they believe to be true of individuals.  Using some sort of "promotability index" as Harry mentions that measures others' perceptions of candidates would involve far less resources.
Steven Partain
HR Manager
Human Resources
Tualatin Valley Fire & Rescue
11945 SW 70th Avenue, Tigard, Oregon 97223
www.tvfr.com<http://www.tvfr.com/>
Ph. 503-259-1292

From: GWEN SCHINDLER [mailto:gschindler at comp.state.md.us]
Sent: Thursday, October 25, 2012 5:34 AM
To: ipac-list at ipacweb.org; Partain, Steven C.
Subject: Re: [IPAC-List] Measurement education

Hello Steven,

What does your promotional examination assess?  Is it a measure of cognitive ability or job knowledge only?  Or does it also include assessment of other leadership components, writing ability, a behavior description interview, and/or some other assessment of soft skills such as communication, interpersonal skill and motivation?





Gwen Schindler, Team Leader, Recruitment and Testing
Comptroller of Maryland
Office of Human Resources
Phone: 410-260-6622
Fax: 410-974-5249


>>> "Partain, Steven C." <Steven.Partain at tvfr.com<mailto:Steven.Partain at tvfr.com>> 10/24/2012 7:26 PM >>>

Folks, I am facing a bit of a crisis of understanding related to measurement in promotional exams.  Every-so-often we have a promotional exam in which the names and ranking of eligibles on the list don't match what our folks know about those candidates.  We've had several recently with the "best" people failing the exams.  As you might imagine, HR is to blame, and we are under great pressure to change our approach to exams to ensure the "best" people pass and are appropriately ranked.  I won't go through all the practices we use to ensure validity, reliability, standardization, etc.  We certainly are always looking at those factors and have room to improve.  But the underlying message is to make our exams "more successful," which means that the resulting eligible list matches the perceptions of our workforce about their true ability.

So, here's my question.  I feel pretty well-versed in the folly of holistic assessments, the relatively low validity of others "sizing up" candidates intuitively, etc.  I have attempted-and obviously failed-to convey some of the science underlying this.  How have others successfully overcome this challenge?  Are there metaphors that have worked?  A written piece published that captures the issue in laymen's terms?

Any help is appreciated.  Otherwise, I fear we will head down the road of having the workforce rank candidates-kind of a popularity contest.

Thanks,

Steven Partain
HR Manager
Human Resources
Tualatin Valley Fire & Rescue
11945 SW 70th Avenue, Tigard, Oregon 97223
www.tvfr.com<http://www.tvfr.com/>
Ph. 503-259-1292

_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org<mailto:IPAC-List at ipacweb.org>
http://www.ipacweb.org/mailman/listinfo/ipac-list
This email and any file transmitted with it may be confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you received this email in error, please notify the Comptroller's System Manager by forwarding this message to postmaster at comp.state.md.us<mailto:postmaster at comp.state.md.us>