[IPAC-List] Measurement education

[David Friedland]

There have been many excellent ideas presented in this thread. I would like
to present my two cents as well. I believe that the problem as presented by
Steven is a very common one, in which the people in the department involved
have very definite opinions about the candidates competing in the
examination. The issues involved are complex but very familiar to us, as
indicated by the interest expressed in this discussion. 

1. Civil service examinations are not designed or intended to be individual
assessments. They are intended to be valid in the sense that they are
intended to improve our true positives and minimize false positives. There
is always a degree of error to be expected. 

2. Impressions of supervisors and managers of who is the best prepared for
promotion or who is the best performer are individual assessments made
without the benefit of formal assessment. These impressions are often not
correct, and may be influenced by subjective perceptions based on factors
that may have nothing to do with performance, such as likability. 

3. When examinations are prepared by HR without the involvement of people in
the appointing authority there is a built-in resistance to accepting
outcomes that disagree with their expectations of who should come out on
top. We can't assume that the examination is flawed if it doesn't produce
results that match the pre-conceived expectations of the appointing
authority. 

4. Perceptions of supervisors and managers are not always accurate. Examples
from my own experience include:

     a. As a result of the outcome of a legal case, more than 30 employees
were moved from civil service exempt jobs to civil service jobs. They were
then required to compete in an examination to retain their jobs. They were
unable to    score high enough on the valid examination to be reachable on
the list and retain their jobs. Their supervisor testified in trial that
they were highly qualified for their jobs and were excellent workers. After
they were replaced by individuals who placed higher on the list, their
supervisor was struck by the fact that their replacements were much more
knowledgeable and better workers than the ones who were replaced. 

     b. In a manufacturing company a sample of 150 quality control employees
in a glass manufacturing company participated in a controlled study
employing an exact replication of their production line inspection job using
a known sample of glassware with known flaws mixed in randomly with
glassware with no flaws. The employees who supervisors believed were the
most accurate inspectors were not the best performers in the simulation. 

A good approach, though there may be no ultimate solution, is to involve
individuals from the department involved in designing the examination
process. Joel Wiesen alluded to this in his post. This approach may create
security challenges, but may have some success in increasing acceptance. 

At the root of this problem is a difference in approach. Supervisors and
managers are making subjective individual assessments concerning who they
believe is best qualified, while the civil service examination process is a
more objective group assessment that may not result in the same ranking as
the informal individual assessment of the supervisors and managers. We may
be able to optimize our assessment processes, but we may always have to live
with some tension between these two approaches. 

David Friedland


David Friedland, Ph.D., SPHR
Friedland & Associates, Inc.
Ph.	(310) 204-0045
Cell	(310) 699-5881
David at FriedlandHR.com