[IPAC-List] How leery should I be?

[GAST, ILENE F]

Kevin is raising a number of really interesting issues.  

First and foremost, we always have to be concerned with "validity," regardless of what we measure.  However, in this situation content related validity evidence is most critical. 

In this situation, maybe a "test" is not the logical first step.  
The manager's analysis --- 

"He *SAYS* the test is only to assess current skill, and if all field technicians have superb skill, then he will be less inclined to push for a "lead" position"

---suggests that skills gap analysis might be what's needed here.  The job analysis that would precede the skills gap study would provide content validity evidence for any subsequent assessment efforts.

A lot of the performance management systems I've seen are way too general (and have to be to generalize across jobs) to give this manager the information he needs.  Also, shifting the focus from "TESTING" to "employee development" might help with the morale issue.  

Ilene


Ilene Gast
Personnel Research & Assessment Division
U.S Customs & Border Protection
-----Original Message-----
From: ipac-list-bounces at ipacweb.org [mailto:ipac-list-bounces at ipacweb.org] On Behalf Of Shekerjian, Rene
Sent: Friday, February 18, 2011 9:23 AM
To: Jason Bowling; IPAC-List at ipacweb.org
Subject: Re: [IPAC-List] How leery should I be?

I can echo what has been said, maybe adding to some sort of consensus. Non-practitioners do not understand what goes on under the surface of a measurement tool, and a little knowledge can be quite dangerous, as it may lead a person to think that they understand more than they do. I would not want to be in a situation where I could be impacted by the selection practices of someone who did not have the knowledge or ability to create and administer a test that met professional standards. 

It reminds me of my past life as a remodeler. Folks thought that if what was done looked pretty good from the outside it must be "good enough." But I have seen many instances where the work of incompetent "professionals" and overconfident do-it-yourselfers lost its good looks long before it should have because what lay underneath was not sufficient. Think sagging walls, loose tiles, leaky roofs, doors that don't close, etc. 

René 

René Shekerjian | Testing Services Division | NYS Department of Civil Service  
=============================================================================

-----Original Message-----
From: Jason Bowling [mailto:jbowling at ssvec.com] 
Sent: Thursday, February 17, 2011 6:29 PM
To: IPAC-List at ipacweb.org
Subject: [IPAC-List] How leery should I be?

All,

Kevin's question regarding the delineation between tests needing validation and other instruments perhaps not needing it reminded me of an issue that has me a bit concerned.

So, the question would be: when does a test become a selection pressure?

An engineering manager has a team of 5 field staking technicians. All perform nearly identical work with any significant variation only a side-effect of the geographic areas they serve.

The engineering manager is considering the need for a "lead" field staking technician to provide closer oversight to the group.

As part of the decision making process for this, the manager would like to assess the knowledge, skills, and abilities of his current staff of technicians. His plan to accomplish this is to develop a test (paper? Practical? Unknown currently). He *SAYS* the test is only to assess current skill, and if all field technicians have superb skill, then he will be less inclined to push for a "lead" position... (note: only less inclined; he may still want it.)

So, I suppose my concerns with this are:

1.      If we do open a "lead" position, then I am certain he will want to use the test results to help determine to whom the job should be awarded. (I know I certainly would.) However, he is not necessarily looking for a validated instrument to assess their skill at this point. In fact he doesn't really want me to go through the process of ensuring the validity of the test he develops.
2.      I also have a hard time with the question: Why are we not assessing KSAs at the employee performance appraisal? That is, shouldn't we already know employee KSAs (arguably better) based on their work performance than based on the results of an artificially created test environment?
3.      The department already suffers a bit of a morale issue. I don't imagine a bit of surprise test anxiety would be helpful in that regard.

Am I overly concerned or right to point out the potential for red flags?

I appreciate any input.

Best regards,

Jason




_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org
http://www.ipacweb.org/mailman/listinfo/ipac-list