[IPAC-List] Reliability for T&Es

[GWEN SCHINDLER]

Hi Bryan, 

Many moons ago I attended an IPMAAC sponsored workshop on T&E development presented by Nancy Abrams. In that workshop, she covered many different types of T&E's, two of which were straight point value T&E's and banded T&E's. From what you are saying, it seems like you are developing straight point value T&E's and wanting to convert scores to bands. In Maryland, we are also required to band scores and report only bands. I have found it most helpful to develop the T&E from the perspective of the band up front. So that means defining broad categories for Best Qualified, Better Qualified and Qualified, by identifying which types and combinations of education, certification and experience distinguish among the pool of applicants for the job in question. I have found that SME's are able to contribute to the development of a banded T&E in a more meaningful way based upon their knowledge of the job and what it takes to perform optimally. And I believe that rating a banded T&E is a lot easier than a straight point value T&E, and will produce better inter rater reliability right from the get go. I'll be interested to hear what others think about this. 


Gwen Schindler, Manager, Recruitment and Testing
Comptroller of Maryland 
Office of Personnel Services 
Phone: 410-260-6622
Fax: 410-974-5249

>>> "Bryan Baldwin" 06/22/10 11:31 AM >>> 

Thanks for the responses. To be honest, I am the beginning of investigating this issue, and the type of T&Es I'm thinking of are primarily task-based. For those of us in a civil service system where banding is mandatory, the increasing use of T&Es begs the question of how these bands should be formed. I suspect in many cases it's done simply by dividing the raw spread by the number of ranks. To speak to Dennis' question, I have also suspected the bands should be quite large, particularly at the top given inflation tendencies. Of course the pass point obviously plays an issue here, and in my experience SMEs tend to want to set a higher pass point than is really necessary, which plays into the size of the bands. 

Dan, I appreciate your questions. As for how I'm defining error, I'm slightly more concerned about internal consistency than I am about test-retest, but in reality most concerned about the accuracy of ratings. What makes this even more complicated is the quality of the T&E items and scales themselves vary dramatically, much more so than your typical m-c item. 

To answer your second question, a single overall score is being generated. I have zero faith in the ability of these exams to measure separate distinct constructs. 

BB 


>>> 6/22/2010 7:06 AM >>> 

Hi Bryan, 

A few questions.... 

1. How are you defining error for your T&E measure? 

a. As inconsistency across items? In other words, do you want to draw 
inferences with regard to the consistency in your T&E scores if they were 
based on different sampling of items? 

b. As inconsistency across occasions? In other words, do you want to draw 
inferences with regard to the consistency in your T&E scores if respondents 
completed your measure on a different occasion? 

c. Are you concerned about both types of inconsistency referenced above? 


2. What is the substantive nature of your T&E measure? 

a. Are you trying to assess multiple, distinct constructs with your T&E 
measure (e.g., each construct indicated by a subset of items comprising the 
T&E measure)? 

b. Or, do you simply have an overarching, heterogenous measure of T&E? 

Depending on how you define error (which is a function of the inferences 
you want to make with regard to the consistency of your scores), and the 
substantive nature of your T&E measure, it will dictate the structure of 
the reliability coefficient that is most appropriate for your situation. 

For a general discussion of this, see: 
Putka, D. J. & Sackett, P .R. (2010). Reliability and validity. In J.L. 
Farr, & N.T. Tippins (Eds.). Handbook of Employee Selection (pp. 9-49). New 
York: Routledge. 


Hope this helps, 

Dan 


******************************************************************* 
Dan J. Putka, Ph.D. 
Principal Staff Scientist 
Personnel Selection & Development Program 
Human Resources Research Organization (HumRRO) 
66 Canal Center Plaza, Suite 700 
Alexandria, VA 22314-1591 
703-706-5640 (o) 
703-548-5574 (f) 
http://www.humrro.org 
******************************************************************* 




From:"Bryan Baldwin" 
To:ipac-list at ipacweb.org 
Date:06/21/2010 07:05 PM 
Subject:[IPAC-List] Reliability for T&Es 
Sent by:ipac-list-bounces at ipacweb.org 



Can anyone point me to a source for how to compute reliability for T&Es 
(assuming it can be done)? Ultimate goal is to use this to create bands. 

Thanks- 

Bryan Baldwin 
Staff Services Manager II 
California Department of Justice 
Division of Administrative Support 
Personnel Programs 
(916) 322-5446 


CONFIDENTIALITY NOTICE: This communication with its contents may contain 
confidential and/or legally privileged information. It is solely for the 
use of the intended recipient(s). Unauthorized interception, review, use or 
disclosure is prohibited and may violate applicable laws including the 
Electronic Communications Privacy Act. If you are not the intended 
recipient, please contact the sender and destroy all copies of the 
communication. 
_______________________________________________________ 
IPAC-List 
IPAC-List at ipacweb.org 
http://www.ipacweb.org/mailman/listinfo/ipac-list 




CONFIDENTIALITY NOTICE: This communication with its contents may contain confidential and/or legally privileged information. It is solely for the use of the intended recipient(s). Unauthorized interception, review, use or disclosure is prohibited and may violate applicable laws including the Electronic Communications Privacy Act. If you are not the intended recipient, please contact the sender and destroy all copies of the communication. 
_______________________________________________________ 
IPAC-List 
IPAC-List at ipacweb.org 
http://www.ipacweb.org/mailman/listinfo/ipac-list

------------------------------------------------------------------------------
This email and any file transmitted with it may be confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you received this email in error, please notify the Comptroller's System Manager by forwarding this message to postmaster at comp.state.md.us
==============================================================================