[IPAC-List] Reliability for T&Es

[Mark Hammer]

Dan raises some excellent points.
 
My sense is that one of the principle challenges to effective use of T&E 
is that of assuring that multiple observers are able to confirm the same
signal in very noisy conditions.  The noise is exacerbated in high-volume
recruitment activities where the range of ways in which a particular 
type and amount of training targeted for job X could come in a multitude
of forms, shapes and sizes.  A single officer cannot be assigned to 
plowing through 1578 applications files without any expectation of drift, 
and even if they are exquisitely consistent, are they correct in their
assessment of whether the applicant truly meets the min-quals?
 
So the real practical challenge is to be able to ascertain, and assure, 
that if you set a handful of officers at the task of going through a large 
pile of applicants, they judge that noise, and identify the signal within, 
applying the same criteria.
 
Certainly there are concerns about the functional equivalent of test-retest, 
for repeat applicants, but my gut tells me the primary concern is for a 
clean and dependable sifting of wheat from chaff in large applicant pools
the first time around.
 
Mark Hammer
Ottawa


>>> <dputka at humrro.org> 2010/06/22 10:06 AM >>>

Hi Bryan,

A few questions....

1. How are you defining error for your T&E measure?

a. As inconsistency across items?   In other words, do you want to draw
inferences with regard to the consistency in your T&E scores if they were
based on different sampling of items?

b. As inconsistency across occasions?  In other words, do you want to draw
inferences with regard to the consistency in your T&E scores if respondents
completed your measure on a different occasion?

c. Are you concerned about both types of inconsistency referenced above?


2. What is the substantive nature of your T&E measure?

a. Are you trying to assess multiple, distinct constructs with your T&E
measure (e.g., each construct indicated by a subset of items comprising the
T&E measure)?

b. Or, do you simply have an overarching, heterogenous measure of T&E?

Depending on how you define error (which is a function of the inferences
you want to make with regard to the consistency of your scores), and the
substantive nature of your T&E measure, it will dictate the structure of
the reliability coefficient that is most appropriate for your situation.

For a general discussion of this, see:
Putka, D. J. & Sackett, P .R. (2010).  Reliability and validity. In J.L.
Farr, & N.T. Tippins (Eds.). Handbook of Employee Selection (pp. 9-49). New
York: Routledge.


Hope this helps,

Dan


*******************************************************************
Dan J. Putka, Ph.D.
Principal Staff Scientist
Personnel Selection & Development Program
Human Resources Research Organization (HumRRO)
66 Canal Center Plaza, Suite 700
Alexandria, VA  22314-1591
703-706-5640 (o)
703-548-5574 (f)
http://www.humrro.org
*******************************************************************




From:"Bryan Baldwin" <Bryan.Baldwin at doj.ca.gov>
To:ipac-list at ipacweb.org
Date:06/21/2010 07:05 PM
Subject:[IPAC-List] Reliability for T&Es
Sent by:ipac-list-bounces at ipacweb.org



Can anyone point me to a source for how to compute reliability for T&Es
(assuming it can be done)?  Ultimate goal is to use this to create bands.

Thanks-

Bryan Baldwin
Staff Services Manager II
California Department of Justice
Division of Administrative Support
Personnel Programs
(916) 322-5446


CONFIDENTIALITY NOTICE: This communication with its contents may contain
confidential and/or legally privileged information. It is solely for the
use of the intended recipient(s). Unauthorized interception, review, use or
disclosure is prohibited and may violate applicable laws including the
Electronic Communications Privacy Act. If you are not the intended
recipient, please contact the sender and destroy all copies of the
communication.
_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org
http://www.ipacweb.org/mailman/listinfo/ipac-list


_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org
http://www.ipacweb.org/mailman/listinfo/ipac-list


This e-mail message is intended for the named recipient(s) and 
may contain information that is privileged, confidential and/or 
exempt from disclosure under applicable law. Unauthorized 
disclosure, copying or re-transmission is prohibited. If you are not 
a named recipient or not authorized by the named recipient(s), or 
if you have received this e-mail in error, then please notify the 
sender immediately and delete the message and any copies.
 
Ce courriel est destine exclusivement au destinataire mentionne 
en titre et peut contenir de l'information privilegiee, confidentielle 
ou soustraite a la communication aux termes des lois 
applicables. Toute divulgation non autorisee, toute reproduction 
ou reacheminement est interdit. Si vous n'etes pas le 
destinataire de ce courriel, ou n'etes pas autorise par le 
destinataire vise, ou encore, si vous l'avez recu par erreur, 
veuillez le mentionner immediatement a l'expediteur et supprimer 
le courriel et les copies.