[IPAC-List] Reliability for T&Es

[Dennis Doverspike]

Bryan

Your response is very interesting. What do you mean when you say that "where
banding is mandatory." How is banding mandatory? Do you mean it is mandatory
by state law or by some type of local rule? Is the type or nature of the
banding specified?

Dennis Doverspike, Ph.D., ABPP
Professor of Psychology
Director, Center for Organizational Research
Senior Fellow of the Institute for Life-Span Development and Gerontology
Psychology Department
University of Akron
Akron, Ohio  44325-4301
330-972-8372 (Office)
330-972-5174 (Office Fax)
ddoverspike at uakron.edu

The information is intended only for the person or entity to which it is
addressed and may contain confidential, privileged and/or a work product for
the sole use of the intended recipient.  No confidentiality or privilege is
waived or lost by any errant transmission. If you receive this message in
error, please destroy all copies of it and notify the sender. If the reader
of this message is not the intended recipient, you are hereby notified that
any dissemination, distribution or copying of this communication is strictly
prohibited. In the case of E-mail or electronic transmission, immediately
delete it and all copies of it from your system and notify the sender.
E-mail and fax transmission cannot be guaranteed to be secure or error-free
as information could be intercepted, corrupted, lost, destroyed, arrive late
or incomplete, or contain viruses. 


-----Original Message-----
From: ipac-list-bounces at ipacweb.org [mailto:ipac-list-bounces at ipacweb.org]
On Behalf Of Bryan Baldwin
Sent: Tuesday, June 22, 2010 11:28 AM
To: dputka at humrro.org
Cc: ipac-list at ipacweb.org
Subject: Re: [IPAC-List] Reliability for T&Es

Thanks for the responses.  To be honest, I am the beginning of investigating
this issue, and the type of T&Es I'm thinking of are primarily task-based.
For those of us in a civil service system where banding is mandatory, the
increasing use of T&Es begs the question of how these bands should be
formed.  I suspect in many cases it's done simply by dividing the raw spread
by the number of ranks.  To speak to Dennis' question, I have also suspected
the bands should be quite large, particularly at the top given inflation
tendencies.  Of course the pass point obviously plays an issue here, and in
my experience SMEs tend to want to set a higher pass point than is really
necessary, which plays into the size of the bands.
 
Dan, I appreciate your questions.  As for how I'm defining error, I'm
slightly more concerned about internal consistency than I am about
test-retest, but in reality most concerned about the accuracy of ratings.
What makes this even more complicated is the quality of the T&E items and
scales themselves vary dramatically, much more so than your typical m-c
item.
 
To answer your second question, a single overall score is being generated.
I have zero faith in the ability of these exams to measure separate distinct
constructs.
 
BB


>>> <dputka at humrro.org> 6/22/2010 7:06 AM >>>

Hi Bryan,

A few questions....

1. How are you defining error for your T&E measure?

a. As inconsistency across items?   In other words, do you want to draw
inferences with regard to the consistency in your T&E scores if they were
based on different sampling of items?

b. As inconsistency across occasions?  In other words, do you want to draw
inferences with regard to the consistency in your T&E scores if respondents
completed your measure on a different occasion?

c. Are you concerned about both types of inconsistency referenced above?


2. What is the substantive nature of your T&E measure?

a. Are you trying to assess multiple, distinct constructs with your T&E
measure (e.g., each construct indicated by a subset of items comprising the
T&E measure)?

b. Or, do you simply have an overarching, heterogenous measure of T&E?

Depending on how you define error (which is a function of the inferences
you want to make with regard to the consistency of your scores), and the
substantive nature of your T&E measure, it will dictate the structure of
the reliability coefficient that is most appropriate for your situation.

For a general discussion of this, see:
Putka, D. J. & Sackett, P .R. (2010).  Reliability and validity. In J.L.
Farr, & N.T. Tippins (Eds.). Handbook of Employee Selection (pp. 9-49). New
York: Routledge.


Hope this helps,

Dan


*******************************************************************
Dan J. Putka, Ph.D.
Principal Staff Scientist
Personnel Selection & Development Program
Human Resources Research Organization (HumRRO)
66 Canal Center Plaza, Suite 700
Alexandria, VA  22314-1591
703-706-5640 (o)
703-548-5574 (f)
http://www.humrro.org 
*******************************************************************




From:"Bryan Baldwin" <Bryan.Baldwin at doj.ca.gov>
To:ipac-list at ipacweb.org 
Date:06/21/2010 07:05 PM
Subject:[IPAC-List] Reliability for T&Es
Sent by:ipac-list-bounces at ipacweb.org 



Can anyone point me to a source for how to compute reliability for T&Es
(assuming it can be done)?  Ultimate goal is to use this to create bands.

Thanks-

Bryan Baldwin
Staff Services Manager II
California Department of Justice
Division of Administrative Support
Personnel Programs
(916) 322-5446


CONFIDENTIALITY NOTICE: This communication with its contents may contain
confidential and/or legally privileged information. It is solely for the
use of the intended recipient(s). Unauthorized interception, review, use or
disclosure is prohibited and may violate applicable laws including the
Electronic Communications Privacy Act. If you are not the intended
recipient, please contact the sender and destroy all copies of the
communication.
_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org 
http://www.ipacweb.org/mailman/listinfo/ipac-list 




CONFIDENTIALITY NOTICE: This communication with its contents may contain
confidential and/or legally privileged information. It is solely for the use
of the intended recipient(s). Unauthorized interception, review, use or
disclosure is prohibited and may violate applicable laws including the
Electronic Communications Privacy Act. If you are not the intended
recipient, please contact the sender and destroy all copies of the
communication.
_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org
http://www.ipacweb.org/mailman/listinfo/ipac-list