[IPAC-List] Reliability for T&Es

[Saul Fine]

Hi Bryan,
I might suggest weighting each response option per item (consistently, 
across the items, e.g., most simply as dichotomous 0/1) based on either a 
judgment/content requirement (if relevant) or empirically against an 
external criteria.
Then the items could be computed for internal consistency in the normal 
fashion, assuming you have an overarching construct being tapped.
I would then approach banding the scores in terms of the variance, such as 
in the form of stanine scores. Of course, also supporting (or modifying) the 
bands based on expectancy tables against an external criteria would be 
ideal.
Good luck,
Saul.

Saul Fine, PhD
Chief R&D Officer
t: +(972)-3-616-1111
f: +(972)-3-616-1515
c: +(972)-544-483-729
e: saul at midot.co.il


----- Original Message ----- 
From: "Bryan Baldwin" <Bryan.Baldwin at doj.ca.gov>
To: <ipac-list at ipacweb.org>
Sent: Tuesday, June 22, 2010 6:30 PM
Subject: Re: [IPAC-List] Reliability for T&Es



>I would actually be perfectly okay with setting these exams up at pass/fail 

>rather than get into banding at all.  Unfortunately civil service rules 

>don't always allow us to do what we want!  Additionally, when we've tried 

>to establish these as pass/fail the unions have argued this degrades merit. 

>Kinda funny/sad that it's the change to T&Es that degrades merit, not what 

>you do with the scores.

>

>>>> "Winfred Arthur, Jr." <w-arthur at neo.tamu.edu> 6/21/2010 8:36 PM >>>

> interestingly, Dennis' position is consonant with the argument to band

> on the (predicted) _*criterion*_ score instead of the predictor score.

> ah banding  :)

>

> - winfred

>

> On 6/21/2010 10:27 PM, Doverspike,Dennis wrote:

>> This suggests I believe the logical problem with banding, especially 

>> based on reliability. Beyond the question of the psychometric basis. If 

>> you are really serious about it, why band based upon reliability, why not 

>> band based upon validity? Then, if we know the validity of T&E to be 

>> extremely low, the band range would be huge. Probably as wide as the 

>> score range. As a purely hypothetical question, if a test had perfect 

>> reliability but zero relationship with merit, what should the band be?

>> ________________________________________

>> From: ipac-list-bounces at ipacweb.org [ipac-list-bounces at ipacweb.org] On 

>> Behalf Of Joel Wiesen [jwiesen at appliedpersonnelresearch.com]

>> Sent: Monday, June 21, 2010 7:24 PM

>> To: Bryan Baldwin

>> Cc: ipac-list at ipacweb.org

>> Subject: Re: [IPAC-List] Reliability for T&Es

>>

>> How about test-retest reliability?

>>

>>

>> Joel

>>

>>

>> - -

>> Joel P. Wiesen, Ph.D., Director

>> Applied Personnel Research

>> 62 Candlewood Road

>> Scarsdale, NY 10583-6040

>> (617) 244-8859

>> http://appliedpersonnelresearch.com

>>

>>

>>

>>

>>

>> Bryan Baldwin wrote:

>>

>>> Can anyone point me to a source for how to compute reliability for T&Es 

>>> (assuming it can be done)?  Ultimate goal is to use this to create 

>>> bands.

>>>

>>> Thanks-

>>>

>>> Bryan Baldwin

>>> Staff Services Manager II

>>> California Department of Justice

>>> Division of Administrative Support

>>> Personnel Programs

>>> (916) 322-5446

>>>

>>>

>>> CONFIDENTIALITY NOTICE: This communication with its contents may contain 

>>> confidential and/or legally privileged information. It is solely for the 

>>> use of the intended recipient(s). Unauthorized interception, review, use 

>>> or disclosure is prohibited and may violate applicable laws including 

>>> the Electronic Communications Privacy Act. If you are not the intended 

>>> recipient, please contact the sender and destroy all copies of the 

>>> communication.

>>>

>>>

>> _______________________________________________________

>> IPAC-List

>> IPAC-List at ipacweb.org

>> http://www.ipacweb.org/mailman/listinfo/ipac-list

>> _______________________________________________________

>> IPAC-List

>> IPAC-List at ipacweb.org

>> http://www.ipacweb.org/mailman/listinfo/ipac-list

>>

> _______________________________________________________

> IPAC-List

> IPAC-List at ipacweb.org

> http://www.ipacweb.org/mailman/listinfo/ipac-list

>

>

> CONFIDENTIALITY NOTICE: This communication with its contents may contain 

> confidential and/or legally privileged information. It is solely for the 

> use of the intended recipient(s). Unauthorized interception, review, use 

> or disclosure is prohibited and may violate applicable laws including the 

> Electronic Communications Privacy Act. If you are not the intended 

> recipient, please contact the sender and destroy all copies of the 

> communication.

> _______________________________________________________

> IPAC-List

> IPAC-List at ipacweb.org

> http://www.ipacweb.org/mailman/listinfo/ipac-list

>