[IPAC-List] Good BARS Inter-rater Agreement?

[Joel Wiesen]

Perhaps you can share your experience or thoughts concerning inter-rater 
agreement.

Based on your experience (or published literature) what level of 
agreement is reasonable to expect between 2 independent rating panels, 
each consisting of 3 raters?

Assume:

1. The exercise being graded is part of a public safety promotional 
process and includes written instructions, with a one hour prep time 
followed by a 15 min role play.

2. Each panel rates each candidate on 4 managerial/supervisory 
dimensions, using one BARS scale per dimension.

3. Each rating panel bases its ratings a video recording of each 
candidate.

4. The two panels are trained together before the rating of candidates 
begins.  There is no communication between the two panels during the 
rating of candidates.

We can consider the scores at two levels: at the dimension level and 
overall.  For each rating panel, a candidate's score on a dimension is 
the mean of 3 ratings (one rating by each of the three raters), and a 
candidate's overall score is the mean of 12 ratings (3 ratings for each 
of 4 dimensions).

We can consider agreement of two types: agreement of the 3 raters within 
each panel, and the agreement of the 2 panels.

In short, what level(s) of agreement among the raters within each panel 
and across the two panels is reasonable to expect: for the 4 dimension 
scores and for the overall scores?

Your comments will be greatly appreciated.

Joel




-- 
Joel P. Wiesen, Ph.D., Director
Applied Personnel Research
62 Candlewood Road
Scarsdale, NY 10583-6040
http://www.linkedin.com/in/joelwiesen
(617) 244-8859
http://appliedpersonnelresearch.com




Note: This e-mail and any attachments may contain confidential and/or 
legally privileged information. Please do not forward any contents 
without permission. If you have received this message in error please 
destroy all copies, completely remove it from your computer, and notify 
the sender. Thank you.