[IPAC-List] Determining rater agreement when unique rater panels are used

Heather Patchell hpatchell at biddle.com
Thu Aug 26 18:14:22 EDT 2010

Suppose you have interview score data from combinations of three-rater
panels that frequently changed members. The result was that virtually
all of the interview panels contained unique groups of three raters.

We originally wanted to do a generalizability study (G and D). However,
typical methods of estimating variance components (e.g., REML, urGENOVA)
failed to converge because virtually all of the panels are unique.

Is there way to calculate a level of agreement of raters with this type
of data or perhaps some way to better understand if raters were
exhibiting an acceptable level of agreement? (Note: We also tried to
correlate matched-pairs of raters, but there were also a very limited
number of matched pairs of raters that we could use).

Thanks for your insight.

Heather J Patchell


Managing Editor, EEO Insight

Biddle Consulting Group, Inc.

193 Blue Ravine Suite 270 | Folsom, CA 95630

P: 916.294.4250 x 155 | 800.999.0834 x 155

F: 916.294.4255

www.biddle.com <http://www.biddle.com> | www.BCGinstitute.com

More information about the IPAC-List mailing list