[IPAC-List] Does the day you take an oral exam matter?

[Don Nott]

Absolutely Mark,  

Beyond the PR & trust building with candidates, and the valid
construction of exercises, Assessor training & practice prior to
administration is an absolute necessity.  Most of the time, assessors
must be calibrated to the department's criteria; assuming assessors are
from outside of the department.  Particular attention must be paid to
whether or not assessors are using their own experiences and judgment to
evaluate candidate's performance, as opposed to the specific criteria
that has been identified during the exercise's development.  You may
have the most wonderful assessment in the world, but if the assessors
don't use it properly, it's worthless.  Proper assessor training &
practice is essential for score consistency and validity.


:D


-----------------------------------------------------
Donald M. Nott
I/O Associate
  
Fields Consulting Group
7926 Jones Branch Drive
McLean, VA 22102
(703) 506-9400 


-----Original Message-----
From: ipac-list-bounces at ipacweb.org
[mailto:ipac-list-bounces at ipacweb.org] On Behalf Of Mark Hammer
Sent: Wednesday, April 01, 2009 12:32 PM
To: ipac-list at ipacweb.org
Subject: Re: [IPAC-List] Does the day you take an oral exam matter?

I gather the complaints are predicated on the assumption that the
examiners all commence and end on the same day.  In which case, they may
be more stringent at the outset, and grow more lenient over candidates
as they "warm up", and be in a hurry to "get it over with" on the last
day, hence more likely to overlook things (i.e., the candidate's finer
points).

That's not an entirely unreasonable set of assumptions, and has some
face validity....IF the examiners all begin and end on the same day and
engage in no assessment prior to or after those days in question.   And
I suspect that is where the information/transparency aspect that Don
notes comes into play.  If examiners have plenty of warm-up time or
practice before, and are simply moving on to another set of examinees
once finished this lot, then the considerations noted in my opening
don't apply nearly as much as people think they do.  And if there are
safeguards built into the process to systematically combat the warm-up
and fatigue factors and produce comparably stringent ratings over the
entire testing period, then that's worth pointing out to candidates.
It's not like they're the very first people to ever consider that
evaluators can drift.  People HAVE thought about this stuff before.

Mark Hammer
Ottawa

_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org
http://www.ipacweb.org/mailman/listinfo/ipac-list