[IPAC-List] Does the day you take an oral exam matter?

Don Nott Don.Nott at fcgtesting.com
Wed Apr 1 15:58:30 EDT 2009

All good suggestions; however, I think that a cost-benefit evaluation is
necessary to truly determine the value of some of the suggestions.

Multiple Role Play Scenarios - Developing specific job-relevant BARS are
very labor-intensive, expect the costs to go way up if you want multiple
parallel role plays. Also, a good point was made as to the validity of
those scenarios being truly "parallel;" that fact alone will get
candidates in an uproar if they find out that the scenarios they
responded to were different from their buddy's scenarios.

One day of Administration - Depending upon the number of assessors, your
facilities, and the number of candidates, this may or may not be
realistic. Again, if cost is no object (which, come on, of course it is;
particularly in public sector), go for it.

Providing Questions to Candidates prior to Administration - Absolutely
not; enough said. Providing candidates with information as to the
potential content & nature of questions, that's a different story.

I don't want to be preaching to the choir, but I believe that almost all
candidate appeals are preventable and deniable if the appropriate
actions are taken in development and administration.

1) Development of job-relevant and behaviorally-based exercises yields
higher validity and reliability.
2) Candidate Orientation is more about PR with candidates than anything
else. Candidates in public sector have the luxury of understanding how
the process was developed and what to expect on administration day.
3) Adequate training & practice for assessors to ensure their
understanding of the exercises and scales, as well as an opportunity to
evaluate their accuracy and reliability in scoring before

I feel the negative PR that looms around testing is because of its
mystique and lay person's ignorance of how tests are developed.
Assuming that you properly developed your assessments, why not explain
how they were developed to be job-relevant, objective, consistent, and
above all else: "fair." Preventing a mess costs a lot less than the
cleaning one up.


Donald M. Nott
I/O Associate

Fields Consulting Group
7926 Jones Branch Drive
McLean, VA 22102
(703) 506-9400

-----Original Message-----
From: ipac-list-bounces at ipacweb.org
[mailto:ipac-list-bounces at ipacweb.org] On Behalf Of Brull, Harry
Sent: Wednesday, April 01, 2009 3:29 PM
To: Reed, Elizabeth; ipac-list at ipacweb.org
Subject: Re: [IPAC-List] Does the day you take an oral exam matter?

I have heard this occasionally (particularly in the past). One of the
bases for the perception is that candidate's who appear later have more
access (real or perceived) to information which helps them prepare.
By the way, the best antidote for structured orals is to give all
candidates sample questions on the front end. That way, all candidates
have time to prepare - no matter what day they appear.
For role-play exercises, one solution is to change scenarios every day
(or use random scenario assignment throughout the process). Then
standard score each scenario.
Using these two methods, I've not ever heard the complaint.
The proofs-in-the-pudding argument is to analyze scores by day and see
if there is any upward trend. The danger with this after-the-fact
analysis is "What do you do if it turns out to be true?"

Harry Brull

Harry Brull
Senior Vice President
Personnel Decisions International-PDI Minneapolis
Expert Partners. Real Leadership Advantage.
Suite 4900
33 South sixth Street
Minneapolis, Minnesota 55402
Direct :612-337-8233 fax: 612-337-3695
harry.brull at personneldecisions.com


-----Original Message-----
From: ipac-list-bounces at ipacweb.org
[mailto:ipac-list-bounces at ipacweb.org] On Behalf Of Reed, Elizabeth
Sent: Wednesday, April 01, 2009 10:47 AM
To: 'ipac-list at ipacweb.org'
Subject: [IPAC-List] Does the day you take an oral exam matter?

IPAC members and friends:

The City of Columbus utilizes oral exams in all of our promotional
testing for both police and fire. These oral exams usually consist of
either two role-play exercises or one role-play exercise and one
structured interview. Depending on the number of candidates, these
exams can take up to six full days to administer. Over the years,
candidates have complained about being assigned on either the first day
of testing or the last day of testing. The candidates' perception is
that the day of the exam makes a difference in the score that they
receive. Questions for you:

Is this a common perception? Is anyone aware of studies conducted to
analyze this? What do you believe would be the best approach to study
this situation?

Thank you,
Liz Reed

Elizabeth A. Reed
Police and Fire Assessment Supervisor
Columbus Civil Service Commission
750 Piedmont Rd.
Columbus, Ohio 43224
Office: (614) 645-6032 Fax: (614) 645-0866
Email: Ereed1 at columbus.gov

IPAC-List at ipacweb.org

Confidentiality Notice: All information in this communication, including
any files or attachments, is intended for the sole use of the individual
or entity to which it is addressed and may contain information that is
confidential, proprietary and/or trade secret information entitled to
protection and/or exempt from disclosure under applicable law. If you
are not the intended recipient, you are hereby notified that any
dissemination, distribution, or copying of this communication is
strictly prohibited. If you have received this communication in error,
please notify the sender by return email and delete this communication
from your system. Thank you for your cooperation.
IPAC-List at ipacweb.org

More information about the IPAC-List mailing list