[IPAC-List] Does the day you take an oral exam matter?

[Brull, Harry]

I need to take issue with dismissing an approach with "absolutely
not-nuff said". 
To clarify, we use the behavioral consistency approach which consists of
opening questions (e.g. "Tell me about the most difficult decision
you've had to make as a law enforcement professional") followed by
follow-up probing questions. As prepatory material for the oral, we give
candidates the opening question only.
I began doing research into this approach 20 years ago and gave
candidates every other question. Analysis of results showed no
difference in scores between given/not given questions but higher
confidence on the part of raters where candidates could think before the
oral and greater comfort on the part of candidates. The lack of security
concerns was a third bonus.
By the way, we use it for entry level, first-line supervisor, and
occasionally mid-management. Beyond that, we expect candidates to handle
questions without preparation.
Having now used this procedure 500+ times, I have never had a negative
outcome. Criterion validation data tends to be robust.
Please do not reject ideas without inquiry. We advance our profession
when we try new things and draw conclusions based on data.


Harry Brull
Senior Vice President
Personnel Decisions International-PDI Minneapolis
Expert Partners. Real Leadership Advantage.
Suite 4900
33 South sixth Street 
Minneapolis, Minnesota  55402
Direct :612-337-8233    fax: 612-337-3695
harry.brull at personneldecisions.com

www.personneldecisions.com


-----Original Message-----
From: ipac-list-bounces at ipacweb.org
[mailto:ipac-list-bounces at ipacweb.org] On Behalf Of Don Nott
Sent: Wednesday, April 01, 2009 2:59 PM
To: Brull, Harry; Reed, Elizabeth; ipac-list at ipacweb.org
Subject: Re: [IPAC-List] Does the day you take an oral exam matter?

All good suggestions; however, I think that a cost-benefit evaluation is
necessary to truly determine the value of some of the suggestions.

Multiple Role Play Scenarios - Developing specific job-relevant BARS are
very labor-intensive, expect the costs to go way up if you want multiple
parallel role plays.  Also, a good point was made as to the validity of
those scenarios being truly "parallel;" that fact alone will get
candidates in an uproar if they find out that the scenarios they
responded to were different from their buddy's scenarios.

One day of Administration - Depending upon the number of assessors, your
facilities, and the number of candidates, this may or may not be
realistic. Again, if cost is no object (which, come on, of course it is;
particularly in public sector), go for it.

Providing Questions to Candidates prior to Administration - Absolutely
not; enough said.  Providing candidates with information as to the
potential content & nature of questions, that's a different story. 

I don't want to be preaching to the choir, but I believe that almost all
candidate appeals are preventable and deniable if the appropriate
actions are taken in development and administration.

1) Development of job-relevant and behaviorally-based exercises yields
higher validity and reliability.
2) Candidate Orientation is more about PR with candidates than anything
else.  Candidates in public sector have the luxury of understanding how
the process was developed and what to expect on administration day.
3) Adequate training & practice for assessors to ensure their
understanding of the exercises and scales, as well as an opportunity to
evaluate their accuracy and reliability in scoring before
administration.

I feel the negative PR that looms around testing is because of its
mystique and lay person's ignorance of how tests are developed.
Assuming that you properly developed your assessments, why not explain
how they were developed to be job-relevant, objective, consistent, and
above all else: "fair."   Preventing a mess costs a lot less than the
cleaning one up.


:Don


-----------------------------------------------------
Donald M. Nott
I/O Associate
  
Fields Consulting Group
7926 Jones Branch Drive
McLean, VA 22102
(703) 506-9400 


-----Original Message-----
From: ipac-list-bounces at ipacweb.org
[mailto:ipac-list-bounces at ipacweb.org] On Behalf Of Brull, Harry
Sent: Wednesday, April 01, 2009 3:29 PM
To: Reed, Elizabeth; ipac-list at ipacweb.org
Subject: Re: [IPAC-List] Does the day you take an oral exam matter?

I have heard this occasionally (particularly in the past). One of the
bases for the perception is that candidate's who appear later have more
access (real or perceived) to information which helps them prepare.
By the way, the best antidote for structured orals is to give all
candidates sample questions on the front end. That way, all candidates
have time to prepare - no matter what day they appear.
For role-play exercises, one solution is to change scenarios every day
(or use random scenario assignment throughout the process). Then
standard score each scenario.
Using these two methods, I've not ever heard the complaint.
The proofs-in-the-pudding argument is to analyze scores by day and see
if there is any upward trend. The danger with this after-the-fact
analysis is "What do you do if it turns out to be true?"

Harry Brull


Harry Brull
Senior Vice President
Personnel Decisions International-PDI Minneapolis
Expert Partners. Real Leadership Advantage.
Suite 4900
33 South sixth Street 
Minneapolis, Minnesota  55402
Direct :612-337-8233    fax: 612-337-3695
harry.brull at personneldecisions.com

www.personneldecisions.com


-----Original Message-----
From: ipac-list-bounces at ipacweb.org
[mailto:ipac-list-bounces at ipacweb.org] On Behalf Of Reed, Elizabeth
Sent: Wednesday, April 01, 2009 10:47 AM
To: 'ipac-list at ipacweb.org'
Subject: [IPAC-List] Does the day you take an oral exam matter?

IPAC members and friends:

The City of Columbus utilizes oral exams in all of our promotional
testing for both police and fire.  These oral exams usually consist of
either two role-play exercises or one role-play exercise and one
structured interview.  Depending on the number of candidates, these
exams can take up to six full days to administer.  Over the years,
candidates have complained about being assigned on either the first day
of testing or the last day of testing.  The candidates' perception is
that the day of the exam makes a difference in the score that they
receive.  Questions for you:

Is this a common perception?  Is anyone aware of studies conducted to
analyze this?  What do you believe would be the best approach to study
this situation?

Thank you,
Liz Reed

Elizabeth A. Reed
Police and Fire Assessment Supervisor
Columbus Civil Service Commission
750 Piedmont Rd.
Columbus, Ohio 43224
Office: (614) 645-6032 Fax: (614) 645-0866
Email: Ereed1 at columbus.gov


_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org
http://www.ipacweb.org/mailman/listinfo/ipac-list

Confidentiality Notice: All information in this communication, including
any files or attachments, is intended for the sole use of the individual
or entity to which it is addressed and may contain information that is
confidential, proprietary and/or trade secret information entitled to
protection and/or exempt from disclosure under applicable law. If you
are not the intended recipient, you are hereby notified that any
dissemination, distribution, or copying of this communication is
strictly prohibited. If you have received this communication in error,
please notify the sender by return email and delete this communication
from your system. Thank you for your cooperation.
_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org
http://www.ipacweb.org/mailman/listinfo/ipac-list
_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org
http://www.ipacweb.org/mailman/listinfo/ipac-list

Confidentiality Notice: All information in this communication, including any files or attachments, is intended for the sole use of the individual or entity to which it is addressed and may contain information that is confidential, proprietary and/or trade secret information entitled to protection and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender by return email and delete this communication from your system. Thank you for your cooperation.