[IPAC-List] Does the day you take an oral exam matter?

[Richard Joines]

Dear Elizabeth:

I dealt with such a challenge to a Fire Battalion Chief exam conducted many 
years ago for the City of San Francisco.  We tested about 80 candidates over 
a five day period of time.  Candidates took an LGD as part of their 
assessment.  The challenge was to the effect that those candidates going 
later in the process would naturally score better since friends who went 
earlier would tell them the discussion problems.  Thus, the issue is whether 
or not prior knowledge of the oral exam content will result in higher 
scores.

While I believe that prior knowledge may help in many types of situations, 
my findings indicated that prior knowledge did not help in the LGD process. 
Candidates had been randomly assigned to one of five days for the 
assessment.  In such cases, I believe you can run T tests for mean score 
differences between groups, or just run a one way ANOVA to determine if any 
differences exist, then do your post hoc tests.  I didn't find any 
differences and the challenge was not upheld.

However, I found that candidate perceptions on this issue are quite strong, 
so I have avoided this approach ever since just for practical reasons.  If I 
can administer the oral component to all candidates on the same day, and 
administratively ensure that those who have not yet tested cannot contact 
those who have completed the process (i.e., by putting all in a holding room 
until their turn), that's what I do.

Again, without knowing the exact oral process you used, I can't formulate an 
opinion on whether or not prior knowledge of the process or content of the 
exercise would likely affect score outcomes.  It's something to be concerned 
about, naturally.  It really comes down to how you score the exercise and 
the extent to which knowledge rather than skill impacts the scores that are 
assigned.  In the case of the LGD,  the results indicated that prior 
knowledge of the discussion problems did not affect the leadership and 
problem analysis and oral communication ratings that we made.  I tend to 
believe that prior knowledge would have a greater impact on structured 
interview results than on role-plays, but that may not be the case if your 
structured interview is based on candidate reports of prior accomplishments.

Good luck,
Richard Joines
Management & Personnel Systems, Inc.



----- Original Message ----- 
From: "Don Nott" <Don.Nott at fcgtesting.com>
To: "Reed, Elizabeth" <EReed1 at Columbus.gov>; <ipac-list at ipacweb.org>
Sent: Wednesday, April 01, 2009 8:55 AM
Subject: Re: [IPAC-List] Does the day you take an oral exam matter?



> Hi Elizabeth,

>

> Our company specializes in the development of public safety (police &

> fire) testing & assessment; so, we deal with your candidate culture all

> the time.

>

> Simple answer: the more the candidate's understand the process, the less

> likely they'll complain; assuming the process is objective and fair.

>

> We provide a candidate orientation to candidate's so that they A)

> understand how the exercises were developed, and B) understand that

> assessors are using objective, job-related, behavioral criteria when

> scoring; therefore, it doesn't matter the time/date that they are

> tested.

>

> My quick answer is that you need to start with a good process and ensure

> that you educate candidates.  This reduces candidate skepticism and

> increases their understanding that, hey, the process/exercises are fair

> and objective.

>

> :Don

>

> -----------------------------------------------------

> Donald M. Nott

> I/O Associate

>

> Fields Consulting Group

> 7926 Jones Branch Drive

> McLean, VA 22102

> (703) 506-9400

>

> -----Original Message-----

> From: ipac-list-bounces at ipacweb.org

> [mailto:ipac-list-bounces at ipacweb.org] On Behalf Of Reed, Elizabeth

> Sent: Wednesday, April 01, 2009 11:47 AM

> To: 'ipac-list at ipacweb.org'

> Subject: [IPAC-List] Does the day you take an oral exam matter?

>

> IPAC members and friends:

>

> The City of Columbus utilizes oral exams in all of our promotional

> testing for both police and fire.  These oral exams usually consist of

> either two role-play exercises or one role-play exercise and one

> structured interview.  Depending on the number of candidates, these

> exams can take up to six full days to administer.  Over the years,

> candidates have complained about being assigned on either the first day

> of testing or the last day of testing.  The candidates' perception is

> that the day of the exam makes a difference in the score that they

> receive.  Questions for you:

>

> Is this a common perception?  Is anyone aware of studies conducted to

> analyze this?  What do you believe would be the best approach to study

> this situation?

>

> Thank you,

> Liz Reed

>

> Elizabeth A. Reed

> Police and Fire Assessment Supervisor

> Columbus Civil Service Commission

> 750 Piedmont Rd.

> Columbus, Ohio 43224

> Office: (614) 645-6032 Fax: (614) 645-0866

> Email: Ereed1 at columbus.gov

>

>

> _______________________________________________________

> IPAC-List

> IPAC-List at ipacweb.org

> http://www.ipacweb.org/mailman/listinfo/ipac-list

> _______________________________________________________

> IPAC-List

> IPAC-List at ipacweb.org

> http://www.ipacweb.org/mailman/listinfo/ipac-list