[IPAC-List] Does the day you take an oral exam matter?

[Reed, Elizabeth]

Wow this has spurred a great deal of discussion!

We conducted such an analysis over a decade ago, although the results are no longer available, I was assured that there was no difference by day. We plan to conduct such a study again, but before we do, we hoped to gain some insights; we got plenty.

I learned quite a bit from IPACers--thank you. First, I learned that those who have conducted such a study have not seen any differences by day. That's reassuring. We hope to conduct the study again and ideally share the results in our information sessions with candidates--we can also state that others who have conducted such a study has had similar results.

I want to apologize, in my original question I selected the wrong verb when I stated candidates "complained." Rather, many "speculate" that the day matters. We typically allow candidates to provide a preference of the day/a.m. or p.m. such that if it's a week long test they would select of ten time options their first, second and third choice. We hear some candidates claim that there's no way they want the first day, while others claim no way on the last day; sometimes each providing identical rationale. Most candidates get their first or second choice in time slots, so we really don't receive "complaints" about when they are scheduled. An interesting note, in all the years we've done this, the time slots for which we have more candidates select as their first choice than we have slots available are almost always the first day in the morning and the second day in the morning. This may have to do more with wanting to get it over with than a perceived advantage or disadvantage--speculation on my part.

As we start this project one of our concerns is, as Harry pointed out, what if we look and there is a difference, how are we going to deal with it?

To answer that question, we received a robust set of suggestions. Because we do allow candidates select the day, we may could postulate that the differences may be due to the level of candidates who would select a specific day, rather than the exam itself. Or, we could consider rater fatigue and follow-up by surveying raters throughout the process to see if fatigue sets in a various times/days. If so, we may be able to build in more breaks and other precautionary measures to reduce fatigue. If we speculate cheating is involved, we could continue to take action in that regard. Some recommended parallel exams; we moved away from using those or using them in a limited way due to the pitfalls mentioned. For us, speculation of sharing information (cheating) is rumored more in the fire promotionals than police promotions. Due to this speculation, we did proposed to fire representatives to give video-based oral exams in which responses are recorded and scored later, but allowing administration to occur in one day and sequestering of candidates during that day. These representatives felt that the assessment is stronger with the live interaction, and that the accusations of cheating are just talk. We do have a prep period prior to the exam, which, in my mind, reduces the concerns of significant advantage that could be gained by inappropriately getting information in advance. Overall, lots to consider.

In the original post I mentioned both police and fire promotional exams, but in the discussion proposing why such complaints were being made, we quickly focused on the cynicism found in law enforcement. I ask, rhetorically, why did we go there? :)

Thanks for your feedback. For those of you who indicated that you have conducted such a study, we may contact you later to pick your brains for the grubby details.

Thanks!
Liz

Elizabeth A. Reed
Police and Fire Assessment Supervisor
Columbus Civil Service Commission
750 Piedmont Rd.
Columbus, Ohio 43224
Office: (614) 645-6032 Fax: (614) 645-0866
Email: Ereed1 at columbus.gov