[IPAC-List] Appealed Items

[mhammer at 295.ca]

1) Safe to assume we are talking about a situational judgment test here? 
Perhaps I didn't read previous posts closely enough, but it didn't seem to
be mentioned explicitly.

2) When I used to teach, I stumbled across some papers in the journal
Teaching of Psychology that examined what they referred to as the "answer
justification option" for multiple-choice tests.  That option permitted
students to provide a brief rationale for why they selected the answer
they did.  If the justification displayed some pertinent knowledge of the
subject matter, the student could receive partial or full credit for the
item, even though the opscan would show it as an error.  The authors
indicated that, while it did not tend to change grades received in any
substantive way, it resulted in greater perceived fairness of the tests.

I used it, and observed that very pattern.  While none of the
multiple-choice tests I gave could be considered speed tests, still I
encouraged students to be judicious in their use of the option, because
writing things out by hand takes time; time that might be better allocated
to going over their answers and verifying their choices.  I found about
10% of my students used the option, used it for about 10% of the items,
and only about 10% of those items needed some scrutiny on my part (i.e.,
they had written justifications for something that would have been scored
as correct anyway).

What it also bought me was the avoidance of having to use the dreaded
phrase "If I do it for you, I'd have to do it for everyone".  Here, it
allowed their answer to be scored in a unique manner that did not obligate
crediting all other unjustified "wrong answers".  Saved me from hours of
irritating office visits from keeners desperate to get that one extra
point.

While I think this presented an elegant solution to my own scenario, I
suspect volume is an important factor to consider.  It was no big deal for
me to assess 100 justifications, several times a year. YMMV as they say. 
besides, the gist of this thread is what to do *after* the fact.  However,
I offer this option up as a way of sidestepping appeals, and problems of
this type.

3) SJTs are a curious beast, as I learned some 17 years ago, when our unit
experimented with one, in addition to a translated version.  Performance
on the translated version was a full S.D. below the nontranslated.  The
publisher had arranged for the translation, and when I inquired as to
whether the translator had access to the answer key during translation,
the reply was "No. They translated it line by line, and we have always
been happy with their service."

Trouble is, SJTs can include subtle verbal cues as to what is a better and
worse response choice.  If the situation described is ill-defined or
ambiguous, any response-choice commencing with "You should..." is
instantly a poorer choice than something commencing with "You could...". 
It is the complementarity of the stem and phrasing of the choices that can
dictate what an optimal response consists of.  We learned this the hard
way, when we observed that the translated version had neglected to pay
attention to these cues.  Those taking the translated version headed for a
particular "wrong" answer in droves, and avoided the correct one like the
plague.

The lesson here is to always be mindful of phrasing in SJT construction,
so as to avoid appeals of the type that began this thread.

Mark Hammer
Ottawa