[IPAC-List] scoring multiple answer questions
cpaullin at humrro.org
cpaullin at humrro.org
Wed Mar 23 08:47:26 EDT 2011
I'm sharing an answer from Gordon Waugh, one of HumRRO's senior scientists
who worked extensively with item types such as this one. He also
presented a paper at the 2010 Association for Test Publishers Conference
comparing different item types. If you are interested in the paper and
can't find it on the ATP website, let me know, and I'd be happy to share
it with you.
Here's Gordon's response (note that it's consistent with one of Dennis
The type of question you are describing (multiple-answer) is commonly
called "Multiple True False (MTF)" or "Multiple Response."
The name, "multiple true-false," actually hints at the most popular way to
give partial credit scoring for this type of item. Treat each response
option as a true-false item. When someone selects an option they are
actually stating that it is true. When someone does not select an option
they are stating that it is false. The scoring key can be viewed the same
way. Each option is, in reality, either true or false.
So, each option is scored in the following way:
- If the key says the option is false, and the examinee does not select
the option (i.e., she thinks it is false), then assign a score of 1 to
- If the key says the option is false, and the examinee selects the option
(i.e., she thinks it is true), then assign a score of 0 to that option.
- If the key says the option is true, and the examinee does not select the
option (i.e., she thinks it is false), then assign a score of 0 to that
- If the key says the option is true, and the examinee selects the option
(i.e., she thinks it is true), then assign a score of 1 to that option.
The score for the entire item is simply the sum of the option scores. So,
if an item has four options, then the possible scores on the item are
You should also weight the item scores. Note that a four-option MTF item
has a maximum score of 4. You should probably multiple the item score by
1/4 so that the maximum item score is 1.0.
By the way, I have done research that compares (a) partial credit vs.
all-or-none scoring and (b) different weighting (item=1 point vs. option=1
point) and described it in an ATP 2010 conference paper/
From: Megan Paul <mpaul2 at unlserve.unl.edu>
To: IPAC-List at ipacweb.org
Date: 03/21/2011 01:49 PM
Subject: [IPAC-List] scoring multiple answer questions
Sent by: ipac-list-bounces at ipacweb.org
I am reviewing an on-the-job knowledge test developed by a client for
in-house use (a pseudo certification test, I guess you could say). Many
of the items are multiple answer, with the number of right and wrong
answers varying across items. They want to give partial credit for
getting some of the right answers correct. To do that, they'll also need
to penalize for selecting wrong answers (otherwise a test taker need
only select all answers--right or wrong--to get full credit). So...I'm
trying to help them figure out a reasonable way to do this. If the wrong
answers are worth the same points as the right answers (albeit with a
negative value), there are cases where a test taker could lose more
points than they'd earn. The only practice I can think to apply is the
correction for guessing, but I've never used such formulas, and it seems
like it won't be so straightforward, given that a) we're talking about
the possibility of multiple wrong answers for each question and b) the
number of right and wrong answers varies across items (i.e., I can't
arrive at a single formula for the whole test; might have to be a
formula for each item? Is there such a thing?).
To be clear, I did not develop this test. I'm just being asked to weigh
in after the fact.
Thanks in advance,
Megan E. Paul, Ph.D.
Research Assistant Professor
University of Nebraska-Lincoln
Center on Children, Families, and the Law
206 S. 13th Street, Suite 1000
Lincoln, NE 68588-0227
(402) 472-9812 Office
(402) 472-8412 Fax
IPAC-List at ipacweb.org
More information about the IPAC-List