[IPAC-List] scoring multiple answer questions

[cpaullin at humrro.org]

I'm sharing an answer from Gordon Waugh, one of HumRRO's senior scientists 
who worked extensively with item types such as this one.  He also 
presented a paper at the 2010 Association for Test Publishers Conference 
comparing different item types.  If you are interested in the paper and 
can't find it on the ATP website, let me know, and I'd be happy to share 
it with you.

Here's Gordon's response (note that it's consistent with one of Dennis 
Doverspike's recommendations):

The type of question you are describing (multiple-answer) is commonly 
called "Multiple True False (MTF)" or "Multiple Response."

The name, "multiple true-false," actually hints at the most popular way to 
give partial credit scoring for this type of item. Treat each response 
option as a true-false item. When someone selects an option they are 
actually stating that it is true. When someone does not select an option 
they are stating that it is false. The scoring key can be viewed the same 
way. Each option is, in reality, either true or false. 

So, each option is scored in the following way:
- If the key says the option is false, and the examinee does not select 
the option (i.e., she thinks it is false), then assign a score of 1 to 
that option.
- If the key says the option is false, and the examinee selects the option 
(i.e., she thinks it is true), then assign a score of 0 to that option.
- If the key says the option is true, and the examinee does not select the 
option (i.e., she thinks it is false), then assign a score of 0 to that 
option.
- If the key says the option is true, and the examinee selects the option 
(i.e., she thinks it is true), then assign a score of 1 to that option.

The score for the entire item is simply the sum of the option scores. So, 
if an item has four options, then the possible scores on the item are 
0,1,2,3,4.

You should also weight the item scores. Note that a four-option MTF item 
has a maximum score of 4. You should probably multiple the item score by 
1/4 so that the maximum item score is 1.0.

By the way, I have done research that compares (a) partial credit vs. 
all-or-none scoring and (b) different weighting (item=1 point vs. option=1 
point) and described it in an ATP 2010 conference paper/




From:   Megan Paul <mpaul2 at unlserve.unl.edu>
To:     IPAC-List at ipacweb.org
Date:   03/21/2011 01:49 PM
Subject:        [IPAC-List] scoring multiple answer questions
Sent by:        ipac-list-bounces at ipacweb.org



I am reviewing an on-the-job knowledge test developed by a client for 
in-house use (a pseudo certification test, I guess you could say). Many 
of the items are multiple answer, with the number of right and wrong 
answers varying across items. They want to give partial credit for 
getting some of the right answers correct. To do that, they'll also need 
to penalize for selecting wrong answers (otherwise a test taker need 
only select all answers--right or wrong--to get full credit). So...I'm 
trying to help them figure out a reasonable way to do this. If the wrong 
answers are worth the same points as the right answers (albeit with a 
negative value), there are cases where a test taker could lose more 
points than they'd earn. The only practice I can think to apply is the 
correction for guessing, but I've never used such formulas, and it seems 
like it won't be so straightforward, given that a) we're talking about 
the possibility of multiple wrong answers for each question and b) the 
number of right and wrong answers varies across items (i.e., I can't 
arrive at a single formula for the whole test; might have to be a 
formula for each item? Is there such a thing?).

Any thoughts?

To be clear, I did not develop this test. I'm just being asked to weigh 
in after the fact.

Thanks in advance,
Megan

-- 
Megan E. Paul, Ph.D.
Research Assistant Professor
University of Nebraska-Lincoln
Center on Children, Families, and the Law
206 S. 13th Street, Suite 1000
Lincoln, NE 68588-0227

(402) 472-9812 Office
(402) 472-8412 Fax


_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org
http://www.ipacweb.org/mailman/listinfo/ipac-list