[IPAC-List] scoring multiple answer questions

[Mark Hammer]

You can always rely on Harry for good ideas, just as you can rely on me
to make a mountain out of a molehill.....
 
The rational scale idea makes eminent sense, but the use of such an
approach to scaling makes far more sense if:
 
a) the test has many items (where the impact of the rational weighting
scheme on potential measurement error is minimal, on an item-wise
basis)
and
b) the test is not counting for a lot.
 
If the client is expecting a handful of items to do a lot of work for
them, then perhaps something more precise, in the way of weighting, is
called for.  But if it has a lot of items (and I won't commit to where
the inflection point between "some" and "a lot" lies), and the test does
not have critical implications, when other things are factored in, and
if you have the appropriate assurances from SMEs, then I think you're
good to go.
 
Mark Hammer
Ottawa
 

>>> <Harry.Brull at pdininthhouse.com> 2011/03/21 3:43 PM >>>

I'd suggest a rational scale, something on the order of:

+2 = essential to do
+1 = nice to do
0 = no harm/no benefit
-1 = bad to do
-2 = terrible to do

Get a bunch of SMEs together and come to consensus

Hope that works for you.

Harry Brull 


------------------------------------------------
Harry Brull |  Senior Vice-President 
PDI Ninth House
Global Leadership Solutions

1.612.337.8233  office
1.612.414.8998  mobile
1.612.337.3695  fax
Harry.Brull at PDINinthHouse.com

33 South Sixth Street
Suite 4900
Minneapolis, MN 55402

www.pdininthhouse.com 

-----Original Message-----
From: ipac-list-bounces at ipacweb.org
[mailto:ipac-list-bounces at ipacweb.org] On Behalf Of Megan Paul
Sent: Monday, March 21, 2011 1:50 PM
To: IPAC-List at ipacweb.org
Subject: [IPAC-List] scoring multiple answer questions

I am reviewing an on-the-job knowledge test developed by a client for 
in-house use (a pseudo certification test, I guess you could say). Many

of the items are multiple answer, with the number of right and wrong 
answers varying across items. They want to give partial credit for 
getting some of the right answers correct. To do that, they'll also
need

to penalize for selecting wrong answers (otherwise a test taker need 
only select all answers--right or wrong--to get full credit). So...I'm

trying to help them figure out a reasonable way to do this. If the
wrong

answers are worth the same points as the right answers (albeit with a 
negative value), there are cases where a test taker could lose more 
points than they'd earn. The only practice I can think to apply is the

correction for guessing, but I've never used such formulas, and it
seems

like it won't be so straightforward, given that a) we're talking about

the possibility of multiple wrong answers for each question and b) the

number of right and wrong answers varies across items (i.e., I can't 
arrive at a single formula for the whole test; might have to be a 
formula for each item? Is there such a thing?).

Any thoughts?

To be clear, I did not develop this test. I'm just being asked to weigh

in after the fact.

Thanks in advance,
Megan

-- 
Megan E. Paul, Ph.D.
Research Assistant Professor
University of Nebraska-Lincoln
Center on Children, Families, and the Law
206 S. 13th Street, Suite 1000
Lincoln, NE 68588-0227

(402) 472-9812 Office
(402) 472-8412 Fax


_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org
http://www.ipacweb.org/mailman/listinfo/ipac-list

Confidentiality Notice: All information in this communication,
including any files or attachments, is intended for the sole use of the
individual or entity to which it is addressed and may contain
information that is confidential, proprietary and/or trade secret
information entitled to protection and/or exempt from disclosure under
applicable law. If you are not the intended recipient, you are hereby
notified that any dissemination, distribution, or copying of this
communication is strictly prohibited. If you have received this
communication in error, please notify the sender by return email and
delete this communication from your system. Thank you for your
cooperation.
_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org
http://www.ipacweb.org/mailman/listinfo/ipac-list


>

This e-mail message is intended for the named recipient(s) and may
contain information that is privileged, confidential and/or exempt from
disclosure under applicable law. Unauthorized disclosure, copying or
re-transmission is prohibited. If you are not a named recipient or not
authorized by the named recipient(s), or if you have received this
e-mail in error, then please notify the sender immediately and delete
the message and any copies.
 >
Ce courriel est destiné exclusivement au destinataire mentionné en titre
et peut contenir de l'information privilégiée, confidentielle ou
soustraite à la communication aux termes des lois applicables. Toute
divulgation non autorisée, toute reproduction ou réacheminement est
interdit. Si vous n'êtes pas le destinataire de ce courriel, ou n'êtes
pas autorisé par le destinataire visé, ou encore, si vous l'avez reçu
par erreur, veuillez le mentionner immédiatement à l'expéditeur et
supprimer le courriel et les copies.