[IPAC-List] scoring multiple answer questions

Mark Hammer Mark.Hammer at psc-cfp.gc.ca
Mon Mar 21 16:58:14 EDT 2011

You can always rely on Harry for good ideas, just as you can rely on me
to make a mountain out of a molehill.....

The rational scale idea makes eminent sense, but the use of such an
approach to scaling makes far more sense if:

a) the test has many items (where the impact of the rational weighting
scheme on potential measurement error is minimal, on an item-wise
b) the test is not counting for a lot.

If the client is expecting a handful of items to do a lot of work for
them, then perhaps something more precise, in the way of weighting, is
called for. But if it has a lot of items (and I won't commit to where
the inflection point between "some" and "a lot" lies), and the test does
not have critical implications, when other things are factored in, and
if you have the appropriate assurances from SMEs, then I think you're
good to go.

Mark Hammer

>>> <Harry.Brull at pdininthhouse.com> 2011/03/21 3:43 PM >>>

I'd suggest a rational scale, something on the order of:

+2 = essential to do
+1 = nice to do
0 = no harm/no benefit
-1 = bad to do
-2 = terrible to do

Get a bunch of SMEs together and come to consensus

Hope that works for you.

Harry Brull

Harry Brull | Senior Vice-President
PDI Ninth House
Global Leadership Solutions

1.612.337.8233 office
1.612.414.8998 mobile
1.612.337.3695 fax
Harry.Brull at PDINinthHouse.com

33 South Sixth Street
Suite 4900
Minneapolis, MN 55402


-----Original Message-----
From: ipac-list-bounces at ipacweb.org
[mailto:ipac-list-bounces at ipacweb.org] On Behalf Of Megan Paul
Sent: Monday, March 21, 2011 1:50 PM
To: IPAC-List at ipacweb.org
Subject: [IPAC-List] scoring multiple answer questions

I am reviewing an on-the-job knowledge test developed by a client for
in-house use (a pseudo certification test, I guess you could say). Many

of the items are multiple answer, with the number of right and wrong
answers varying across items. They want to give partial credit for
getting some of the right answers correct. To do that, they'll also

to penalize for selecting wrong answers (otherwise a test taker need
only select all answers--right or wrong--to get full credit). So...I'm

trying to help them figure out a reasonable way to do this. If the

answers are worth the same points as the right answers (albeit with a
negative value), there are cases where a test taker could lose more
points than they'd earn. The only practice I can think to apply is the

correction for guessing, but I've never used such formulas, and it

like it won't be so straightforward, given that a) we're talking about

the possibility of multiple wrong answers for each question and b) the

number of right and wrong answers varies across items (i.e., I can't
arrive at a single formula for the whole test; might have to be a
formula for each item? Is there such a thing?).

Any thoughts?

To be clear, I did not develop this test. I'm just being asked to weigh

in after the fact.

Thanks in advance,

Megan E. Paul, Ph.D.
Research Assistant Professor
University of Nebraska-Lincoln
Center on Children, Families, and the Law
206 S. 13th Street, Suite 1000
Lincoln, NE 68588-0227

(402) 472-9812 Office
(402) 472-8412 Fax

IPAC-List at ipacweb.org

Confidentiality Notice: All information in this communication,
including any files or attachments, is intended for the sole use of the
individual or entity to which it is addressed and may contain
information that is confidential, proprietary and/or trade secret
information entitled to protection and/or exempt from disclosure under
applicable law. If you are not the intended recipient, you are hereby
notified that any dissemination, distribution, or copying of this
communication is strictly prohibited. If you have received this
communication in error, please notify the sender by return email and
delete this communication from your system. Thank you for your
IPAC-List at ipacweb.org


This e-mail message is intended for the named recipient(s) and may
contain information that is privileged, confidential and/or exempt from
disclosure under applicable law. Unauthorized disclosure, copying or
re-transmission is prohibited. If you are not a named recipient or not
authorized by the named recipient(s), or if you have received this
e-mail in error, then please notify the sender immediately and delete
the message and any copies.
Ce courriel est destiné exclusivement au destinataire mentionné en titre
et peut contenir de l'information privilégiée, confidentielle ou
soustraite à la communication aux termes des lois applicables. Toute
divulgation non autorisée, toute reproduction ou réacheminement est
interdit. Si vous n'êtes pas le destinataire de ce courriel, ou n'êtes
pas autorisé par le destinataire visé, ou encore, si vous l'avez reçu
par erreur, veuillez le mentionner immédiatement à l'expéditeur et
supprimer le courriel et les copies.

More information about the IPAC-List mailing list