[IPAC-List] Divergent views of validation
jwiesen at appliedpersonnelresearch.com
Wed Sep 1 10:40:41 EDT 2010
Borsboom holds that validity is a property of a test, not a property of
test score interpretations or uses. He says a test is valid to the
extent that it measures what it should measure. He says validity is not
a function of evidence but a function of truth. He suggests truth is
the commonsense view of validity.
He gives as one example of the illogic of property of test score use,
17th century measures of phlogiston (i.e., the fire-stuff in material
that burns, and whose weight can be inferred by the decrease in weight
between the material and the ashes). Measures of phlogiston were
supported by scientific theory of the time. That is, they were
supported by a nomological net. However, after 100 years, burning was
found to be better described as the combination of material with oxygen.
The measures of phlogiston were invalid, even tho their use was
supported by evidence. He says it is illogical to use a definition of
validity that has scientists saying at one point in time that measures
of phlogiston are valid but later saying they are invalid.
He further holds that the nomological net envisioned by construct
validation just does not exist in psychology. He says our theories are
too weak to be called a nomological net. For example, physicists can
explain how a hygrometer functions. Psychometricians cannot explain why
our tests function as they do. We cannot predict if one question will
be harder than another, or more valid than another, or which questions
will correlate with other questions or to what degree. Physicists have
different types of thermometers (mercury, thermistors, etc.) but they
all measure temperature as defined by physical laws. We cannot say the
same of our measures of psychological KSAPs; two tests of intelligence
may be measuring very different abilities.
Mitchell faults us (in part) for assuming test questions each measure
one unit. Mitchell suggests some questions reflect more of the ability
being measured than other questions. This calls into question our use
of tests as interval scales, and our calculation of total test scores.
This is a very brief summary of their longer, more complete logical
analyses and critiques of validity.
Joel P. Wiesen, Ph.D., Director
Applied Personnel Research
62 Candlewood Road
Scarsdale, NY 10583-6040
Note: This e-mail and any attachments may contain confidential and/or
legally privileged information. Please do not forward any contents
without permission. If you have received this message in error please
destroy all copies, completely remove it from your computer, and notify
the sender. Thank you.
Lance Seberhagen wrote:
> Does "validity" in educational testing mean the same as "validity" in
> employment testing (or psychological testing in general)? If the
> current APA Standards are "flawed" or even "useless," it's possible that
> someone is trying to justify something that is not valid under
> traditional concepts of validity. Maybe the APA Standards need to be
> revised, maybe not, depending on the scientific evidence to support the
> proposed changes.
> Lance Seberhagen, Ph.D.
> Seberhagen & Associates
> 9021 Trailridge Ct
> Vienna, VA 22182
> Tel 703-790-0796
> On 8/31/2010 7:33 PM, Joel Wiesen wrote:
>> For those who like considering non-conventional views...
>> Chapters by Joel Michell and Denny Borsboom in an edited book on
>> Validity by Lissitz suggest that the prevalent definition of validity,
>> as in the current APA Standards, is flawed, perhaps useless. A brief
>> review is available at http://www.edrev.info/reviews/rev965.pdf
More information about the IPAC-List