[IPAC-List] Divergent views of validation

Pluta, Paul ppluta at hr.lacounty.gov
Wed Sep 1 11:07:07 EDT 2010

In the real world, truth evolves. Today's truth is tomorrow's fallacy.
Borsboom contends that validity is a property of a test. Let us say,
for the sake of argument, that a test does measure what it purports to
measure (i.e., there is evidence to support the construct validity of
the test). Let us further assume that the test measures "mathematical
ability". Now, let's assume that we use the scores of examinees on this
test to predict who will be more or less likely to exhibit violent
behaviors in social contexts. Can we state that the validity of
inferences made on the basis of test scores is valid because the test is
valid? This is why we also look at something we call criterion-related
validity evidence. The validity of the inferences we make from test
scores is just as important as the construct validity of the test
itself. Tests do not possess validity in a vacuum.

Paul E. Pluta, ABD, SPHR
Human Resources Analyst
Los Angeles County Department of Human Resources
Talent Management Division

-----Original Message-----
From: ipac-list-bounces at ipacweb.org
[mailto:ipac-list-bounces at ipacweb.org] On Behalf Of Joel Wiesen
Sent: Wednesday, September 01, 2010 7:41 AM
To: Lance Seberhagen
Cc: IPAC-List
Subject: Re: [IPAC-List] Divergent views of validation

Hi Lance,

Borsboom holds that validity is a property of a test, not a property of
test score interpretations or uses. He says a test is valid to the
extent that it measures what it should measure. He says validity is not

a function of evidence but a function of truth. He suggests truth is
the commonsense view of validity.

He gives as one example of the illogic of property of test score use,
17th century measures of phlogiston (i.e., the fire-stuff in material
that burns, and whose weight can be inferred by the decrease in weight
between the material and the ashes). Measures of phlogiston were
supported by scientific theory of the time. That is, they were
supported by a nomological net. However, after 100 years, burning was
found to be better described as the combination of material with oxygen.

The measures of phlogiston were invalid, even tho their use was
supported by evidence. He says it is illogical to use a definition of
validity that has scientists saying at one point in time that measures
of phlogiston are valid but later saying they are invalid.

He further holds that the nomological net envisioned by construct
validation just does not exist in psychology. He says our theories are
too weak to be called a nomological net. For example, physicists can
explain how a hygrometer functions. Psychometricians cannot explain why

our tests function as they do. We cannot predict if one question will
be harder than another, or more valid than another, or which questions
will correlate with other questions or to what degree. Physicists have
different types of thermometers (mercury, thermistors, etc.) but they
all measure temperature as defined by physical laws. We cannot say the
same of our measures of psychological KSAPs; two tests of intelligence
may be measuring very different abilities.

Mitchell faults us (in part) for assuming test questions each measure
one unit. Mitchell suggests some questions reflect more of the ability
being measured than other questions. This calls into question our use
of tests as interval scales, and our calculation of total test scores.

This is a very brief summary of their longer, more complete logical
analyses and critiques of validity.


- -
Joel P. Wiesen, Ph.D., Director
Applied Personnel Research
62 Candlewood Road
Scarsdale, NY 10583-6040
(617) 244-8859

Note: This e-mail and any attachments may contain confidential and/or
legally privileged information. Please do not forward any contents
without permission. If you have received this message in error please
destroy all copies, completely remove it from your computer, and notify
the sender. Thank you.

Lance Seberhagen wrote:

> Joel:


> Does "validity" in educational testing mean the same as "validity" in

> employment testing (or psychological testing in general)? If the

> current APA Standards are "flawed" or even "useless," it's possible


> someone is trying to justify something that is not valid under

> traditional concepts of validity. Maybe the APA Standards need to be

> revised, maybe not, depending on the scientific evidence to support


> proposed changes.


> Lance Seberhagen, Ph.D.

> Seberhagen & Associates

> 9021 Trailridge Ct

> Vienna, VA 22182

> Tel 703-790-0796

> www.seberhagen.com



> On 8/31/2010 7:33 PM, Joel Wiesen wrote:

>> For those who like considering non-conventional views...


>> Chapters by Joel Michell and Denny Borsboom in an edited book on

>> Validity by Lissitz suggest that the prevalent definition of


>> as in the current APA Standards, is flawed, perhaps useless. A brief

>> review is available at http://www.edrev.info/reviews/rev965.pdf


>> Joel



IPAC-List at ipacweb.org

More information about the IPAC-List mailing list