[IPAC-List] Divergent views of validation

[Pluta, Paul]

In the real world, truth evolves. Today's truth is tomorrow's fallacy.
Borsboom contends that validity is a property of a test.  Let us say,
for the sake of argument, that a test does measure what it purports to
measure (i.e., there is evidence to support the construct validity of
the test).  Let us further assume that the test measures "mathematical
ability".  Now, let's assume that we use the scores of examinees on this
test to predict who will be more or less likely to exhibit violent
behaviors in social contexts.  Can we state that the validity of
inferences made on the basis of test scores is valid because the test is
valid?  This is why we also look at something we call criterion-related
validity evidence.  The validity of the inferences we make from test
scores is just as important as the construct validity of the test
itself.  Tests do not possess validity in a vacuum.

Paul E. Pluta, ABD, SPHR
Human Resources Analyst 
Los Angeles County Department of Human Resources
Talent Management Division

-----Original Message-----
From: ipac-list-bounces at ipacweb.org
[mailto:ipac-list-bounces at ipacweb.org] On Behalf Of Joel Wiesen
Sent: Wednesday, September 01, 2010 7:41 AM
To: Lance Seberhagen
Cc: IPAC-List
Subject: Re: [IPAC-List] Divergent views of validation

Hi Lance,

Borsboom holds that validity is a property of a test, not a property of 
test score interpretations or uses.  He says a test is valid to the 
extent that it measures what it should measure.  He says validity is not

a function of evidence but a function of truth.  He suggests truth is 
the commonsense view of validity.

He gives as one example of the illogic of property of test score use, 
17th century measures of phlogiston (i.e., the fire-stuff in material 
that burns, and whose weight can be inferred by the decrease in weight 
between the material and the ashes).  Measures of phlogiston were 
supported by scientific theory of the time.  That is, they were 
supported by a nomological net.  However, after 100 years, burning was 
found to be better described as the combination of material with oxygen.

  The measures of phlogiston were invalid, even tho their use was 
supported by evidence.  He says it is illogical to use a definition of 
validity that has scientists saying at one point in time that measures 
of phlogiston are valid but later saying they are invalid.

He further holds that the nomological net envisioned by construct 
validation just does not exist in psychology.  He says our theories are 
too weak to be called a nomological net.  For example, physicists can 
explain how a hygrometer functions.  Psychometricians cannot explain why

our tests function as they do.  We cannot predict if one question will 
be harder than another, or more valid than another, or which questions 
will correlate with other questions or to what degree.  Physicists have 
different types of thermometers (mercury, thermistors, etc.) but they 
all measure temperature as defined by physical laws.  We cannot say the 
same of our measures of psychological KSAPs; two tests of intelligence 
may be measuring very different abilities.

Mitchell faults us (in part) for assuming test questions each measure 
one unit.  Mitchell suggests some questions reflect more of the ability 
being measured than other questions.  This calls into question our use 
of tests as interval scales, and our calculation of total test scores.

This is a very brief summary of their longer, more complete logical 
analyses and critiques of validity.

Joel


- -
Joel P. Wiesen, Ph.D., Director
Applied Personnel Research
62 Candlewood Road
Scarsdale, NY 10583-6040
(617) 244-8859
http://appliedpersonnelresearch.com




Note: This e-mail and any attachments may contain confidential and/or 
legally privileged information. Please do not forward any contents 
without permission. If you have received this message in error please 
destroy all copies, completely remove it from your computer, and notify 
the sender. Thank you.



Lance Seberhagen wrote:

>   Joel:

> 

> Does "validity" in educational testing mean the same as "validity" in 

> employment testing (or psychological testing in general)?  If the 

> current APA Standards are "flawed" or even "useless," it's possible

that 

> someone is trying to justify something that is not valid under 

> traditional concepts of validity.  Maybe the APA Standards need to be 

> revised, maybe not, depending on the scientific evidence to support

the 

> proposed changes. 

> 

> Lance Seberhagen, Ph.D.

> Seberhagen & Associates

> 9021 Trailridge Ct

> Vienna, VA 22182

> Tel 703-790-0796

> www.seberhagen.com

> 

> 

> On 8/31/2010 7:33 PM, Joel Wiesen wrote:

>> For those who like considering non-conventional views...

>>

>> Chapters by Joel Michell and Denny Borsboom in an edited book on 

>> Validity by Lissitz suggest that the prevalent definition of

validity, 

>> as in the current APA Standards, is flawed, perhaps useless.  A brief



>> review is available at http://www.edrev.info/reviews/rev965.pdf

>>

>> Joel

>>

>>

_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org
http://www.ipacweb.org/mailman/listinfo/ipac-list