[IPAC-List] Divergent views of validation

[Shekerjian, Rene]

I don't know. Maybe I am off on the wrong track, but it seems to me that it isn't quite phlogiston. While I can't guarantee job performance based on test score, I should be able, in many cases, to say how candidates relate to one another at least in terms of certain required knowledges and abilities. 

If a person needs to know certain laws, or have knowledge of algebra, or have the ability to work with numeric information that is presented in tables, or have the ability to correctly apply the rules of grammar and punctuation, I can test for that. Can I be assured that the person with the highest score will be the best performer? No. 

But I can say whether a person has the knowledge or ability needed to perform certain tasks and activities. And I should be able to say that a person with a low score on a test of the laws, rules, and regulations governing banking activities will have more trouble being an effective bank examiner than a person with a high score, all other things being equal. 

And given enough time and money, I could probably come up with tests that would do a good job of ranking people in terms of their likely performance on the job to be held. Heck, a bunch of you have given presentations talking about how to do that. My problem is that it is very expensive and time consuming. So we are down to trying to get the best bang for the buck. 

Now I have to get back to my experiments trying to transmute soda cans into gold...

René

René Shekerjian | Testing Services Division | NYS Department of Civil Service | 518-474-3778
==========================================================================


-----Original Message-----
From: Jeff Feuquay [mailto:jfeuquay at gmail.com] 
Sent: Wednesday, September 01, 2010 4:22 PM
To: IPAC-List at ipacweb.org
Subject: Re: [IPAC-List] Divergent views of validation

If Borsboom asserts validity is what a test "should" measure, then it seems
he is discussing a concept quite different from what most assessment folk
think of as validity, i.e., whether a test measures what it purports to
measure. That idiosyncratic definition would quite reasonably be expected to
lead him down the rabbit hole to a very different world.

Some of the discussion seems to imply we are again caught in the trap of
hoping/believing that correlation reflects causation. Ah, were it only so.
Then, maybe we could say that whatever our tests are measuring (not
predicting . . . actually measuring) cause the future behavior sought. But,
we continue to measure the employee version of phlogiston. We've got some
understanding of which characteristics are likely to result in improved
performance, but it remains rudimentary. I mean, as Borsboom knows, it would
be a whole lot simpler if we were measuring temperature, something we now
undersatnd way better than we understand ourselves.

As to the level of measurment: Unless one has gone through a thorough
scaling process, certainly variations in item difficulty make us fall short
of interval mesurement. But on a practical level, how short of interval is
too short? When some jurisdictions continue to round T&E scores to
thousandths of a point and then take the top 3, is ordinal versus interval
or even ratio a big issue? Yeah, it is. Scaling is an issue, significant
digits is an issue, but most scales and most of the stats we commonly use
are extremely robust. We may bend the prediction but we're unlikely to break
it.

Given another hundred years to work on this, measurement of employee
phlogistan could conceivably evolve into something more direct and more
definitive . . . I mean, everybody knows that variations in the long arm of
chromsome 6 of the human genome account for 4% of the variance in
intelligence. See, we're on the way.
Jeff
-----------------------------------------
Dr. Jeffrey P Feuquay, I/O Psychologist & Attorney
Managing Consultant, Psychology-Law Center, LLC
108 W. Walnut, Nevada, Mo 64772
ofc: 417-667-5076   cell: 417-549-0997


On Wed, Sep 1, 2010 at 11:37 AM, Mark Hammer <Mark.Hammer at psc-cfp.gc.ca>wrote:


> Would it be wrong to suggest that this is one of those perpetual

> discrepancies between theoreticians and clinicians?  In this case, the

> discrepancy is between test/measurement specialists (the theoreticians),

> and test-users (the clinicians).

>

> Mark Hammer

> Ottawa

>

> >

> This e-mail message is intended for the named recipient(s) and may

> contain information that is privileged, confidential and/or exempt from

> disclosure under applicable law. Unauthorized disclosure, copying or

> re-transmission is prohibited. If you are not a named recipient or not

> authorized by the named recipient(s), or if you have received this

> e-mail in error, then please notify the sender immediately and delete

> the message and any copies.

>  >

> Ce courriel est destiné exclusivement au destinataire mentionné en titre

> et peut contenir de l'information privilégiée, confidentielle ou

> soustraite à la communication aux termes des lois applicables. Toute

> divulgation non autorisée, toute reproduction ou réacheminement est

> interdit. Si vous n'êtes pas le destinataire de ce courriel, ou n'êtes

> pas autorisé par le destinataire visé, ou encore, si vous l'avez reçu

> par erreur, veuillez le mentionner immédiatement à l'expéditeur et

> supprimer le courriel et les copies.

>

> _______________________________________________________

> IPAC-List

> IPAC-List at ipacweb.org

> http://www.ipacweb.org/mailman/listinfo/ipac-list

>