[IPAC-List] Michael McDaniel's Reference to the so-called Validity-Diversity Dilemma
Lance Seberhagen
sebe at erols.com
Sun Jun 5 12:21:32 EDT 2016
Hi Richard:
PRDC did not publicize them very well, but PRDC produced wonderful
research reports on employment testing and related topics. These
reports were based on good science, and I always updated my collection
whenever I visited PRDC. However, PRDC's job element method of job
analysis was not good science. The job element method was unreliable
because failed to link tasks to KSAs, as well as collect date from
independent sources, as demonstrated by PRDC's ill-fated New York State
Police exam. NYS contracted with PRDC to develop the exam. PRDC
attempted to develop a content valid exam using the job element method.
SMEs (state troopers) wanted to keep women out of their ranks, so they
created bogus KSAs (e.g., ability to shoot a shotgun over the roof of a
police car) to screen out women. PRDC incorporated many of these bogus
KSAs into the exam because the job element method failed to catch them.
As a result, the federal court threw out the exam because it had high
adverse impact and no validity, in violation of Title VII. It was a
valuable lesson for all I/Os. KSAs should be linked to observable work
behavior (e.g., job tasks), and a multi-method approach is best to
ensure reliable job analysis results.
Lance S.
On 6/4/2016 8:08 PM, Richard Joines wrote:
> Hi Lance,
> That was a long time ago, Lance. The PRDC group was terrific. I got
> to attend periodic briefings by people like Bill Gorham and our hero
> in PRDC, Frank Schmidt, as well as others who were terrifically
> skilled and knowledgeable. They were all great.
> Regarding the Uniform Guidelines, most of what I remember is the
> feeling that we were under seige, and that merit hiring was being put
> to the test. I remember a lot of debates internally about banding or
> other approaches that would expand the number of people certified,
> such as the rule of reliability (Michigan used this, as I recall), and
> while I believe we were open to more latitude than a rule of 3, our
> main impetus was to keep a "reasonable" certification rule (e.g., 3,
> 5, 7). OPM as an agency was totally opposed to a rule of the list
> (for ranking devices), and totally opposed to selective certification
> based on race. Beyond this, I don't have any specific recollection of
> a position taken on banding in writing or otherwise, but based on my
> understanding of our overall stance, I'm pretty sure PRDC was against
> banding. The idea that tests weren't perfectly reliable or
> valid didn't come as news to any of our psychologists, thus the idea
> that tests weren't perfect didn't convince anyone that I knew that we
> should therefore greatly expand certification rules. Basically, we
> were all committed to the concepts of individual merit and test
> utility. You might say we were pretty effectively indoctrinated, or
> you might say we were trained very well. Take your pick.
> Regarding job analysis provisions in the Uniform Guidelines, I know
> that Ernie Primoff and other adherents of our job element job analysis
> method weren't happy because identifying job tasks was not necessary
> in the job element system. Our X-118C staffing guide listed the exam
> plan (the job elements) for every blue-collar occupation in the
> federal system, and everyone thought that the system used to arrive at
> those elements was very sound -- yet if you bought into the Uniform
> Guidelines, you had to conclude that our job element job
> analysis system was flawed. I think our PRDC group thought the job
> element method was fine as practiced, and I agree with that.
> However...we didn't hold firm on this.
> As I recall, our people were resistant to a lot of the early pressure
> by EEOC for organizations to have to conduct differential validity
> studies in order to meet federal standards. I think that was one of
> their initial negotiating positions. It was my understanding that our
> OPM team fought to have three co-equal validation methods, and they
> prevailed over the EEOC on this. Their main argument, I was told at
> the time, was that most government agencies could not conduct
> criterion-related validity studies, much less differential validity
> studies, owing to sample size limitations and they wouldn't budge on
> the need to let government agencies use content validity in order to
> be in compliance. So, while holding firm on this, I believe we
> did give in on things like the need to include tasks in any job
> analysis and link them to KSA's (thus undermining our own job element
> job analysis system).
> Overall, my understanding was that a series of compromises were made
> to reach agreement and finally issue the Uniform Guidelines. Frankly,
> the process of developing the guidelines went on so long that everyone
> was sick of it. So, we got some things, but agreed to other things
> that we didn't think were truly appropriate.
> This was a very exciting era. PRDC was a terrific organization. PRDC
> produced many outstanding publications on a variety of testing topics
> -- truly a great outfit under Gorham's leadership. I just want to add
> that I got my initial training in assessment centers from the PRDC
> assessment team headed up by Dale Baker. What a great resource they
> were for those of us out in the boondocks!
>
> ----- Original Message -----
> *From:* Lance Seberhagen <mailto:sebe at erols.com>
> *To:* Richard Joines <mailto:mpscorp at value.net> ; IPAC List
> <mailto:ipac-list at ipacweb.org>
> *Sent:* Saturday, June 04, 2016 12:55 PM
> *Subject:* Re: [IPAC-List] Michael McDaniel's Reference to the
> so-called Validity-Diversity Dilemma
>
> Hi Richard:
>
> PRDC had a great staff, and the 1970s were important in I/O
> history. As a formerOPM Regional Psychologist during that time,
> can you share any other battles that PRDC had with EEOC over the
> development of the Uniform Guidelines and EEO compliance? For
> example, weren't Bill Gorham and PRDC opposed to category ranking
> (aka "banding") as a way to reduce unnecessary adverse impact?
>
> Lance S.
>
>
> On 6/4/2016 2:48 PM, Richard Joines wrote:
>> Hi Lance:
>> You're absolutely correct, and as an OPM Regional Psychologist
>> located in S.F. during that era, we got weekly reports on the
>> battles and skirmishes taking place with EEOC. Our PRDC group
>> did the best they could to block some of the more extreme
>> measures that the EEOC wanted...one sore point with PRDC was the
>> battle over whether or not tasks had to be identified with
>> linkages to KSA's. We didn't do that in our job element method
>> for blue-collar jobs, and our method was very successful, but
>> finally...well, you know that some compromises had to be made,
>> and they weren't all based on the science that existed at that
>> point in time. People entering our field today must believe that
>> there were some really great scientific reasons for all of
>> the decisions reflected in the Uniform Guidelines, else why would
>> those decisions be promulgated???
>> And as I'm sure you know, there is nothing in the Uniform
>> Guidelines about a validity-diversity dilemma, thankfully.
>>
>> ----- Original Message -----
>> *From:* Lance Seberhagen <mailto:sebe at erols.com>
>> *To:* Richard Joines <mailto:mpscorp at value.net> ; IPAC List
>> <mailto:ipac-list at ipacweb.org>
>> *Sent:* Saturday, June 04, 2016 11:06 AM
>> *Subject:* Re: [IPAC-List] Michael McDaniel's Reference to
>> the so-called Validity-Diversity Dilemma
>>
>> Lest we forget, didn't Bill Gorham and PRDC help to write the...*
>> *
>>
>> *UNIFORM GUIDELINES ON EMPLOYEE SELECTION PROCEDURES*
>>
>> § 1607.1Statement of purpose.
>>
>> A./Need for uniformity—Issuing agencies./The Federal
>> government's need for a uniform set of principles on the
>> question of the use of tests and other selection procedures
>> has long been recognized. The Equal Employment Opportunity
>> Commission, _/*the Civil Service Commission*/_, the
>> Department of Labor, and the Department of Justice jointly
>> have adopted these uniform guidelines to meet that need, and
>> to apply the same principles to the Federal Government as are
>> applied to other employers.
>>
>> Lance S.
>>
>> On 6/4/2016 1:50 PM, Richard Joines wrote:
>>> Hi Mark,
>>>
>>>> From the Taylor-Russell Tables:
>>>
>>> If you increase the test time allowed and this reduces
>>> validity from .5 to .4 where you have a selection ratio of
>>> .20 and current percent successful on the job at 50%, you're
>>> talking approximately a 10% reduction in the overall success
>>> rate that the test produces (78% down to 73%, where 5%
>>> reduction/50% base rate=10% reduction in success rate). Why
>>> not accept a 25% or 50 reduction?
>>>
>>> You indicate it wouldn't be ok to drop validity from .6 to
>>> .2, which would basically result in dropping the success
>>> rate from 84% to 61% of those hired (46% reduction in
>>> success rate), but maybe someone else would say that's
>>> fine. Why isn't this ok with you? Seriously, why not?
>>>
>>> You have to forgive me for how I think about these
>>> issues...I was successfully indoctrinated by OPM during the
>>> 1970's where our 80+ psychologists (in OPM's Personnel
>>> Research and Development Center), led by Bill Gorham, and
>>> greatly assisted by Frank Schmidt and others, debunked most
>>> of what was coming out of the EEOC. Well, temporarily
>>> debunked it...looks like we've got a sequel, The EEOC
>>> Strikes Back.
>>>
>>>
>>>
>>> ----- Original Message ----- From: <mhammer at 295.ca>
>>> To: <ipac-list at ipacweb.org>
>>> Sent: Friday, June 03, 2016 3:19 PM
>>> Subject: Re: [IPAC-List] Michael McDaniel's Reference to the
>>> so-called Validity-Diversity Dilemma
>>>
>>>
>>>> I liked Dan's post very much. The magnitude of the
>>>> forfeiture is
>>>> important in considering the balance. If bending timing
>>>> drops predictive
>>>> validity from .5 down to .42, but improves
>>>> representativeness, I'm okay
>>>> with that. If we're talking a drop from .6 to .2, that's a
>>>> whole other
>>>> story.
>>>>
>>>> _______________________________________________________
>>>> IPAC-List
>>>> IPAC-List at ipacweb.org
>>>> https://pairlist9.pair.net/mailman/listinfo/ipac-list
>>>>
>>>
>>>
>>> _______________________________________________________
>>> IPAC-List
>>> IPAC-List at ipacweb.org
>>> https://pairlist9.pair.net/mailman/listinfo/ipac-list
>>>
>>
>>
>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient&utm_term=icon>
>> Virus-free. www.avast.com
>>
>>
>>
>> _______________________________________________________
>> IPAC-List
>> IPAC-List at ipacweb.org
>> https://pairlist9.pair.net/mailman/listinfo/ipac-list
>
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient&utm_term=icon>
> Virus-free. www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient&utm_term=link>
>
>
>
>
> _______________________________________________________
> IPAC-List
> IPAC-List at ipacweb.org
> https://pairlist9.pair.net/mailman/listinfo/ipac-list
---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist9.pair.net/pipermail/ipac-list/attachments/20160605/40c4d34e/attachment-0001.html>
More information about the IPAC-List
mailing list