[IPAC-List] Michael McDaniel's Reference to the so-called Validity-Diversity Dilemma

Sun Jun 5 13:20:31 EDT 2016

Hi Lance,

I'm sure you're right.  However, I think just about any system can be rigged.  Also, the job element method probably should not have been used for a job like police officer or state trooper.  On the other hand, is there any evidence of the job element method resulting in bogus KSA's for federal blue-collar occupations?  I haven't heard anything to that effect.  The elements were usually very obvious.  If you see knowledge of electrical theory for an electrician, it's pretty clear they have a lot of different tasks that probably involve some degree of knowledge of electrical theory (or principles, if you will).  That was my experience from reviewing out X-188C.  No surprises, usually about 5-6 job elements.

I would expect something far beyond this for a State Trooper, so that's why I'm wondering about using the job element method for that position.

There were other heavy hitters involved with the job element method and research, including as I now recall, Professor Robert Wherry from Ohio State University (multiple regression analysis).  There's also published research supporting the job element method for use in devising training & experience ratings.  I'm not sure about recent publications, perhaps you are...but there's an article on this by McDaniel, Schmidt and Hunter (1988), A Meta-Analysis of the Validity of Methods for Rating Training and Experience in Personnel Selection.  The long and short of this is that I don't think we can scientifically trash the job element method because it went wrong as applied for the New York State Troopers, but that's just my opinion.

If you're the I/O Psychologist and asleep at the wheel, a lot can go wrong, I'm sure.  I don't think anyone should have blind faith in any method.  Even when tasks are identified, the listing that gets developed can greatly vary from one study to the next, and tasks can vary in their breadth, from very narrow to very broadly worded statements, and this can affect the types of KSA's that are identified.  I don't know...I'm not a big advocate of content validity.

Do you remember who was the leader for PRDC on that study?  I wasn't following this at the time...but if you tell me that someone on the level of a Frank Schmidt got faked out by SME's, I'll be shocked.

----- Original Message ----- 
  From: Lance Seberhagen 
  To: Richard Joines ; IPAC List 
  Sent: Sunday, June 05, 2016 9:21 AM
  Subject: Re: [IPAC-List] Michael McDaniel's Reference to the so-called Validity-Diversity Dilemma

  Hi Richard:

  PRDC did not publicize them very well, but PRDC produced wonderful research reports on employment testing and related topics.  These reports were based on good science, and I always updated my collection whenever I visited PRDC.  However, PRDC's job element method of job analysis was not good science.  The job element method was unreliable because failed to link tasks to KSAs, as well as collect date from independent sources, as demonstrated by PRDC's ill-fated New York State Police exam.  NYS contracted with PRDC to develop the exam.  PRDC attempted to develop a content valid exam using the job element method.  SMEs (state troopers) wanted to keep women out of their ranks, so they created bogus KSAs (e.g., ability to shoot a shotgun over the roof of a police car) to screen out women.  PRDC incorporated many of these bogus KSAs into the exam because the job element method failed to catch them.  As a result, the federal court threw out the exam because it had high adverse impact and no validity, in violation of Title VII.  It was a valuable lesson for all I/Os.  KSAs should be linked to observable work behavior (e.g., job tasks), and a multi-method approach is best to ensure reliable job analysis results.  

  Lance S.   

  On 6/4/2016 8:08 PM, Richard Joines wrote:

    Hi Lance,

    That was a long time ago, Lance.  The PRDC group was terrific.  I got to attend periodic briefings by people like Bill Gorham and our hero in PRDC, Frank Schmidt, as well as others who were terrifically skilled and knowledgeable.  They were all great.

    Regarding the Uniform Guidelines, most of what I remember is the feeling that we were under seige, and that merit hiring was being put to the test.  I remember a lot of debates internally about banding or other approaches that would expand the number of people certified, such as the rule of reliability (Michigan used this, as I recall), and while I believe we were open to more latitude than a rule of 3, our main impetus was to keep a "reasonable" certification rule (e.g., 3, 5, 7).  OPM as an agency was totally opposed to a rule of the list (for ranking devices), and totally opposed to selective certification based on race.  Beyond this, I don't have any specific recollection of a position taken on banding in writing or otherwise, but based on my understanding of our overall stance, I'm pretty sure PRDC was against banding.  The idea that tests weren't perfectly reliable or valid didn't come as news to any of our psychologists, thus the idea that tests weren't perfect didn't convince anyone that I knew that we should therefore greatly expand certification rules.  Basically, we were all committed to the concepts of individual merit and test utility.  You might say we were pretty effectively indoctrinated, or you might say we were trained very well.   Take your pick.

    Regarding job analysis provisions in the Uniform Guidelines, I know that Ernie Primoff and other adherents of our job element job analysis method weren't happy because identifying job tasks was not necessary in the job element system.  Our X-118C staffing guide listed the exam plan (the job elements) for every blue-collar occupation in the federal system, and everyone thought that the system used to arrive at those elements was very sound -- yet if you bought into the Uniform Guidelines, you had to conclude that our job element job analysis system was flawed.  I think our PRDC group thought the job element method was fine as practiced, and I agree with that.  However...we didn't hold firm on this.

    As I recall, our people were resistant to a lot of the early pressure by EEOC for organizations to have to conduct differential validity studies in order to meet federal standards.  I think that was one of their initial negotiating positions.  It was my understanding that our OPM team fought to have three co-equal validation methods, and they prevailed over the EEOC on this.  Their main argument, I was told at the time, was that most government agencies could not conduct criterion-related validity studies, much less differential validity studies, owing to sample size limitations and they wouldn't budge on the need to let government agencies use content validity in order to be in compliance.  So, while holding firm on this, I believe we did give in on things like the need to include tasks in any job analysis and link them to KSA's (thus undermining our own job element job analysis system).

    Overall, my understanding was that a series of compromises were made to reach agreement and finally issue the Uniform Guidelines.  Frankly, the process of developing the guidelines went on so long that everyone was sick of it.  So, we got some things, but agreed to other things that we didn't think were truly appropriate.

    This was a very exciting era.  PRDC was a terrific organization.  PRDC produced many outstanding publications on a variety of testing topics -- truly a great outfit under Gorham's leadership.  I just want to add that I got my initial training in assessment centers from the PRDC assessment team headed up by Dale Baker.  What a great resource they were for those of us out in the boondocks!

      ----- Original Message ----- 
      From: Lance Seberhagen 
      To: Richard Joines ; IPAC List 
      Sent: Saturday, June 04, 2016 12:55 PM
      Subject: Re: [IPAC-List] Michael McDaniel's Reference to the so-called Validity-Diversity Dilemma

      Hi Richard:

      PRDC had a great staff, and the 1970s were important in I/O history.  As a former OPM Regional Psychologist during that time, can you share any other battles that PRDC had with EEOC over the development of the Uniform Guidelines and EEO compliance?  For example, weren't Bill Gorham and PRDC opposed to category ranking (aka "banding") as a way to reduce unnecessary adverse impact?

      Lance S.

      On 6/4/2016 2:48 PM, Richard Joines wrote:

        Hi Lance:

        You're absolutely correct, and as an OPM Regional Psychologist located in S.F. during that era, we got weekly reports on the battles and skirmishes taking place with EEOC.  Our PRDC group did the best they could to block some of the more extreme measures that the EEOC wanted...one sore point with PRDC was the battle over whether or not tasks had to be identified with linkages to KSA's.  We didn't do that in our job element method for blue-collar jobs, and our method was very successful, but finally...well, you know that some compromises had to be made, and they weren't all based on the science that existed at that point in time.  People entering our field today must believe that there were some really great scientific reasons for all of the decisions reflected in the Uniform Guidelines, else why would those decisions be promulgated???

        And as I'm sure you know, there is nothing in the Uniform Guidelines about a validity-diversity dilemma, thankfully.

          ----- Original Message ----- 
          From: Lance Seberhagen 
          To: Richard Joines ; IPAC List 
          Sent: Saturday, June 04, 2016 11:06 AM
          Subject: Re: [IPAC-List] Michael McDaniel's Reference to the so-called Validity-Diversity Dilemma

          Lest we forget, didn't Bill Gorham and PRDC help to write the...

          UNIFORM GUIDELINES ON EMPLOYEE SELECTION PROCEDURES

          § 1607.1 Statement of purpose.

          A. Need for uniformity—Issuing agencies. The Federal government's need for a uniform set of principles on the question of the use of tests and other selection procedures has long been recognized. The Equal Employment Opportunity Commission, the Civil Service Commission, the Department of Labor, and the Department of Justice jointly have adopted these uniform guidelines to meet that need, and to apply the same principles to the Federal Government as are applied to other employers.

          Lance S.

          On 6/4/2016 1:50 PM, Richard Joines wrote:

            Hi Mark, 

              From the Taylor-Russell Tables: 

            If you increase the test time allowed and this reduces validity from .5 to .4 where you have a selection ratio of .20 and current percent successful on the job at 50%, you're talking approximately a 10% reduction in the overall success rate that the test produces (78% down to 73%, where 5% reduction/50% base rate=10% reduction in success rate).  Why not accept a 25% or 50 reduction? 

            You indicate it wouldn't be ok to drop validity from .6 to .2, which would basically result in dropping the success rate from 84% to 61% of those hired (46% reduction in success rate), but maybe someone else would say that's fine.  Why isn't this ok with you?  Seriously, why not? 

            You have to forgive me for how I think about these issues...I was successfully indoctrinated by OPM during the 1970's where our 80+ psychologists (in OPM's Personnel Research and Development Center), led by Bill Gorham, and greatly assisted by Frank Schmidt and others, debunked most of what was coming out of the EEOC.  Well, temporarily debunked it...looks like we've got a sequel, The EEOC Strikes Back. 

            ----- Original Message ----- From: <mhammer at 295.ca> 
            To: <ipac-list at ipacweb.org> 
            Sent: Friday, June 03, 2016 3:19 PM 
            Subject: Re: [IPAC-List] Michael McDaniel's Reference to the so-called Validity-Diversity Dilemma 

              I liked Dan's post very much.  The magnitude of the forfeiture is 
              important in considering the balance.  If bending timing drops predictive 
              validity from .5 down to .42, but improves representativeness, I'm okay 
              with that.  If we're talking a drop from .6 to .2, that's a whole other 
              story. 

              _______________________________________________________ 
              IPAC-List 
              IPAC-List at ipacweb.org 
              https://pairlist9.pair.net/mailman/listinfo/ipac-list 

            _______________________________________________________ 
            IPAC-List 
            IPAC-List at ipacweb.org 
            https://pairlist9.pair.net/mailman/listinfo/ipac-list 

               Virus-free. www.avast.com  

_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org
https://pairlist9.pair.net/mailman/listinfo/ipac-list

           Virus-free. www.avast.com  

_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org
https://pairlist9.pair.net/mailman/listinfo/ipac-list

       Virus-free. www.avast.com  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist9.pair.net/pipermail/ipac-list/attachments/20160605/1cd80555/attachment-0001.html>