[IPAC-List] Training and Experience Evaluations--the evidence

Thu May 18 16:28:54 EDT 2017

Although dated, I believe that this is most current meta-analysis on
validity of T&Es:
McDaniel, M. A., Schmidt, F. L., & Hunter, J. E. (1988). A meta-analysis of
methods for rating training and experience in personnel selection. *
Personnel Psychology*, *41*, 283-314.
<http://www.people.vcu.edu/%7Emamcdani/Publications/McDaniel,%20Schmidt,%20Hunter%20%281988%29%29%20T&E.pdf>

I no longer think that the validity estimates for the accomplishment
records in the above study are trustworthy. That distribution has failed a
publication bias test that I ran in the lasy yaer and I am told that some
of the articles partialed out age before reporting the correlation
coefficient. This study is on my list to update.

There are two common problems with T&Es:
1)  When you have a minimum qualification concerning years of experience
needed to apply, you create range restriction on experience in your
applicant pool. Then, when you try to score a T&E everyone scores about the
same on experience, so you do not differentiate the applicants very well.

2) To the extent that the T&E is primarily a length of experience measure,
often the analyst does not realize that experience is typically not
linearly related to job knowledge and job performance.  At least for job
knowledge learned on the job, most of that learning occurs in the first
several years of job experience. Thus, the job performance of someone with
five years of experience may not differ much from someone with 20 years of
experience. A relevant citation is: McDaniel, M. A., Schmidt, F. L., &
Hunter, J. E. (1988).Job experience correlates of job performance.  *Journal
of Applied Psychology, 73*, 327-330.
<http://www.people.vcu.edu/%7Emamcdani/Publications/McDaniel,%20Schmidt%20&%20Hunter%201988%20Job%20experience.pdf>

For this engineer job, I would suggest that you use a job knowledge test or
a cognitive ability test rather than a T&E. A senior engineer is an
important job. Do you really want less than the best of a selection system
to hire for an important job?

Even, if a reader of this does not care about T&Es, the linked articles
above are really handy if you are having trouble sleeping. Most people are
asleep before the third page.

Best wishes,

Mike

On Thu, May 18, 2017 at 10:54 AM, Herman Aguinis <haguinis at email.gwu.edu>
wrote:

> Richard,
>
> Thank you very much for your kind words! In fact, I benefited from an
> earlier edition as a doctoral student myself… This will be the third
> edition that Wayne and I are co-authoring (beginning with the 6th edition
> in 2005). We are conducting a massive literature review involving about
> 11,000 articles from almost 30 journals and the 8th edition will include
> many important updates and improvements.
>
> All the best,
>
> --Herman.
>
>
>
> *Herman Aguinis, Ph.D.*
>
> Avram Tucker Distinguished Scholar and Professor of Management
>
> George Washington University School of Business
>
> 2201 G Street, NW
>
> Washington, DC 20052
>
> *http://hermanaguinis.com/ <http://hermanaguinis.com/> *
>
>
>
> *From:* richard carter [mailto:rickcarter1957 at yahoo.com]
> *Sent:* Thursday, May 18, 2017 10:09 AM
>
> *To:* haguinis at gwu.edu; IPAC-List at ipacweb.org
> *Subject:* Re: [IPAC-List] Training and Experience Evaluations--the
> evidence
>
>
>
> I"m glad to hear about the new edition.  A much earlier edition was a text
> book of mine in grad school more than 30 years ago.  I found it
> informative, inclusive, and well-written.  You can't say that about all
> scholarly writings.
>
>
>
> On Thursday, May 18, 2017 8:37 AM, Herman Aguinis <haguinis at email.gwu.edu>
> wrote:
>
>
>
> Richard,
> Yes, precisely. Wayne and I are in the process of writing the 8th
> edition, which will be available in 2018. So, the paragraphs below reflect
> the updates.
>
> Regards,
>
> --Herman.
>
>
>
> *Herman Aguinis, Ph.D.*
>
> Avram Tucker Distinguished Scholar and Professor of Management
>
> George Washington University School of Business
>
> 2201 G Street, NW
>
> Washington, DC 20052
>
> *http://hermanaguinis.com/ <http://hermanaguinis.com/> *
>
>
>
> *From:* richard carter [mailto:rickcarter1957 at yahoo.com
> <rickcarter1957 at yahoo.com>]
> *Sent:* Thursday, May 18, 2017 9:34 AM
> *To:* haguinis at gwu.edu; IPAC-List at ipacweb.org
> *Subject:* Re: [IPAC-List] Training and Experience Evaluations--the
> evidence
>
>
>
> How does a book published in 2011 cite a journal article in 2016?  Do
> y'all have a new edition in the works?  just wondering . .
>
>
>
> Rick Carter
>
> New Orleans Civil Service (mostly retired)
>
>
>
> On Thursday, May 18, 2017 7:55 AM, Herman Aguinis <haguinis at email.gwu.edu>
> wrote:
>
>
>
> Adele,
>
>
>
> The following text is from Chapter 12 in Cascio, W. F, and Aguinis H.
> (2011). Applied psychology in human resource management
> <https://www.pearsonhighered.com/program/Cascio-Applied-Psychology-in-Human-Resource-Management-7th-Edition/PGM248374.html>,
> 7th edition. Upper Saddle River, NJ: Pearson Prentice Hall.
>
>
>
> *Evaluating job experience is not as easy as one may think because
> experience includes both qualitative and quantitative components that
> interact and accrue over time (Aguinis, O’Boyle, Gonzalez-Mulé, & Joo,
> 2016); hence, work experience is multidimensional and temporally dynamic
> (Tesluk & Jacobs, 1998). However, using experience as a predictor of future
> performance can pay off. Specifically, a study including more than 800 U.S.
> Air Force enlisted personnel indicated that ability and experience seem to
> have linear and noninteractive effects (Lance & Bennett, 2000). Another
> study that also used military personnel showed that work experience items
> predict performance above and beyond cognitive abilities and personality
> (Jerry & Borman, 2002). These findings explain why the results of a survey
> of more than 200 staffing professionals of the National Association of
> Colleges and Employers revealed that experienced hires were evaluated more
> highly than new graduates on most characteristics (Rynes, Orlitzky, &
> Bretz, 1997).*
>
>
>
> *An empirical comparison of four methods for evaluating work experience
> indicated that the “behavioral consistency” method showed the highest mean
> validity (.45) (McDaniel, Schmidt, & Hunter, 1988). This method requires
> applicants to describe their major achievements in several job-related
> areas. These areas are behavioral dimensions rated by supervisors as
> showing maximal differences between superior and minimally acceptable
> performers. The applicants’ achievement statements are then evaluated using
> anchored rating scales. The anchors are achievement descriptors whose
> values along a behavioral dimension have been determined reliably by
> subject matter experts.*
>
>
>
> *A similar approach to the evaluation of training and experience, one most
> appropriate for selecting professionals, is the accomplishment record (AR)
> method (Hough, 1984). A comment frequently heard from professionals is “My
> record speaks for itself.” The AR is an objective method for evaluating
> those records. It is a type of biodata/maximum performance/self-report
> instrument that appears to tap a component of an individual’s history that
> is not measured by typical biographical inventories. It correlates
> essentially zero with aptitude test scores, honors, grades, and prior
> activities and interests.*
>
>
>
> *Development of the AR begins with the collection of critical incidents to
> identify important dimensions of job performance. Then rating principles
> and scales are developed for rating an individual’s set of job-relevant
> achievements. The method yields (1) complete definitions of the important
> dimensions of the job, (2) summary principles that highlight key
> characteristics to look for when determining the level of achievement
> demonstrated by an accomplishment, (3) actual examples of accomplishments
> that job experts agree represent various levels of achievement, and (4)
> numerical equivalents that allow the accomplishments to be translated into
> quantitative indexes of achievement. When the AR was applied in a sample of
> 329 attorneys, the reliability of the overall performance ratings was a
> respectable .82, and the AR demonstrated a validity of .25. Moreover, the
> method appears to be fair for females, minorities, and white males.*
>
>
>
> I hope this helps!
>
>
>
> All the best,
>
>
>
> --Herman.
>
>
>
> *Herman Aguinis, Ph.D.*
>
> Avram Tucker Distinguished Scholar and Professor of Management
>
> George Washington University School of Business
>
> 2201 G Street, NW
>
> Washington, DC 20052
>
> *http://hermanaguinis.com/ <http://hermanaguinis.com/> *
>
>
>
> *From:* IPAC-List [mailto:ipac-list-bounces at ipacweb.org] *On Behalf Of *Demooy,
> Adele
> *Sent:* Wednesday, May 17, 2017 1:36 PM
> *To:* 'IPAC-List at ipacweb.org' <IPAC-List at ipacweb.org>
> *Subject:* [IPAC-List] Training and Experience Evaluations
>
>
>
> We do not have a lot of experience using training and experience
> evaluations at the District but would like to consider using a T&E for the
> selection process for senior level engineers (PEs) in combination with a
> structured oral test.  If anyone has a sample T&E they could share with me,
> I would be most appreciative.
>
>
>
> Look forward to seeing you in Birmingham.
>
>
>
>
>
> Adele De Mooy
>
> Employment Manager
>
> The Metropolitan Water Reclamation District of Greater Chicago
>
> 111 E. Erie
>
> Chicago, IL 60660
>
> (312) 751-5172
>
>
>
>
>
>
>
> _______________________________________________________
> IPAC-List
> IPAC-List at ipacweb.org
> https://pairlist9.pair.net/mailman/listinfo/ipac-list
>
>
>
> _______________________________________________________
> IPAC-List
> IPAC-List at ipacweb.org
> https://pairlist9.pair.net/mailman/listinfo/ipac-list
>
>

-- 
Michael A. McDaniel, Ph.D.
Work Skills First, Inc.
Voice: 804-277-9730
E-Mail: McDaniel at WorkSkillsFirst.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist9.pair.net/pipermail/ipac-list/attachments/20170518/8f7e28db/attachment-0001.html>