[IPAC-List] Value of cognitive ability testing for Journey-Level skilled trade jobs

Wed Mar 16 09:40:37 EDT 2016

Kevin,

This is a great discussion-Here's what I have to add:

1)      You mention that the basic aptitude and cognitive tests are valid and that older candidates who haven't taken test for a while perform as well as younger candidates with more recent experience taking tests. You also said the validity was based upon 'similar' jobs. I don't have the benefit of seeing those studies, so I'm left with questions: How similar were the jobs? The context in which you are currently using the exam is different now that is was before-how well does the context match the context of the study? I've read several validation studies for off-the shelf tests for these types of job and have been disappointed in the amount of validity evidence that is actually there. How much validity evidence are in the studies?  What I'm saying, is without actually reading the validity studies it's difficult to agree that the evidence is compelling enough to continue using this exam in this context.

2)      You pose the question,  are these experienced candidates compensating with other skills? That's a great question. Since you have a battery of tests available and not much additional costs to use them all-why not go to a compensatory model and weight the basic exam at a lower level? We use compensatory models quite often. Some candidates who are strong in the specific focused exam will pass and be hired-who may not have previously passed due to the cut- score on the basic aptitude/cognitive test. You'll have a greater range of scores from each component to compare to actual performance on the job. Maybe you'll learn which exam are in fact more predictive in this environment.

3)      As for Columbus, we don't have tons of mechanical types of jobs, but here are two examples of what we do:

a.       Auto Mechanic: Cognitive (reading a manual, problem sensitivity, parts and tool identification, safety) 40%; Performance exam (we break a mechanical item and they diagnose x2) 30%; computer skills (Word, Excel, etc) 10% and Experience (ASC certifications) 20%

b.      Plant Maintenance Mechanic: Cognitive (safety, math, mechanical aptitude, tool identification and usage) 80% and Computer 20%.
Good Luck!
Liz

Elizabeth A. Reed
Public Safety Assessment Team Manager
Columbus Civil Service Commission
______________________________

Direct: 614.645.6032

From: IPAC-List [mailto:ipac-list-bounces at ipacweb.org] On Behalf Of Reindl, Kevin
Sent: Tuesday, March 15, 2016 7:28 PM
To: Lance Seberhagen; 'IPAC-List' (ipac-list at ipacweb.org)
Subject: Re: [IPAC-List] Value of cognitive ability testing for Journey-Level skilled trade jobs

Thanks Lance for the insights. There are a fair number of experienced folks who fail the cognitive test (spatial/numerical), and interestingly, this set of candidates actually fail at a higher rate than our general applicant population for these jobs. The reality is that our hiring managers and even our HR team is starting to question the value of a test when experienced, presumably successful performers cannot pass.  There are several theories about why these folks are failing including:

*         Test is not valid (although we have evidence saying otherwise)

*         Because many of these candidates are heavily recruited, they may not be taking the test seriously (e.g., "if you really want me to work for you, why are you giving me this test to pass?")

*         Many of these candidates are older and may not be used to taking tests (we have evidence saying otherwise)

*         Since these candidates are experienced, and may not have been screened on these skills previously, perhaps they have learned to "compensate" for lack of these skills on the job and still perform successfully
My hunch is that it may be a combination of different factors...but we may never know without doing some controlled research.

There is minimal cost, and little time savings by eliminating, so it is not an efficiency/cost issue. The issues are around 1) the test screening out candidates a large number of candidates who we've sourced/recruited because of their experience, 2) the job is difficult to fill because of the low supply/high demand, and 3) the test being viewed as a "basic skills test" being used for highly skilled/technical/experienced candidates.

Kevin Reindl
Human Resources
Assessments and Organizational Insights
Office: 415-973-7013
Mobile: 619-322-3368

From: Lance Seberhagen [mailto:sebe at erols.com]
Sent: Tuesday, March 15, 2016 4:06 PM
To: Reindl, Kevin; 'IPAC-List' (ipac-list at ipacweb.org<mailto:ipac-list at ipacweb.org>)
Subject: Re: [IPAC-List] Value of cognitive ability testing for Journey-Level skilled trade jobs

Alert: This message originated outside of PG&E. Use caution when opening attachments, clicking links or responding to requests for information.
*************************************
Hi Kevin:

I have faced this question with some of my clients.  If PG&E has been giving the full test battery to experienced journey-level candidates for some time now, what percentage of these candidates pass the basic aptitude and cognitive tests?  If all candidates pass with flying colors, perhaps those tests aren't really needed.  On the other hand, if a significant number of candidates fail, perhaps those tests are needed, if PG&E wants to maintain the same quality of employees that were hired in the past.

PG&E's personality tests probably measure things like safety, teamwork, and work ethic.  These worker characteristics still have a high degree of variability among experienced candidates, so PG&E should probably continue administering those tests to experienced journey-level candidates.

So how much savings are we really talking about?  How much does it cost to administer the basic aptitude and cognitive tests?  How much extra time is required to administer these tests?  Would the benefit from dropping these tests be enough to out-weigh the risk of hiring skilled trades people who lack basic reading and math ability?  Is that risk greater for PG&E than for other employers?  What could possibly go wrong?

Lance Seberhagen, Ph.D.
Seberhagen & Associates
9021 Trailridge Court
Vienna, VA 22182
703-790-0796
www.seberhagen.com<http://cp.mcafee.com/d/avndz8sd6Qm7THLecendTdETu7fcLfc8CQrL3DCnDDAPqaabwVNVB5cSyyMye79CzBVATYvZ145mGuSGgrFN6FATqFXqF1KD4qCjpEVpjRXT7-LOtTQSk-LsKCOZvC4S3hOYVR4kRHFGTjsVkffGhBrwqrhdFCXYCyOyyMevohdTdAVPmEBCdNfgZTUk_BERGWJmk9YLxBf6UDACPysVH5Sh0l5iDkYxYiuAvxOXgg121MO0US75Obgg15zyol61kcyESm1oAo0oe4go92VAih0oC6hh0C0Gq0oD4b6zAf0shopyd0sdq208ys6qcyBw84wC0Mw5Megu1o32N3hDj3gg12FTxss81k1knoFxki1oKlH0pKdLT1wIdAoDA079wBQKjaxNobPEVhE80yxMGh3bbMO8Ef8zsG3pXUUGwy4xQg91bVno2gECXUs8-m4qKQ46ww2fedHhcQeKYYPtxd2ZzFbroKvaAOV2Hsbvg5imSM0K00CSnHTpdwLQzh0qmMQb20An2IGjdbFEw3eDJ-q8aKCy2Ric9jApYQgqgAvapoQgj-AVg8Cy1SIjh015mUMYIn2LCvfd452IzP_gd44UWxVEw4qAa6TxNIg5rDtNjD2hi>

On 3/15/2016 5:10 PM, Reindl, Kevin wrote:
Hi all,

I wanted to get others' opinions/experiences on a question that our company is grappling with (if you can point to specific research that helps shed light on this, even better):

Here's the situation: We typically hire employees into entry-level training or apprentice-level jobs for our skilled trades (electrical, mechanical maintenance, etc.). When we hire entry level employees, we use an array of basic aptitude, cognitive and personality-based assessments in the selection process (e.g., mechanical aptitude, numerical reasoning, spatial skills, conscientiousness, etc.).

However, because of recent gaps in our journey-level workforce (e.g., due to retirements, etc.), we have been recruiting and hiring experienced journey-level employees...i.e., those who have been performing the same work at other companies or have already completed apprenticeships elsewhere. For these jobs we have been using the same set of aptitude/cognitive and personality assessments, PLUS a test of technical knowledge required in the trade. There is also a behavior-based interview and technical interview to round out the selection process.

There is debate in our company regarding whether there is value in continuing the use of broad-based aptitude/cognitive tests for these journey level hires.

1.       On one side of the argument, the cognitive tests are well-validated assessments that have shown to be predictive of success in a wide range of similar jobs, and since we "require" candidates to take them for the entry level jobs, why wouldn't we require them for higher level jobs?

2.       On the other side of the argument, why would we need to assess general cognitive ability for experienced journey-level candidates, since they have presumably been performing similar work at other companies. Also, since we also assess their technical knowledge in other tests, require X years of experience, and in many cases require them to have completed a certified apprenticeship...then what added value is measuring their basic aptitude to do this kind of work?

I know that internal local research/a validity study might provide some answers, but before we initiate that level of research, I'd be interested in thoughts from the IPAC group...

*         Which side do you lean toward #1 or #2 above, and why?

*         What do you do for similar situations in your organization?

*         Do you know of any research (single study or meta-analysis) that may shed some light on the matter?

As always, I look forward to your thoughts!

Kevin Reindl
Pacific Gas & Electric
Human Resources
Assessments and Organizational Insights
245 Market Street, N2J
San Francisco, CA 94105
Office: 415-973-7013
Mobile: 619-322-3368
k1rq at pge.com<mailto:k1rq at pge.com> or kevin.reindl at pge.com<mailto:kevin.reindl at pge.com>

_______________________________________________________

IPAC-List

IPAC-List at ipacweb.org<mailto:IPAC-List at ipacweb.org>

https://pairlist9.pair.net/mailman/listinfo/ipac-list<http://cp.mcafee.com/d/FZsSd38Q939J5xZWXPz3BPtPqdTxPPbPP29J6XMVVBVVVcSyyyUesuphjdEEI8zxOpEVupd_7_gh1lGDJGA6WshGpdSGuSGgrFN6FASqemkZuZN_HYDtZdBfHTbFILnVxdwQsLeth5dqWqJQTel3PWApmU6CQPqpK_9EIEEI3DS4jtPpesRG9pzsjQft-5fVqdqKHlB2vbUpjNK9V9IUDeqR2X8wayGSOyvaAPhPOSOzMd5hdno2FYGjB1SEH9rr5QMU-l9J10487383zosn8J104me9xko5gOazpo5yhw1wUh1wAbCh941yop542o2FE1ysgIqegY1N5xC8Q1MRE80y9MpEOam0wi2o320n0V1U5wcb4d6tcd104aDu5NMw5g5htyC5h85yVmI1CUS_s62MShyug0sC2niVcG75wLezB6ww2a76zAM1gO2Sipkx4bKh2gG4EcwgG91MsqehM2zoQ1M6Yi2q84BNgaxU91w5nq208YUSJ4PgWXPPdS4QbSeAJJyVYGjbAaJMJZ0l9rr02U02rpuLtAS2_id41Fr3gI82hsaOFcQKCy0cWuTVEwGWq8bl8MBehDPh1F2hYFBzh1fWjB0yq87qNd404lrz3ONsa-pYYQgkaOffZ0QgjzG7Cy0hGgEru76Xfhr7kcV>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist9.pair.net/pipermail/ipac-list/attachments/20160316/bbb98321/attachment-0001.html>