[IPAC-List] How significant is significant enough?

[Blair, Michael D[EQ]]

Jason -

If your goal is to determine if the math test predicts success in the apprenticeship program, then you've met with success.  As Winfred pointed out, the sample size is small, so the results are not as robust as you might prefer and the measurement error may be greater than desired.  However, as also mentioned by Winfred, due to the range restriction (incumbent only sample), the corrected correlation in likely quite higher.

If you stop here, you're likely going to run into adverse impact issues and, if challenged, be in a tenuous position for two reasons: (1) Training/apprenticeship success is not a sufficient criterion without a clear connection between training success and job performance.  You've already indicated you're skeptical of this relationship. (2) Your predictor space is exceedingly narrow as you're only testing for one KSA.  With a little bit of work, you should be able to identify additional critical KSAs that can be measured with predictors that exhibit lower group differences.  Not only would this put you in a much better position to defend a challenge, it would also improve the quality of your pre-hire process resulting in better overall performance.

My advice would be to conduct a job analysis, identify a larger sample of KSAs to measure, seek out alternative predictors with lower AI, and relate test success directly to job performance bypassing the apprenticeship program.  If the job analysis indicates that the apprenticeship program needs to be re-designed, then do that as well.


Michael D. Blair
Manager, Recruitment, Selection & Assessment
EMBARQ
Voice: 913-345-6334  |  Cell: 913-832-6130  |  Fax: 913-345-6417
Email: Michael.D.Blair at embarq.com



This e-mail is the property of EMBARQ and may contain confidential and privileged material for the sole use of the intended
recipient(s). Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended
recipient (or authorized to receive for the recipient), please contact the sender and delete all copies of the message.


-----Original Message-----
From: ipac-list-bounces at ipacweb.org [mailto:ipac-list-bounces at ipacweb.org] On Behalf Of Jason Bowling
Sent: Thursday, April 09, 2009 11:13 AM
To: ipac-list at ipacweb.org
Subject: Re: [IPAC-List] How significant is significant enough?

The pre-employment test I am studying is a 25 question, multiple-choice, math test based on a 6th-grade math proficiency exam administered in the state. 

The apprenticeship program tests are not 100% math. They also test knowledge of electrical theory, line building practices, high-voltage approach distances, and other safe work methods. I am looking at the average of 40 tests administered to apprentices over a 4-year period. 

The question I am trying to answer by comparing the two results is, does the math test predict performance in the apprenticeship program. I should acknowledge, I do not know if performance in the apprenticeship program predicts performance on the job... anecdotally, I suspect not so much. I do know there is a threshold of performance an apprentice must maintain as a condition of employment.

At minimum, I would like to minimize the likelihood of hiring candidates who will fail our apprenticeship program while not causing adverse impact. Without a doubt, safety considerations alone make the ability to perform some basic calculations a bona fide requirement. Bad math can and has killed line workers.



-----Original Message-----
From: Lance Seberhagen [mailto:sebe at erols.com]
Sent: Wednesday, April 08, 2009 6:10 PM
To: Jason Bowling
Cc: ipac-list at ipacweb.org
Subject: Re: [IPAC-List] How significant is significant enough?

What are you trying to predict -- training performance or work performance?  The basic math test from the Apprenticeship entrance exam correlates .36 with the Apprenticeship final exam, which has a strong math component.  In other words, one math test correlates .36 with another math test.  Thus, the basic math test predicts passing the final

exam, but you don't really know if math skills are actually needed for performance as an Electrical Lineman (or Line Worker).  A content validation study would be helpful in answering this question.

What is the date of the current job description for Electrical 
Lineman?   Is there a job analysis report to go with it?   Does the job 
analysis comply with the Uniform Guidelines?  You probably need to conduct a new job analysis (including the apprenticeship program) to get

a clear understanding of what math skills, as well as other KSAs, are needed for the job of Electrical Lineman.  If the job analysis shows that math skills are needed, the next step would be to have job experts evaluate the relevance of math test items on the entrance exam and final

exam, as well as to obtain suggestions for improving the relevance of the math questions on both exams.  Does the entrance exam consist only of a basic math test?  If so, you might want to consider adding other tests to the entrance exam (e.g., reading, teamwork, safety), based on the results of the job analysis.

Lance Seberhagen, Ph.D.
Seberhagen & Associates
9021 Trailridge Ct
Vienna, VA 22182
Tel 703-790-0796
www.seberhagen.com


Jason Bowling wrote:


>Hi all,

>

> 

>

>I recently joined this list-serv and greatly enjoyed the conversations 

>thus far.

>

> 

>

>So, I have a question and would appreciate any thoughts.

>

> 

>

>I am considering administering a test of basic math skills to

candidates

>for apprentice electrical lineman. Actually, prior to my starting with 

>the company, a math test was SOP for candidates to these jobs, but no 

>one ever asked if the test was a valid predictor. Furthermore, the 

>weight placed on test results was always ambiguous. We seemed to be 

>taking a pragmatic approach to our employment processes... that is, it 

>always seemed to work in the past.

>

> 

>

>Our apprenticeship does include testing components, and math does play

a

>large role in those tests. Furthermore, the job description does 

>describe math skills to be an essential competency, and we have had to 

>let apprentices go for failing these apprenticeship tests.

>

> 

>

>So, I gave the test to our current apprentices and recent graduates 

>(N=30), and compared their test results to their average apprenticeship 

>test results (r=.36, and p=.02). The correlation seems weak to me, but

I

>have no basis of comparison. 

>

> 

>

>I could probably get more subjects to bolster the significance, but 

>again, I'm not sure at what point I should feel satisfied.

>

> 

>

>Any help?

>

> 

>

>Thanking you I am,

>

> 

>

> 

>

>Jason Bowling, PHR

>

>Human Resources Manager

>

>S.S.V.E.C

>

>Ofc: 520.515.3480

>

>Cel: 520.220.6563

>  

>



_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org
http://www.ipacweb.org/mailman/listinfo/ipac-list