[IPAC-List] Vendors for written exams - entry police and fire

Tue Jan 5 18:43:20 EST 2016

Hi Oscar,

Can you give a little more info for those of us who are just curious?  A prior post from Steve Partain mentioned that the test measured four components: 1) human relations, 2) mechanical reasoning, 3) reading comprehension, and 4) math comprehension. 

Your post indicates that your FireTeam test is a simulation exam.  I'm curious to know a little more about the simulation technique that provides for measurement of the four components that Steve gave, especially components 2, 3 and 4.  Also, can you share how you estimated reliability for each of the four components?  Are you using raters to assign scores while observing candidate behavior and then calculating inter-rater reliability, or using some other method?  I'm just having a hard time understanding how you can get simulation test scores for fire fighter candidates on such a diverse batch of abilities.

Thanks,
Richard Joines
Management & Personnel Systems, Inc.
www.mps-corp.com
925-932-0203

----- Original Message ----- 
  From: Oscar Spurlin 
  To: 'Partain, Steven C.' ; 'Hughes, Kelley A.' 
  Cc: ipac-list at ipacweb.org 
  Sent: Tuesday, January 05, 2016 11:22 AM
  Subject: Re: [IPAC-List] Vendors for written exams - entry police and fire

  Hi Kelley,

  As Steve mentioned, FireTEAM is an extremely effective tool for identifying potential top performers as demonstrated through our rigorous development and validation process.  This simulation exam is the most sophisticated firefighter simulation exam available.  It is used to measure critical aspects of a candidate's potential, such as teamwork, human relations and mechanical aptitude.  FireTEAM is used by over 500 fire departments across the country, most of which have used it many times over because of the effectiveness of the exam.  

  While we understand what Steve is saying, and have worked directly with TVFR stakeholders regarding candidate and department member perceptions, this is not a perception that is pervasive across our clients.  We have reviewed the exam with hundreds of professionals in the field with overwhelmingly positive feedback and confirmation that the exam is highly job related.  Below are a few quotes from our clients.

        Senior Analyst 

        Santa Clara Fire Department, CA
       "Upon completing the FireTEAM video based exam, the raters conducting interviews prior to being accepted into the academy indicated this was one of the most qualified group of recruits that had been screened based off interview results."

        Battalion Chief

        Redmond Fire , WA
       "We have been impressed with the FireTEAM exam since inception. Repeatedly our department has filled the top 5 positions in the Fire academy that has over 100 participants from 5 local agencies. The other 4 agencies are using another exam."

        Fire Captain 

        Mayer Fire District, Arizona
       "Transitioning to Ergometrics has proven most valuable for our organization.  Expanded reach of our job postings, mainstream evaluations, and prompt communication has produced the highest level of candidates.  We will continue to utilize FIRETEAM for years to come."

        Fire Chief

        City of Gainesville, Florida
       "We are extremely satisfied with the FireTEAM process.  I can wholeheartedly recommend the National Testing Network from firefighter entry up to Battalion-level promotional testing.  FireTEAM is fast becoming the preferred format for firefighter selection, we have moved from an entire in-house process to using FireTEAM to develop our hiring lists.  The detail of information from the reading, math and mechanical components translate well to the fire service and have resulted in producing outstanding candidates from which to choose.  The Personal History Questionnaire option is another factor that provides us with a unique perspective; these together with the CPAT provide an impressive array of tools from which to make the critical hiring decisions in the highly competitive firefighter market.  Beyond the quality of the FireTEAM product is the outstanding level of personalized client service and attention, these all together form a refreshing and quality experience."

        Human Resources Director

        Miami Township, Montgomery County - OH
       "Ergometrics offers critical value to my organization's selection processes. The tools produced are quality-based solutions which assist us in consistently selecting the most qualified candidates. They are job specific and criterion validated. This same quality applies to Ergometrics' level of customer service and it is a high standard that is practiced by each Ergometrics professional."

  The success of FireTEAM in not limited to just client satisfaction, it has also been through extremely rigorous validation and independent evaluation.  The United States Department of Justice has reviewed the validation and results from FireTEAM on multiple occasions, all resulting in favorable outcomes and no litigation for our clients.

  If you would like more information regarding FireTEAM or a product demo please let us know.  Good luck with your search for an effective firefighter examination process.

  Thanks,
  Oscar

  From: IPAC-List [mailto:ipac-list-bounces at ipacweb.org] On Behalf Of Partain, Steven C.
  Sent: Monday, January 04, 2016 1:27 PM
  To: 'Hughes, Kelley A.' <kahughes at littlerock.org>
  Cc: ipac-list at ipacweb.org
  Subject: Re: [IPAC-List] Vendors for written exams - entry police and fire

  Hi Kelley.  We have been utilizing the FireTEAM test from Ergometrics for about 10 years.  The test has four components: 1) human relations, 2) mechanical reasoning, 3) reading comprehension, and 4) math comprehension.  The first two components involve video-based situational judgement test items.  From an assessment professional's perspective, it appears to perform well.  The psychometrics (reliability, validity, low adverse impact) are at least as good as any alternatives we've seen.  The content domain it targets is quite good.  

  From a non-assessment professional's perspective, it is fatally flawed.  It cannot be "mastered" via hard work (effort put into studying) and there is little effective coaching that can be offered to test takers to give them an edge.   (Most assessment professionals would see this as a positive.)  I believe the fire service tends to attract individuals who believe everything can be mastered via hard work and training.  This test frustrates them as they try to support others trying to get into our organization.  

  Also, there are test takers who are deemed high-quality candidates by members of our workforce who don't score high enough on the test to be considered for hire, and people who do score high who are deemed to be of lesser quality by members of our workforce (false positives and false negatives, which I'm sure you'll appreciate are present in any test instrument).  For these reasons, a large portion of our workforce has rejected the test, and we have been charged with researching alternatives.  Despite our efforts, we have been unable to overcome the negative perception among laypersons in our workforce and communicate persuasively re: the science of assessment.  

  I share all of that out of recognition that in shopping for a test instrument (or package of instruments), workforce acceptance is at least as important as test instrument quality.  (That may be why you are focusing on written tests?)

  Good luck!

  ____________________________________________________

  Steven Partain | HR Manager

  Tualatin Valley Fire & Rescue 

  Direct: 503-259-1292

  www.tvfr.com

  From: IPAC-List [mailto:ipac-list-bounces at ipacweb.org] On Behalf Of Hughes, Kelley A.
  Sent: Monday, January 04, 2016 11:59 AM
  To: ipac-list at ipacweb.org
  Subject: [IPAC-List] Vendors for written exams - entry police and fire

  Hello all and Happy New Year,

  We are currently in the process of evaluating vendors for both entry fire and police written exams and I would be interested in any feedback and/or recommendations the group might have on the subject.  Thank you in advance.

  Kelley Hughes

  Human Resources Analyst, Sr.

  City of Little Rock Human Resources

  500 W. Markham Suite W-130

  Little Rock AR 72205

  (501) 371-4564

  mailto:kahughes at littlerock.org

------------------------------------------------------------------------------
          This email has been checked for viruses by Avast antivirus software. 
        www.avast.com 

------------------------------------------------------------------------------

  _______________________________________________________
  IPAC-List
  IPAC-List at ipacweb.org
  https://pairlist9.pair.net/mailman/listinfo/ipac-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist9.pair.net/pipermail/ipac-list/attachments/20160105/d1362f9c/attachment-0001.html>