[IPAC-List] Ricci Update Prompts a Question

[Mark Hammer]

In the process of an employment equity review of a selection test a dozen years ago, we encountered pretty much exactly what John describes in his scenario.  The labour market availability of those classifiable as "members of a visible minority" (under Canadian law) was in the neighbourhood of 10%.  However, given the push to recruit and the likely slant of the ads, visible minority members ended up constituting some 34% of test-takers.  The odds of us having over-sampled equivalently across the entire distribution of competence/qualifications in that group, such that we had three times as many highly-competent applicants as expected, and *only* 3x as many low-quality candidates, are poor, to say the least.

When we looked at success rates on the test within groups, we missed the 4/5 benchmark (which does not have the same force in Canada as it does in the USA) by a wide margin.  When I did the "forbidden analysis" and simply examined what percentage of those who passed the exam were visible minority members, it was around 12% or more, just above their LMA rate, but close to what you'd expect if you drew a random sample of ONLY high-calibre applicants from the population at large.  The only conclusion I could come to was that in the push to attract minority applicants we attracted a disproportionately high number of poorer quality candidates from that group.  You'll forgive the dismissive analogy, but it was a bit like shopping for produce in a large budget supermarket: plenty of top-notch apples in there, but boy oh boy do you have to sift your way through a lot of bruised ones to find the nice ones when the store buys produce by the truckload.

So, to respond to John's "hypothetical" scenario, yes it CAN happen that there are ways of trying harder to recruit under-represented groups that can backfire on you with respect to some indices of fair process/testing.  In a perfect world, it might be possible to have an index of ability quite apart from what it is one is specifically assessing for, in order to be able to say that a distinction needs to be made between the basic calibre of applicants one got from group X vs group Y and Z.

Just to note, we ultimately dropped the test because it appeared too easy for people and resulted in a highly skewed score distribution that made it ill-suited to selection.  And for a final bit of info, assessment godfather Robert Guion makes note in the chapter on testing for bias in his 1998 book, that there can be the appearance of test bias simply as a result of differential sampling across the ability spectrum in two groups.  You never get to draw a truly random sample of test-takers.  In the real world, they always self-select to apply for your job and take your test.

Mark Hammer
Ottawa