[IPAC-List] It's not what they say, but who is saying it

[Mark Hammer]

My organization is tasked with oversight of merit, and assurance of
integrity, in hiring in the Canadian federal context.  As one component
of that mandate, we have an annual survey of about 95% of the public
service, and ask people to tell us about one specific staffing activity
they participated in, whether as hiring manager or somebody who tried to
snag a job, or ended up getting one.  We ask a number of questions abut
procedure, to get the hang of whether commendable practices are being
observed.  We also ask a number of attitudinal and evaluative
questions.
 
Among the questions are those concerning perceived fairness of how one
was assessed and the fairness of the process overall, as well as whether
candidates felted the assessment reflected the job.  And since we are
interested in the efficiency of staffing, we also ask about how long
things take.
 
One of the patterns observed over the last 3 years is that several
indices and predictors of perceived fairness have gone downhill; enough
that if you had no information other than the specific
percent-positive/agree figures, you'd be worried about the health of the
system.
 
Here is where it gets interesting, though.
 
Like governments nearly everywhere, ours has immersed itself into the
austerity mindset, a little more each year, over the past few years. 
One of the outcomes of that is that the composition of our survey
respondents has been changing.  I should note that about 6 years ago, we
embarked on a new survey methodology (for us, at least).  Understanding
that the health of the merit system is not indexed solely by the
opinions of the winners, we began including the losers.  So, the survey
is a kind of "cold call", where we send the survey out to everyone, find
out if people were involved in anything during the preceding year, ask
them to tell us about the most recently completed staffing activity, and
only after finding out about what they went through do we ask them if
they were successful or not.
 
What you see is to be expected: winners have more positive views on the
process than losers.  Sometimes there is a 20% spread between them. 
Winners and losers all get combined and when we report to Parliament
each year, one of the numbers we provide is what percentage of
candidates said that the staffing process they were involved in was
generally fair.
 
Although the aggregate number is falling by 4-6 percentage points per
year (!!), the actual expressed opinions are no different.  Break out
all those respondents into people who were successful in a competition,
and those who applied and weren't, and you see no change whatsoever in
how fair each group thought things were.  But what you DO see is a
continuing shift in how much of the overall sample is made up of
"winners" and "losers".  There are fewer jobs to give out, but you can't
stop people from applying for them, even if their odds of snagging that
promotion are slim under the present context.  And as the proportion of
unsuccessful candidates in our sample increases, the aggregate
attitudinal data descends.
 
But wait!  There's more!
 
The shifting ratio of successful to unsuccessful candidates has also
altered estimates of time-to-staff.  We ask people how long things took
to "conclude for you", and only use data from folks who give unambiguous
signs they participated in the entire duration of the process and were
not screened out near the start.  When we asked hiring managers to tell
us how long processes were taking for them, it appears things are taking
longer this year than in the recent past.  When we ask candidates how
long things took for them, time-to-staff gone down.  What the....? 
Well, consider that if you applied to a competition, and were
unsuccessful, things end for you when you find that out.  You don't have
to give anyone 2 weeks notice because you aren't going anywhere.  In our
legal context, many jobs have a dual official-language requirement, so
you don't need to be assessed for whether you meet the language fluencrequirement, and of course, no one needs to do a security check on you
to see if you meet the security requirements.  So successful candidates
report processes taking about 3 more weeks than unsuccessful candidates
report, for obvious reasons.  And, as you can imagine, with there being
more unsuccessful candidates in our survey sample this year, magically
staffing appeared to take about 2-3 weeks less than it did last year,
from their perspective, even though successful and unsuccessful
candidates are each reporting almost the identical times (<0.1month
difference) they did last year.
 
The challenge here is reporting such results to the sort of folks who
want a "dashboard" that presumes no underlying changes in the landscape
against which staffing occurs.  Lest they think something is seriously
wrong, we need to inform them about that landscape as simply as
possible.
 
The moral of the story?  Pay attention to WHO is telling you things on
organizational surveys.  It's an important consideration when surveying
at one time only, but it becomes especially important when engaging in
any sort of longitudinal comparison.
 
 
Mark Hammer
Ottawa


>

This e-mail message is intended for the named recipient(s) and may
contain information that is privileged, confidential and/or exempt from
disclosure under applicable law. Unauthorized disclosure, copying or
re-transmission is prohibited. If you are not a named recipient or not
authorized by the named recipient(s), or if you have received this
e-mail in error, then please notify the sender immediately and delete
the message and any copies.
 >
Ce courriel est destiné exclusivement au destinataire mentionné en titre
et peut contenir de l'information privilégiée, confidentielle ou
soustraite à la communication aux termes des lois applicables. Toute
divulgation non autorisée, toute reproduction ou réacheminement est
interdit. Si vous n'êtes pas le destinataire de ce courriel, ou n'êtes
pas autorisé par le destinataire visé, ou encore, si vous l'avez reçu
par erreur, veuillez le mentionner immédiatement à l'expéditeur et
supprimer le courriel et les copies.      

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://nine.pairlist.net/pipermail/ipac-list/attachments/20130528/f0488a79/attachment.html