[IPAC-List] IPAC-List Digest - Computer-Administered Testing
John.Kraus at sfgov.org
John.Kraus at sfgov.org
Wed Feb 9 18:05:33 EST 2011
We use two software products for proctored, computer-administered tests.
One has been used extensively, FastTest, and the other, Questionmark, was
recently purchased. Both products have item-banking and test assembly
features, and both products allow for a variety of question/item formats.
Test administration software can be loaded/run on C drives or over a
network. We have a designated facility for testing with 23 pc's, but also
use computer-training rooms at various locations which typically hold about
20-25 pc's each. In these various locations, the pc either sits alone on a
desk or there will be two pc's situated on a narrow table. Haven't done
unproctored testing over the net yet, primarily due to firewall issues.
John Kraus
City and County of San Francisco
ipac-list-request
@ipacweb.org
Sent by: To
ipac-list-bounces ipac-list at ipacweb.org
@ipacweb.org cc
Subject
02/09/2011 01:50 IPAC-List Digest, Vol 5, Issue 13
PM
Please respond to
ipac-list at ipacweb
.org
Send IPAC-List mailing list submissions to
ipac-list at ipacweb.org
To subscribe or unsubscribe via the World Wide Web, visit
http://nine.pairlist.net/mailman/listinfo/ipac-list
or, via email, send a message with subject or body 'help' to
ipac-list-request at ipacweb.org
You can reach the person managing the list at
ipac-list-owner at ipacweb.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of IPAC-List digest..."
Today's Topics:
1. Re: Computer assisted testing (Wilkening, Kurt)
2. Re: Computer assisted testing (Reindl, Kevin)
3. Re: Computer assisted testing (White, Don)
4. I love it when data tells a story (Mark Hammer)
----- Message from "Wilkening, Kurt" <WilkeningK at HillsboroughCounty.ORG> on
Wed, 9 Feb 2011 10:26:48 -0500 -----
To: "keith.poole at phoenix.gov" <keith.poole at phoenix.gov>, "IPAC-List at ipacweb.org" <IPAC-List at ipacweb.org>
Subject: Re: [IPAC-List] Computer assisted testing
I'm interested in learning what other jurisdictions are doing in this area
as well. Not only in terms of software, but the space and hardware
(furniture) requirements needed to turn a basic card tables/fold up
chairs/scantron forms/paper and pencil type operation into a dual purpose
testing/training type facility. For example,
http://www.versatables.com/pages/products/school/revolution-desk-72.php#
(no affiliation or previous purchase, just an example).
So I'd be very grateful as well if someone from the previous mentioned
jurisdictions, and others not listed who've implemented computer-based
proctored testing could forward to the listserv (or Keith) any information
they have on software and hardware setup.
Regards,
Kurt Wilkening, Division Chief
Recruiting, Testing, and Certification
Hillsborough County Civil Service Board
601 E. Kennedy Blvd., 17th Floor
Tampa, Florida 33602
(813) 274-6764 (Direct)
-----Original Message-----
From: ipac-list-bounces at ipacweb.org [mailto:ipac-list-bounces at ipacweb.org]
On Behalf Of keith.poole at phoenix.gov
Sent: Tuesday, February 08, 2011 4:11 PM
To: IPAC-List at ipacweb.org
Subject: [IPAC-List] Computer assisted testing
One item on our division to-do list this year is to "develop
recommendations regarding the use of computer assisted testing." Basically
we understand this to mean any technology as an alternative to using paper
and pencils when conducting multiple choice tests (e.g., anything from
off-the-shelf software loaded onto individual PCs or network running
in-house developed tests, to using online tests from vendors, proctored or
unproctored, etc.).
I have seen the slides from last years IPAC conference (CPS did a couple
on this topic) and those were helpful, I've also signed up for their
webinar next week. We're also familiar with National Testing Network.
I would appreciate hearing from agencies that have researched or adopted
computer assisted testing in one form or another. Thanks.
Keith Poole
Human Resources Supervisor
City of Phoenix Human Resources Dept
Employment Services
(602) 262-7140
keith.poole at phoenix.gov
"Phoenix - 2009 All-America City"
_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org
http://www.ipacweb.org/mailman/listinfo/ipac-list
----- Message from "Reindl, Kevin" <KReindl at semprautilities.com> on Wed, 9
Feb 2011 09:49:36 -0800 -----
|--------->
| To:|
|--------->
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|"'Wilkening, Kurt'" <WilkeningK at HillsboroughCounty.ORG>, "keith.poole at phoenix.gov" <keith.poole at phoenix.gov>, "IPAC-List at ipacweb.org" |
|<IPAC-List at ipacweb.org> |
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|--------->
| Subject:|
|--------->
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|Re: [IPAC-List] Computer assisted testing |
>--------------------------------------------------------------------------------------------------------------------------------------------------|
Our company has been slowly transitioning all of our paper/pencil testing
to computer-based (web-based) testing over the past several years.
Regarding hardware - we began with using standard computer training rooms
(which had about 10-20 PC's set up and network connected), but these rooms
did not lend themselves particularly well to paper/pencil testing (not
enough room on the tables), and the testing team had a difficult time
reserving these highly utilized rooms, since the priority for them were for
training. So, we found a second option, which was to outsource test
proctoring to a testing vendor (PSI, Inc.) who owns and operates
computer-based testing centers across the country. This is a nice option,
but does cost additional money for administration time, and does not work
well if you are testing candidates for a job in a remote/rural area. No one
wants to have candidates drive 50 miles to a testing center to take a 20
minute test. So now we are setting up a 3rd, more portable option, which is
to purchase a set of laptops and take these on the road. This will
hopefully allow us to transport and set up a testing center anywhere there
are tables/chairs and a web-connection. We are trying to determine whether
this can be done using a wireless internet connection, or if we need to
have access to hard-wired network connections. If all goes well, we will
then have 3 different options for hardware/facilities for our staffing
organizations to use.
As far as software, since the majority of our tests are published by PSI,
the bulk of our testing is administered on their web-based platform (i.e.,
"Atlas"). If you use published tests from several different vendors, then
this option can be a bit more challenging (PAN Testing might be a good
option). And, if you have custom tests that were developed/validated
in-house, then you could contract with an online testing vendor to host
your content...but this gets pretty expensive as we have recently found
out.
Kevin Reindl
San Diego Gas & Electric
"This message is intended only for the use of the individual or entity to
which it is addressed, and may contain information that is privileged
and/or confidential. Receipt by an unintended recipient does not constitute
a waiver of any applicable privilege. Reading, disclosure, discussion,
dissemination, distribution or copying of this information by anyone other
than the intended recipient or his or her employees or agents is strictly
prohibited. If you have received this communication in error, please
immediately notify the sender."
-----Original Message-----
From: ipac-list-bounces at ipacweb.org [mailto:ipac-list-bounces at ipacweb.org]
On Behalf Of Wilkening, Kurt
Sent: Wednesday, February 09, 2011 7:27 AM
To: keith.poole at phoenix.gov; IPAC-List at ipacweb.org
Subject: Re: [IPAC-List] Computer assisted testing
I'm interested in learning what other jurisdictions are doing in this area
as well. Not only in terms of software, but the space and hardware
(furniture) requirements needed to turn a basic card tables/fold up
chairs/scantron forms/paper and pencil type operation into a dual purpose
testing/training type facility. For example,
http://www.versatables.com/pages/products/school/revolution-desk-72.php#
(no affiliation or previous purchase, just an example).
So I'd be very grateful as well if someone from the previous mentioned
jurisdictions, and others not listed who've implemented computer-based
proctored testing could forward to the listserv (or Keith) any information
they have on software and hardware setup.
Regards,
Kurt Wilkening, Division Chief
Recruiting, Testing, and Certification
Hillsborough County Civil Service Board
601 E. Kennedy Blvd., 17th Floor
Tampa, Florida 33602
(813) 274-6764 (Direct)
-----Original Message-----
From: ipac-list-bounces at ipacweb.org [mailto:ipac-list-bounces at ipacweb.org]
On Behalf Of keith.poole at phoenix.gov
Sent: Tuesday, February 08, 2011 4:11 PM
To: IPAC-List at ipacweb.org
Subject: [IPAC-List] Computer assisted testing
One item on our division to-do list this year is to "develop
recommendations regarding the use of computer assisted testing." Basically
we understand this to mean any technology as an alternative to using paper
and pencils when conducting multiple choice tests (e.g., anything from
off-the-shelf software loaded onto individual PCs or network running
in-house developed tests, to using online tests from vendors, proctored or
unproctored, etc.).
I have seen the slides from last years IPAC conference (CPS did a couple
on this topic) and those were helpful, I've also signed up for their
webinar next week. We're also familiar with National Testing Network.
I would appreciate hearing from agencies that have researched or adopted
computer assisted testing in one form or another. Thanks.
Keith Poole
Human Resources Supervisor
City of Phoenix Human Resources Dept
Employment Services
(602) 262-7140
keith.poole at phoenix.gov
"Phoenix - 2009 All-America City"
_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org
http://www.ipacweb.org/mailman/listinfo/ipac-list
_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org
http://www.ipacweb.org/mailman/listinfo/ipac-list
----- Message from "White, Don" <DWhite1 at Columbus.gov> on Wed, 9 Feb 2011
16:18:37 -0500 -----
|--------->
| To:|
|--------->
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|"'Wilkening, Kurt'" <WilkeningK at HillsboroughCounty.ORG>, "keith.poole at phoenix.gov" <keith.poole at phoenix.gov>, "IPAC-List at ipacweb.org" |
|<IPAC-List at ipacweb.org> |
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|--------->
| Subject:|
|--------->
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|Re: [IPAC-List] Computer assisted testing |
>--------------------------------------------------------------------------------------------------------------------------------------------------|
Here's a copy of what I sent to Keith re: the software we use . . . .
We are using 3 different software programs in Columbus for different
aspects of computer testing. All three are loaded on our network - we're
not on the cloud with them. We administer in our test center - 25
computers - proctored. The programs are:
SkillCheck (First Advantage Co. www.fadv.com)- used to test basic clerical
skills (filing, spelling, grammar, letter formatting, typing, data entry),
software skills (Access, Word, Excel, Outlook, the internet, etc.),
computer tech skills (digital literacy, Windows, etc.), and call
center/customer service skills (uses headphones, runs through a call center
simulation, candidates respond on the computer via simulated data entry
screens - used for our non-emergency call center customer service
positions). The stimulus material is very face valid. For example,
candidates might be presented with a simulated Word screen and asked to
bold a certain passage, cut and paste, change font, move columns/cells/rows
in Excel - they actually perform the task and the software determines
correct or not. We can select number of tries permitted. The program will
keep time and cut off at the time limit. We can also write our own
multiple-choice items and put them into a test.
CritiCall (www.biddle.com, www.criticall911.com) - used for 911 dispatcher
positions. Another very face valid job simulation package. Candidates wear
headphones and receive stimulus material both orally and on screen. The
program has a lot of subtest options, so you set up to test the skills that
are most relevant to your particular job, such as data entry (oral and
written), decision making, prioritizing, multi-tasking, memory/recall, etc.
I can't say enough about this program - extremely robust/versatile/face
valid - and pretty much runs flawlessly (we are testing 911 operator as I
write this - of the over 300 candidates tested so far, we've had 1 computer
problem, and I'm not convinced it was a software issue!).
Sigma (www.neogov.com): Sigma was bought by NeoGov a couple of years ago.
NeoGov is working on an interface between the Sigma Test Management System
(TMS - item bank, test publishing) and the NeoGov applicant tracking
program (on the cloud/enterprise). We use Sigma TMS to develop all of our
multiple-choice, T&E, and written work sample tests. It has a feature that
allows us to administer on computer/responses are entered directly into the
software program. We have not used this as much as we should, but the
option is there. I'm not sure what NeoGov's plans are for the TMS side of
things, but I think they have a process in place to use the NeoGov on-line
application package in conjunction with Sigma TMS.
Please note that we have the facilities to administer computer exams. We
also use the computers for most of our written work sample/short answer
exams by setting up a Word document as a form and having candidates type
their answers into the form, which eliminates the illegible handwriting
concern. We also proctor all of our tests - still a little concerned with
security issues of candidates sitting at home taking a test on their
computer.
And here's some additional info on the facility:
The test center has 50 carrels with computers at 25 - I have photos I can
email direct, but I don't think the list will allow me to send attachments.
We hope to move to a new location in 2012 that will have 40 carrels, each
with a computer. Current carrels are 30" wide, which we have found to be a
little cramped, so the new ones will be 36" wide. Side walls are 40" tall,
which requires filters on the computer screens - 48" side walls would
eliminate the need for filters. There should be at least 5' between the
rows (from keyboard to keyboard). I recommend at least that much to be
able to walk between the rows. Although our computers are on the network,
they are locked down in terms of access (no internet, etc.). I think our
original cost was around $1000 per carrel, but I have no idea what it would
be in today's market.
Don White
Personnel Analyst Supervisor
Civil Service Commission
Columbus, Ohio
614-645-8021
-----Original Message-----
From: ipac-list-bounces at ipacweb.org [mailto:ipac-list-bounces at ipacweb.org]
On Behalf Of Wilkening, Kurt
Sent: Wednesday, February 09, 2011 10:27 AM
To: keith.poole at phoenix.gov; IPAC-List at ipacweb.org
Subject: Re: [IPAC-List] Computer assisted testing
I'm interested in learning what other jurisdictions are doing in this area
as well. Not only in terms of software, but the space and hardware
(furniture) requirements needed to turn a basic card tables/fold up
chairs/scantron forms/paper and pencil type operation into a dual purpose
testing/training type facility. For example,
http://www.versatables.com/pages/products/school/revolution-desk-72.php#
(no affiliation or previous purchase, just an example).
So I'd be very grateful as well if someone from the previous mentioned
jurisdictions, and others not listed who've implemented computer-based
proctored testing could forward to the listserv (or Keith) any information
they have on software and hardware setup.
Regards,
Kurt Wilkening, Division Chief
Recruiting, Testing, and Certification
Hillsborough County Civil Service Board
601 E. Kennedy Blvd., 17th Floor
Tampa, Florida 33602
(813) 274-6764 (Direct)
-----Original Message-----
From: ipac-list-bounces at ipacweb.org [mailto:ipac-list-bounces at ipacweb.org]
On Behalf Of keith.poole at phoenix.gov
Sent: Tuesday, February 08, 2011 4:11 PM
To: IPAC-List at ipacweb.org
Subject: [IPAC-List] Computer assisted testing
One item on our division to-do list this year is to "develop
recommendations regarding the use of computer assisted testing." Basically
we understand this to mean any technology as an alternative to using paper
and pencils when conducting multiple choice tests (e.g., anything from
off-the-shelf software loaded onto individual PCs or network running
in-house developed tests, to using online tests from vendors, proctored or
unproctored, etc.).
I have seen the slides from last years IPAC conference (CPS did a couple
on this topic) and those were helpful, I've also signed up for their
webinar next week. We're also familiar with National Testing Network.
I would appreciate hearing from agencies that have researched or adopted
computer assisted testing in one form or another. Thanks.
Keith Poole
Human Resources Supervisor
City of Phoenix Human Resources Dept
Employment Services
(602) 262-7140
keith.poole at phoenix.gov
"Phoenix - 2009 All-America City"
_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org http://www.ipacweb.org/mailman/listinfo/ipac-list
_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org http://www.ipacweb.org/mailman/listinfo/ipac-list
----- Message from "Mark Hammer" <Mark.Hammer at psc-cfp.gc.ca> on Wed, 09 Feb
2011 16:51:05 -0500 -----
To: "IPAC-List at ipacweb.org" <IPAC-List at ipacweb.org>
Subject: [IPAC-List] I love it when data tells a story
Since I began my job, and probably for ages before I ever got here,
managers have kvetched about how long staffing took. You can imagine
that, as a federal central agency concerned with oversight of the merit
system, we are concerned about the time it takes to staff.
But what slows things down? Lots. People in some quarters have made a
fairly cogent case that effective HR planning would necessarily expedite
things, and assure that managers get the legendary "right people in the
right place at the right time".
I've tried to make the case that while this is probably somewhat true
(better planning is more effective than no planning), there is a whole
stinking big heap of factors that you have precious little control over,
and that effective planning can only partially overcome.
So, with that in mind, a little over a year ago, I decided to ask our
managers in the annual survey (where they tell us about one staffing
process) whether they were filling a brand new position, simply ramping
up capacity in an area, or backfilling a vacancy.
If they were backfilling a vacancy, we asked them how far in advance of
the incumbent's departure they were notified. Not surprisingly,
somewhere around 60% of managers reporting indicated that they has 2
months or less advance warning that the incumbent was leaving. A
significant proportion of those involved less than a month's notice.
T'was ever thus.
We asked those same managers how long the position remained vacant for
before being filled. The working assumption was that the more advance
notice you had, and the less you had to scramble, the greater the
likelihood that the job would not sit idle for very long. As one of our
recently retired directors said to me once, what matters to the average
citizen is not how long the manager had to spend in hiring someone, but
how long the job sits there with no one doing it.
As you might expect, when managers had more advance warning (and often
this would be impending retirements), a much larger share of those
vacancies were filled in a timely manner such that the position was
either never vacant or only vacant for a brief period. Makes perfect
sense, just as it makes sense that when managers were faced with "I've
accepted a position at XX and I start there in 2 weeks. Have a nice
life.", those positions could often sit vacant for months until all the
paperwork was done, the poster posted, candidates assessed, etc.
So far so good.
Last year, I noticed a number of managers who provided comments at the
end of the survey complaining about being shuffled from one HR advisor
to another during the course of the same process, often receiving
contradictory advice about policies, regulations, or optimal strategies.
The record was 6 (!!) consecutive HR people for the same process. I
wondered whether there was something to this or not. It was clear that,
being situated in the capitol, and in the midst of both a wave of boomer
retirements and frenzied "PS renewal", folks working in HR were
extremely mobile, and were hopping from job to job at an alarming rate.
But still, you can't rely on what people decide to volunteer in their
comments. So I decided to ask managers in our recently concluded survey
just how many HR advisors they had for the process they were reporting
on, and see just how common this was.
I was somewhat surprised to see that an estimated 45% or so (we're
still cleaning data so that could be off by a few % points) reported
having more than one HR advisor advising them during the course of the
competitive process they were overseeing. About 10% or so reported 3 or
more.
In the interests of clarity and simplicity, I collapsed our data so
that I had the following 2 x 2 x 2 matrix:
How much notice?: 2 mos or less / More than 2 mos
How long vacant?: 2 mos or less / More than 2 mos
How many advisors?: One / 2 or more
As previously observed, if you had more lead time before the position
became vacant, a larger share of positions only remained vacant for a
short period (understanding that 2 months may not be "short" where you
work). As anxiously anticipated, the impact of only having short notice
was much greater when the manager reporting not having the same singular
HR person connected to the process throughout. In sum, if HR in your
organization was moving around (with many of them new), and your
employee took off suddenly for a new job, you were likely going to have
to wait a while before there was someone to do that employee's work
again.
That has helped to make sense of other things managers were reporting,
such as low satisfaction with staffing services provided the more HR
advisors were involved. Given how historically constant their
aggravation over slow staffing has been, small wonder that when there
are more HR cooks passing through the kitchen, managers complain more.
Of course, in the realm of "surprises" that even the best of planning
cannot completely offset, we need to include churn and sudden departures
within the HR community itself. Hopefully, once all those blasted
boomers have retired and things simmer down to more traditional
departure rates, things might be expected to improve. But for now,
instability makes things grind to a halt.
More to come in the months ahead.
Mark Hammer
Ottawa
>
This e-mail message is intended for the named recipient(s) and may
contain information that is privileged, confidential and/or exempt from
disclosure under applicable law. Unauthorized disclosure, copying or
re-transmission is prohibited. If you are not a named recipient or not
authorized by the named recipient(s), or if you have received this
e-mail in error, then please notify the sender immediately and delete
the message and any copies.
>
Ce courriel est destiné exclusivement au destinataire mentionné en titre
et peut contenir de l'information privilégiée, confidentielle ou
soustraite à la communication aux termes des lois applicables. Toute
divulgation non autorisée, toute reproduction ou réacheminement est
interdit. Si vous n'êtes pas le destinataire de ce courriel, ou n'êtes
pas autorisé par le destinataire visé, ou encore, si vous l'avez reçu
par erreur, veuillez le mentionner immédiatement à l'expéditeur et
supprimer le courriel et les copies.
_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org
http://www.ipacweb.org/mailman/listinfo/ipac-list
More information about the IPAC-List
mailing list