[IPAC-List] Wall Street journal article on SAT coaching courses

RPClare at aol.com RPClare at aol.com
Fri May 22 10:27:09 EDT 2009

I always enjoy and appreciate Mark's contributions. In this discussion (and
in others) we neglect who the candidate is and what they're bringing to
the table as well as what the coach brings. Most prep programs are targeted
for specific areas of improvement. Most candidates aren't sophisticated
enough to realize what they really need.

In a message dated 5/22/2009 9:15:50 A.M. Eastern Daylight Time,
Mark.Hammer at psc-cfp.gc.ca writes:

Both outcomes can be true;. Coaching can have little effect, AND practice
and/or coaching can have moderately powerful effects.

The basic question one has to ask is "What could a person, and more
specifically THIS person, learn during coaching and/or practice?". The answer
is: a buncha stuff. And it would depend on what the nature of the test is
(and the cognitive ability tests that Hausknecht et al. looked at likely show
different benefit than might an in-basket), and the current testee

If I've done all manner of tests with modest success - multiple choice,
written, oral presentations to selection panels, etc. - and have a reasonable
amount of recent experience under my belt, then what I get from coaching
or a retest is much more focussed on the specific content of the test, and
perhaps things like test-specific time management strategies. If I'm a 17
year-old who freezes at the thought of a test, or who lacks any of the
metacognitive strategicness they will have acquired in another 6 years, then
what I get out of coaching or retests is likely somewhat different, and may
lie more in the domain of simply deflecting negative self-comments during
test-taking (all that Carol Dweck "entity theory" stuff, which is fascinating
and a highly recommended read).

When I used to teach reasonable-sized classes, where written answers were
feasible, I would often pass out little "metacognitive prosthetic" cards to
each student to park on their desk before an exam. The cards contained a
half dozen or so questions for the student to ask themselves, and I told
students before the exam started that these were the sorts of questions that
A+ students *always* asked themselves throughout, and B or C students asked
themselves only now and then. The advice was to consult the card and let
it assist you in building up those A+ habits. The questions were things
like: "Did I answer the question that was asked?", "Did I say what I wanted
to say?", "Could someone not in this course understand what I wrote?", etc.
All essentially self-monitoring queries that nudged the testee towards
effective calibration of effort and product.

These are the sorts of metacognitive advances that develop over the course
of many tests. I have no illusions about students looking at such a card
on one occasion and suddenly leap-frogging ahead in their performance.
However, the idea of pausing to reflect, monitor one's performance, and
re-calibrate as necessary, has to start somewhere. That sort of learning,
though, I would expect to take much longer than what transpires in a test-retest
interval, a 2 month prep workshop, or whatnot. And as much of a role as it
might play in optimum performance in ANY test, it is different from the
many other things a person might acquire over practice tests, re-tests,
workshop, or simply applying to things over and over again.

One of the things I learned during my thousands of hours "running rats"
(ah, the sweet musty poopy smell of science!) was to always ask "What *could*
they learn here?". What I tried to get them to learn was not always what
they learned nor the full extent of what they learned. I think the same
question must be asked when one examines practice and/or coaching effects.
What is it we *think* they will learn, and what DO they learn? To my mind,
the interactions with both test-type, age, and test experience will likely
be significant, as will the effects of duration of coaching or extent of

Mark Hammer

>>> "Winfred Arthur, Jr." <w-arthur at neo.tamu.edu> 2009/05/21 10:03 pm >>>

relatedly, the Hausknecht et al. (2007, JAP, 92, 373-385, "Retesting in
selection: A meta-analysis of coaching and practice effects for test of
cognitive ability") meta-analysis might be equally informative. their
abstract reads as follows:

"Previous studies have indicated that as many as 25% to 50% of
applicants in organizational and educational settings are retested with
measures of cognitive ability. Researchers have shown that practice
effects are found across measurement occasions such that scores improve
when these applicants retest. In this study, the authors used
meta-analysis to summarize the results of 50 studies of practice effects
for tests of cognitive ability. Results from 107 samples and 134,436
participants revealed an adjusted overall effect size of .26. Moderator
analyses indicated that effects were larger when practice was
accompanied by test coaching and when identical forms were used.
Additional research is needed to understand the impact of retesting on
the validity inferences drawn from test scores."

- winfred
IPAC-List at ipacweb.org

**************Kick start your favorite grad’s career with mobile email for
under $50.

More information about the IPAC-List mailing list