[IPAC-List] Speed vs Power Tests

Wed Jun 1 08:18:51 EDT 2016

The discussion on speed versus power tests reminded me of some research that four of my graduate students (Dave Sharrer, Marcelle Clavette, Jeanne Donaghy, & Rob Lytle) did back in 2010 (they presented their research at IOOB but did not write a formal paper).  They were investigating a construct called “personal tempo” which relates to the speed with which we complete daily tasks such as talking, walking, brushing teeth, and taking tests.  The idea here is that some people do things quickly and others take more time.

Given that the research on test-taking times shows a negligible relationship between time to complete an exam and performance on the exam, the students wondered if test-taking speed was a function of personal tempo rather than cognitive ability or knowledge. For those who have taught, you have probably noticed that the same students finish all of their exams way ahead of the class and there are always two or three, bless their hearts, who stay to the bitter end of a three-hour class.

Some interesting findings:

-          - A meta-analysis of 34 studies yielded a mean correlation of .00 between order of finish and test scores

 -

-           - Test completion times were similar for whites and minorities and men and women

-

-          -  Test completion times across four exams were highly correlated (this is consistent with meta-analytic results)

-          - The concept of personal tempo was not unidimensional: It had six factors (personal hygiene, classroom behavior, communication, motor skills, ordering food, and decision making)

-          Only the classroom behavior factor was correlated with test-taking times

Not sure if this info is helpful but it was a nice trip down memory lane for me :)
_________________________________________

Michael G. Aamodt, Ph.D. (Mike)
Professor Emeritus
Department of Psychology
Radford University
Radford, VA  24142-6946
(202) 280-2172
maamodt at radford.edu<mailto:maamodt at radford.edu>

________________________________
From: IPAC-List [ipac-list-bounces at ipacweb.org] on behalf of Winfred Arthur, Jr. [w-arthur at tamu.edu]
Sent: Tuesday, May 31, 2016 9:29 PM
To: ipac-list at ipacweb.org; RICHARD.TONOWSKI at EEOC.GOV
Subject: Re: [IPAC-List] Speed vs Power Tests

Richard, i think the speed/power continuum is just another test characteristic along which (the administration of) tests vary.  however, it is important that where a test falls on this continuum be consonant w/ the intended construct and the inferences one wants to draw from the test scores.  hence, one would typically expect a typing test (to extent that we still have those) or a data entry test to be more on the speed end of things b/c there is typically an interest in making inferences about how quickly (and accurately) a person can type or enter data.  likewise, 1st responders, such as EMT or paramedics, typically have to access walking knowledge and make decisions based thereon fairly rapidly.  consequently, most job analyses would support having tests that assess this activity fall more on the speed end of the continuum.

in shape contrast, our comprehensive exams for our phd candidates (which are not atypical) are multi-day exams, somewhat reflecting the extreme power end of the continuum, b/c we as a program have decided that this more accurately reflects how the students will typically be originating solutions to the problems and queries they encounter as scientists and scholars, and yes, as practitioners as well.

of course, the speed/power continuum, which has to do specifically w/ the amt of time provided to the test taker to complete an assessment, is quite different from constructs such as "processing speed", "perceptual speed", etc., which are specific abilities under the general ability (i.e., g) construct.

thus, in answer to your question Richard, whether "the speed component is construct contamination or a bonus for g-loaded constructs that are supposedly being measured" again depends on the intended construct.  if i have a specific ability measure of processing speed or perceptual speed, or even my EMT/paramedic example, then speeding the test will be aligned w/ the intended construct and there not a contaminate.  however, in my comprehensive exam example, if we gave students an hr to dsgn a study and write w/ write up the associated rsch proposal as an NSF grant application --- yes, i know i am being extreme  :)  ---  then the speed component would definitely be construct-irrelevant and therefore a contaminant.

is this helpful?

- winfred

On 5/31/2016 7:42 PM, RICHARD TONOWSKI wrote:
I have a vague recollection that Jack Hunter did a factor analysis of some test (GATB?  ASVAB?) where, to his surprise, he found a speed factor independent of g.  It was in a research report; I haven't seen this mentioned in a journal article.

While trying to find this, I noted that John Carrol's three-stratum model (with its fluid and crystallized components) had "broad cognitive speed" and "processing speed" as constructs.  Speed may involve different constructs to be measured.  I'm not familiar with the model, but it seems that everything interrelates, with g as the superordinate stratum.

There's more I stumbled across regarding what is meant by speed or speededness.  This from W.J. van der Linden (2011).  Test design and speededness. Journal of Educational Measurement, 48, 44-60:

"The notion of speededness in testing refers to an interaction between three important factors: the cognitive speed at which the test taker works during the test, the amount of labor required by the items, and the time limit on the test. A test is more speeded when a test taker has to work faster, answering the items requires more labor,and/or the time limit is tightened. As the speed of the test taker is one of these factors, it actually is incorrect to refer to the speededness of a test."

The issue here seems to be that introduction of speededness may introduce a construct other than what was intended.  He continues with an example of someone with a disability not related to the intended construct, but who takes the test slower than others because of the disability.  The speed measured in the test situation may not be the speed in using the intended construct (assuming that speed is intended to be included in the measurement).

Taking the liberty to rephrase Joel's questions, I'm wondering if the speed component is construct contamination or a bonus for g-loaded constructs that are supposedly being measured.

Somebody please provide answers!

Rich

.

>>> <mhammer at 295.ca><redir.aspx?REF=vfulvKOmIaPnDu97DnJpFSglPhbZNYzUmCgascCn3fNxI7l2ForTCAFtYWlsdG86bWhhbW1lckAyOTUuY2E.> 5/31/2016 6:53 PM >>>
And in keeping with Winfred's comments, the format of each can provide one
with different sorts of measurement of the construct in question; speed
tests being prone to multiple-choice and power more amenable to
open-ended.

Insomuch as speed tests are often a product of convenience contingencies
(e.g., automated scoring), I would think the real question is whether
power tests increment to validity over speed tests.  After all, if there
were a reasonably large number of testees to process, one would opt for a
speed test first since it is less labor-intensive as far as scoring. And
once you have those scores, does an additional power test of some kind
tell you more?

Mark Hammer
Ottawa

> Joel, i think it would have to be a function of the construct assessed,
> right?  so, if by "To what extent do speed and power tests measure the
> same constructs?" you mean the _*same test*_ administered under speeded
> and power conditions, then the answer would have to be "no" b/c
> additional variance, maybe construct-relevant or construct-irrelevant
> has been introduced into the test scores.  consequently, one would
> expect the convergence b/n the two sets of scores to be low
>
> and "Do speed tests add validity over power tests?" => again, it will
> seem the answer would depend on the constructs assessed by the two
> tests.  thus, for example, tests of processing speed and reaction time,
> are by definition speeded tests; whereas for example tests of fluid
> intelligence (e.g., the Ravens) and declarative knowledge are typically
> better conceptualized as being closer to the power end of the continuum.
>
> hope this helpful?
>
> - winfred
>
> On 5/31/2016 12:34 PM, Joel Wiesen wrote:
>> What is the current thinking on these questions?
>>
>> To what extent do speed and power tests measure the same constructs?
>>
>> Do speed tests add validity over power tests?
>>
>> Thx
>>
>> Joel
>>
>>
>
> _______________________________________________________
> IPAC-List
> IPAC-List at ipacweb.org<redir.aspx?REF=o7Hz8ArglQlt5tx_PzT1d9Hl2DBnQ6ThnPFW9jINNxBxI7l2ForTCAFtYWlsdG86SVBBQy1MaXN0QGlwYWN3ZWIub3Jn>
> https://pairlist9.pair.net/mailman/listinfo/ipac-list<redir.aspx?REF=xey8N9KH_vFKEW66W_R09TbaIbfivdfrlZPk3ND53etxI7l2ForTCAFodHRwczovL3BhaXJsaXN0OS5wYWlyLm5ldC9tYWlsbWFuL2xpc3RpbmZvL2lwYWMtbGlzdA..>
>

_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org<redir.aspx?REF=o7Hz8ArglQlt5tx_PzT1d9Hl2DBnQ6ThnPFW9jINNxBxI7l2ForTCAFtYWlsdG86SVBBQy1MaXN0QGlwYWN3ZWIub3Jn>
https://pairlist9.pair.net/mailman/listinfo/ipac-list<redir.aspx?REF=xey8N9KH_vFKEW66W_R09TbaIbfivdfrlZPk3ND53etxI7l2ForTCAFodHRwczovL3BhaXJsaXN0OS5wYWlyLm5ldC9tYWlsbWFuL2xpc3RpbmZvL2lwYWMtbGlzdA..>

_______________________________________________________
IPAC-List
IPAC-List at ipacweb.org<redir.aspx?REF=o7Hz8ArglQlt5tx_PzT1d9Hl2DBnQ6ThnPFW9jINNxBxI7l2ForTCAFtYWlsdG86SVBBQy1MaXN0QGlwYWN3ZWIub3Jn>
https://pairlist9.pair.net/mailman/listinfo/ipac-list<redir.aspx?REF=xey8N9KH_vFKEW66W_R09TbaIbfivdfrlZPk3ND53etxI7l2ForTCAFodHRwczovL3BhaXJsaXN0OS5wYWlyLm5ldC9tYWlsbWFuL2xpc3RpbmZvL2lwYWMtbGlzdA..>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist9.pair.net/pipermail/ipac-list/attachments/20160601/48c9ff52/attachment.html>