[IPAC-List] "False Positive Psychology"

[Mark Hammer]

Dennis can confirm, or clarify, but one of the recurring challenges in
the field of aging research is that the "exciting" result can often come
from the demonstration of NO difference between younger and older
individuals; for example, in the case of manipulations intended to
improve the performance of older individuals in some manner.  This tends
to reward researchers for not having enough statistical power. 
Hopefully, able researchers pay attention to this weak link.
 
Mark Hammer,
Ottawa


>>> Dennis Doverspike <dennisdoverspike at gmail.com> 2011/11/29 12:24 PM

>>>

Reid,

The article makes a great point. Although I would argue that the term
"authors" or "researchers" should not be take too narrowly. The problem
is
perhaps even more acute among consultants under pressure to report
statistically significant results from small, operational sample sizes.
Of
course, it reaches its pinnacle in our search for adverse impact,
where
regulatory agencies may require conducting significance tests on large
number of job groups and then narrow attention to the one job group
demonstrating statistically significant differences.

Thanks to bringing this article to our attention.

Dennis

On Tue, Nov 29, 2011 at 11:59 AM, Reid Klion <RKlion at panpowered.com>
wrote:


> There is an interesting article in Psychological Science (free

> download!: http://pss.sagepub.com/content/22/11/1359.full.pdf) on

the

> perils of what the authors call unchecked "researcher degrees of

> freedom" that can readily lead to false positive findings in the

> never-ending search for "statistically significant" publishable

results.

> They point out how when researchers don't disclose things like all

the

> variables collected in the study, failed manipulations, and the

specific

> rationale for the statistical analysis that was ultimately used, the

> probability of obtaining false positive findings increases (which

they

> demonstrate by "showing" how subjects listening to the Beatles song

"

> When I'm Sixty-Four" makes them older vs. subjects who listened to

other

> music.) They go on to provide useful suggestions for both authors

and

> reviewers to help mitigate these risks.

>

>

>

> Reid



>

>

> Reid E. Klion, Ph.D.

>

> Chief Science Officer

>

> pan - A TALX Company

>

> 11590 North Meridian St.

>

> Suite 200

>

> Carmel, IN 46032 USA

>

> 317.814.8808  Office

>

> 317.908.4312  Mobile

>

> 317.814.8888  Fax

>

> rklion at panpowered.com <mailto:rklion at panpowered.com>

>

> www.panpowered.com <http://www.panpowered.com/>



>

This e-mail message is intended for the named recipient(s) and may
contain information that is privileged, confidential and/or exempt from
disclosure under applicable law. Unauthorized disclosure, copying or
re-transmission is prohibited. If you are not a named recipient or not
authorized by the named recipient(s), or if you have received this
e-mail in error, then please notify the sender immediately and delete
the message and any copies.
 >
Ce courriel est destiné exclusivement au destinataire mentionné en titre
et peut contenir de l'information privilégiée, confidentielle ou
soustraite à la communication aux termes des lois applicables. Toute
divulgation non autorisée, toute reproduction ou réacheminement est
interdit. Si vous n'êtes pas le destinataire de ce courriel, ou n'êtes
pas autorisé par le destinataire visé, ou encore, si vous l'avez reçu
par erreur, veuillez le mentionner immédiatement à l'expéditeur et
supprimer le courriel et les copies.