R. Silberzahn
Many analysts, one dataset: making transparent how variations in analytical choices affect results
Silberzahn, R.; Uhlmann, E.L.; Martin, D.P.; Anselmi, P.; Aust, F.; Awtrey, E.; Bahník, Š.; Bai, F.; Bannard, C.; Bonnier, E.; Carlsson, R.; Cheung, F.; Christensen, G.; Clay, R.; Craig, M.A.; Dalla Rosa, A.; Dam, L.; Evans, M.H.; Flores Cervantes, I.; Fong, N.; Gamez-Djokic, M.; Glenz, A.; Gordon-McKeon, S.; Heaton, T.J.; Hederos, K.; Heene, M.; Hofelich Mohr, A.J.; Högden, F.; Hui, K.; Johannesson, M.; Kalodimos, J.; Kaszubowski, E.; Kennedy, D.M.; Lei, R.; Lindsay, T.A.; Liverani, S.; Madan, C.R.; Molden, D.; Molleman, E.; Morey, R.D.; Mulder, L.B.; Nijstad, B.R.; Pope, N.G.; Pope, B.; Prenoveau, J.M.; Rink, F.; Robusto, E.; Roderique, H.; Sandberg, A.; Schlüter, E.; Schönbrodt, F.D.; Sherman, M.F.; Sommer, S.A.; Sotak, K.; Spain, S.; Spörlein, C.; Stafford, T.; Stefanutti, L.; Tauber, S.; Ullrich, J.; Vianello, M.; Wagenmakers, E.-J.; Witkowiak, M.; Yoon, S.; Nosek, B.A.
Authors
E.L. Uhlmann
D.P. Martin
P. Anselmi
F. Aust
E. Awtrey
Š. Bahník
F. Bai
C. Bannard
E. Bonnier
R. Carlsson
F. Cheung
G. Christensen
R. Clay
M.A. Craig
A. Dalla Rosa
L. Dam
M.H. Evans
I. Flores Cervantes
N. Fong
M. Gamez-Djokic
A. Glenz
S. Gordon-McKeon
T.J. Heaton
K. Hederos
M. Heene
A.J. Hofelich Mohr
F. Högden
K. Hui
M. Johannesson
J. Kalodimos
E. Kaszubowski
D.M. Kennedy
R. Lei
T.A. Lindsay
S. Liverani
CHRISTOPHER MADAN CHRISTOPHER.MADAN@NOTTINGHAM.AC.UK
Assistant Professor
D. Molden
E. Molleman
R.D. Morey
L.B. Mulder
B.R. Nijstad
N.G. Pope
B. Pope
J.M. Prenoveau
F. Rink
E. Robusto
H. Roderique
A. Sandberg
E. Schlüter
F.D. Schönbrodt
M.F. Sherman
S.A. Sommer
K. Sotak
S. Spain
C. Spörlein
T. Stafford
L. Stefanutti
S. Tauber
J. Ullrich
M. Vianello
E.-J. Wagenmakers
M. Witkowiak
S. Yoon
B.A. Nosek
Abstract
Twenty-nine teams involving 61 analysts used the same dataset to address the same research question: whether soccer referees are more likely to give red cards to dark skin toned players than light skin toned players. Analytic approaches varied widely across teams, and estimated effect sizes ranged from 0.89 to 2.93 in odds ratio units, with a median of 1.31. Twenty teams (69%) found a statistically significant positive effect and nine teams (31%) observed a nonsignificant relationship. Overall 29 different analyses used 21 unique combinations of covariates. We found that neither analysts' prior beliefs about the effect, nor their level of expertise, nor peer-reviewed quality of analysis readily explained variation in analysis outcomes. This suggests that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy by which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective analytic choices influence research results.
Citation
Silberzahn, R., Uhlmann, E., Martin, D., Anselmi, P., Aust, F., Awtrey, E., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., Carlsson, R., Cheung, F., Christensen, G., Clay, R., Craig, M., Dalla Rosa, A., Dam, L., Evans, M., Flores Cervantes, I., Fong, N., …Nosek, B. (2018). Many analysts, one dataset: making transparent how variations in analytical choices affect results. Advances in Methods and Practices in Psychological Science, 2(1), 337-356. https://doi.org/10.1177/2515245917747646
Journal Article Type | Article |
---|---|
Acceptance Date | Sep 25, 2017 |
Online Publication Date | Aug 23, 2018 |
Publication Date | Aug 23, 2018 |
Deposit Date | Nov 15, 2017 |
Publicly Available Date | Aug 23, 2018 |
Journal | Advances in Methods and Practices in Psychological Science |
Print ISSN | 2515-2459 |
Electronic ISSN | 2515-2467 |
Publisher | SAGE Publications |
Peer Reviewed | Peer Reviewed |
Volume | 2 |
Issue | 1 |
Pages | 337-356 |
DOI | https://doi.org/10.1177/2515245917747646 |
Public URL | https://nottingham-repository.worktribe.com/output/884400 |
Publisher URL | http://journals.sagepub.com/doi/10.1177/2515245917747646 |
Contract Date | Nov 15, 2017 |
Files
Crowdsourcing Analytics - Final Manuscript - 201709.pdf
(251 Kb)
PDF
Publisher Licence URL
https://creativecommons.org/licenses/by-nc/4.0/
Copyright Statement
Copyright information regarding this work can be found at the following address: http://eprints.nottingham.ac.uk/end_user_agreement.pdf
You might also like
Shock and awe: distinct effects of taboo words on lexical decision and free recall
(2017)
Journal Article
Priming memories of past wins induces risk seeking
(2014)
Journal Article
Temporal summation of global form signals in dynamic glass patterns
(2014)
Journal Article
High reward makes items easier to remember, but harder to bind to a new temporal context
(2012)
Journal Article
Cortical complexity as a measure of age-related brain atrophy
(2016)
Journal Article
Downloadable Citations
About Repository@Nottingham
Administrator e-mail: discovery-access-systems@nottingham.ac.uk
This application uses the following open-source libraries:
SheetJS Community Edition
Apache License Version 2.0 (http://www.apache.org/licenses/)
PDF.js
Apache License Version 2.0 (http://www.apache.org/licenses/)
Font Awesome
SIL OFL 1.1 (http://scripts.sil.org/OFL)
MIT License (http://opensource.org/licenses/mit-license.html)
CC BY 3.0 ( http://creativecommons.org/licenses/by/3.0/)
Powered by Worktribe © 2024
Advanced Search