Skip to main content

Research Repository

See what's under the surface

Advanced Search

Many analysts, one dataset: making transparent how variations in analytical choices affect results

Silberzahn, R.; Uhlmann, E.L.; Martin, D.P.; Anselmi, P.; Aust, F.; Awtrey, E.; Bahník, Š.; Bai, F.; Bannard, C.; Bonnier, E.; Carlsson, R.; Cheung, F.; Christensen, G.; Clay, R.; Craig, M.A.; Dalla Rosa, A.; Dam, L.; Evans, M.H.; Flores Cervantes, I.; Fong, N.; Gamez-Djokic, M.; Glenz, A.; Gordon-McKeon, S.; Heaton, T.J.; Hederos, K.; Heene, M.; Hofelich Mohr, A.J.; Högden, F.; Hui, K.; Johannesson, M.; Kalodimos, J.; Kaszubowski, E.; Kennedy, D.M.; Lei, R.; Lindsay, T.A.; Liverani, S.; Madan, C.R.; Molden, D.; Molleman, E.; Morey, R.D.; Mulder, L.B.; Nijstad, B.R.; Pope, N.G.; Pope, B.; Prenoveau, J.M.; Rink, F.; Robusto, E.; Roderique, H.; Sandberg, A.; Schlüter, E.; Schönbrodt, F.D.; Sherman, M.F.; Sommer, S.A.; Sotak, K.; Spain, S.; Spörlein, C.; Stafford, T.; Stefanutti, L.; Tauber, S.; Ullrich, J.; Vianello, M.; Wagenmakers, E.-J.; Witkowiak, M.; Yoon, S.; Nosek, B.A.


R. Silberzahn

E.L. Uhlmann

D.P. Martin

P. Anselmi

F. Aust

E. Awtrey

Š. Bahník

F. Bai

C. Bannard

E. Bonnier

R. Carlsson

F. Cheung

G. Christensen

R. Clay

M.A. Craig

A. Dalla Rosa

L. Dam

M.H. Evans

I. Flores Cervantes

N. Fong

M. Gamez-Djokic

A. Glenz

S. Gordon-McKeon

T.J. Heaton

K. Hederos

M. Heene

A.J. Hofelich Mohr

F. Högden

K. Hui

M. Johannesson

J. Kalodimos

E. Kaszubowski

D.M. Kennedy

R. Lei

T.A. Lindsay

S. Liverani

C.R. Madan

D. Molden

E. Molleman

R.D. Morey

L.B. Mulder

B.R. Nijstad

N.G. Pope

B. Pope

J.M. Prenoveau

F. Rink

E. Robusto

H. Roderique

A. Sandberg

E. Schlüter

F.D. Schönbrodt

M.F. Sherman

S.A. Sommer

K. Sotak

S. Spain

C. Spörlein

T. Stafford

L. Stefanutti

S. Tauber

J. Ullrich

M. Vianello

E.-J. Wagenmakers

M. Witkowiak

S. Yoon

B.A. Nosek


Twenty-nine teams involving 61 analysts used the same dataset to address the same research question: whether soccer referees are more likely to give red cards to dark skin toned players than light skin toned players. Analytic approaches varied widely across teams, and estimated effect sizes ranged from 0.89 to 2.93 in odds ratio units, with a median of 1.31. Twenty teams (69%) found a statistically significant positive effect and nine teams (31%) observed a nonsignificant relationship. Overall 29 different analyses used 21 unique combinations of covariates. We found that neither analysts' prior beliefs about the effect, nor their level of expertise, nor peer-reviewed quality of analysis readily explained variation in analysis outcomes. This suggests that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy by which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective analytic choices influence research results.

Journal Article Type Article
Publication Date Aug 23, 2018
Journal Advances in Methods and Practices in Psychological Science
Print ISSN 2515-2459
Electronic ISSN 2515-2467
Publisher SAGE Publications
Peer Reviewed Peer Reviewed
Volume 2
Issue 1
Pages 337-356
APA6 Citation Silberzahn, R., Uhlmann, E., Martin, D., Anselmi, P., Aust, F., Awtrey, E., …Nosek, B. (2018). Many analysts, one dataset: making transparent how variations in analytical choices affect results. Advances in Methods and Practices in Psychological Science, 2(1), (337-356). doi:10.1177/2515245917747646. ISSN 2515-2459
Publisher URL
Copyright Statement Copyright information regarding this work can be found at the following address: http://eprints.nottingh.../end_user_agreement.pdf
Additional Information

Composition Type:


Crowdsourcing Analytics - Final Manuscript - 201709.pdf (251 Kb)

Copyright Statement
Copyright information regarding this work can be found at the following address:

You might also like

Downloadable Citations