The pipeline project: Pre-publication independent replications of a single laboratory's research pipeline

Martin Schweinsberg; Nikhil Madan; Michelangelo Vianello; S. Amy Sommer; Jennifer Jordan; Warren Tierney; Eli Awtrey; Luke Lei Zhu; Daniel Diermeier; Justin E. Heinze; Malavika Srinivasan; David Tannenbaum; Eliza Bivolaru; Jason Dana; Clintin P. Davis-Stober; Christilene du Plessis; Quentin F. Gronau; Andrew C. Hafenbrack; Eko Yi Liao; Alexander Ly; Maarten Marsman; Toshio Murase; Israr Qureshi; Michael Schaerer; Nico Thornley; Christina M. Tworek; Eric-Jan Wagenmakers; Lynn Wong; Tabitha Anderson; Christopher W. Bauman; Wendy L. Bedwell; Victoria Brescoll; Andrew Canavan; Jesse J. Chandler; Erik Cheries; Sapna Cheryan; Felix Cheung; Andrei Cimpian; Mark A. Clark; Diana Cordon; Fiery Cushman; Peter H. Ditto; Thomas Donahue; Sarah E. Frick; Monica Gamez-Djokic; Rebecca Hofstein Grady; Jesse Graham; Jun Gu; Adam Hahn; Brittany E. Hanson; Nicole J. Hartwich; Kristie Hein; Yoel Inbar; Lily Jiang; Tehlyr Kellogg; Deanna M. Kennedy; Nicole Legate; Timo P. Luoma; Heidi Maibuecher; Peter Meindl; Jennifer Miles; Alexandra Mislin; Daniel C. Molden; Matt Motyl; George Newman; Hoai Huong Ngo; Harvey Packham; Philip S. Ramsay; Jennifer L. Ray; Aaron M. Sackett; Anne Laure Sellier; Tatiana Sokolova; Walter Sowden; Daniel Storage; Xiaomin Sun; Jay J. Van Bavel; Anthony N. Washburn; Cong Wei; Erik Wetter; Carlos T. Wilson; Sophie-Charlotte Darroux; Eric Luis Uhlmann

doi:10.1016/j.jesp.2015.10.001

The pipeline project: Pre-publication independent replications of a single laboratory's research pipeline

Martin Schweinsberg^*, Nikhil Madan, Michelangelo Vianello, S. Amy Sommer, Jennifer Jordan, Warren Tierney, Eli Awtrey, Luke Lei Zhu, Daniel Diermeier, Justin E. Heinze, Malavika Srinivasan, David Tannenbaum, Eliza Bivolaru, Jason Dana, Clintin P. Davis-Stober, Christilene du Plessis, Quentin F. Gronau, Andrew C. Hafenbrack, Eko Yi Liao, Alexander LyMaarten Marsman, Toshio Murase, Israr Qureshi, Michael Schaerer, Nico Thornley, Christina M. Tworek, Eric-Jan Wagenmakers, Lynn Wong, Tabitha Anderson, Christopher W. Bauman, Wendy L. Bedwell, Victoria Brescoll, Andrew Canavan, Jesse J. Chandler, Erik Cheries, Sapna Cheryan, Felix Cheung, Andrei Cimpian, Mark A. Clark, Diana Cordon, Fiery Cushman, Peter H. Ditto, Thomas Donahue, Sarah E. Frick, Monica Gamez-Djokic, Rebecca Hofstein Grady, Jesse Graham, Jun Gu, Adam Hahn, Brittany E. Hanson, Nicole J. Hartwich, Kristie Hein, Yoel Inbar, Lily Jiang, Tehlyr Kellogg, Deanna M. Kennedy, Nicole Legate, Timo P. Luoma, Heidi Maibuecher, Peter Meindl, Jennifer Miles, Alexandra Mislin, Daniel C. Molden, Matt Motyl, George Newman, Hoai Huong Ngo, Harvey Packham, Philip S. Ramsay, Jennifer L. Ray, Aaron M. Sackett, Anne Laure Sellier, Tatiana Sokolova, Walter Sowden, Daniel Storage, Xiaomin Sun, Jay J. Van Bavel, Anthony N. Washburn, Cong Wei, Erik Wetter, Carlos T. Wilson, Sophie-Charlotte Darroux, Eric Luis Uhlmann

^*Corresponding author for this work

Research output: Contribution to journal › Article › peer-review

82 Citations (Scopus)

33 Downloads (Pure)

Abstract

This crowdsourced project introduces a collaborative approach to improving the reproducibility of scientific research, in which findings are replicated in qualified independent laboratories before (rather than after) they are published. Our goal is to establish a non-adversarial replication process with highly informative final results. To illustrate the Pre-Publication Independent Replication (PPIR) approach, 25 research groups conducted replications of all ten moral judgment effects which the last author and his collaborators had “in the pipeline” as of August 2014. Six findings replicated according to all replication criteria, one finding replicated but with a significantly smaller effect size than the original, one finding replicated consistently in the original culture but not outside of it, and two findings failed to find support. In total, 40% of the original findings failed at least one major replication criterion. Potential ways to implement and incentivize pre-publication independent replication on a large scale are discussed.

Original language	English
Pages (from-to)	55-67
Number of pages	13
Journal	Journal of Experimental Social Psychology
Volume	66
DOIs	https://doi.org/10.1016/j.jesp.2015.10.001
Publication status	Published - 1 Sept 2016
Externally published	Yes

Bibliographical note

Copyright © 2015 The Authors. Published by Elsevier Inc.. Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.

Keywords

Crowdsourcing science
Meta-science
Methodology
Replication
Reproducibility
Research transparency

Access to Document

10.1016/j.jesp.2015.10.001Licence: CC BY-NC-ND

Publisher version (open access)Final published version, 657 KBLicence: CC BY-NC-ND

Cite this

Schweinsberg, M., Madan, N., Vianello, M., Sommer, S. A., Jordan, J., Tierney, W., Awtrey, E., Zhu, L. L., Diermeier, D., Heinze, J. E., Srinivasan, M., Tannenbaum, D., Bivolaru, E., Dana, J., Davis-Stober, C. P., du Plessis, C., Gronau, Q. F., Hafenbrack, A. C., Liao, E. Y., ... Uhlmann, E. L. (2016). The pipeline project: Pre-publication independent replications of a single laboratory's research pipeline. Journal of Experimental Social Psychology, 66, 55-67. https://doi.org/10.1016/j.jesp.2015.10.001

@article{03aa4e438d904ff4a4cca6a2eda46bab,

title = "The pipeline project: Pre-publication independent replications of a single laboratory's research pipeline",

abstract = "This crowdsourced project introduces a collaborative approach to improving the reproducibility of scientific research, in which findings are replicated in qualified independent laboratories before (rather than after) they are published. Our goal is to establish a non-adversarial replication process with highly informative final results. To illustrate the Pre-Publication Independent Replication (PPIR) approach, 25 research groups conducted replications of all ten moral judgment effects which the last author and his collaborators had “in the pipeline” as of August 2014. Six findings replicated according to all replication criteria, one finding replicated but with a significantly smaller effect size than the original, one finding replicated consistently in the original culture but not outside of it, and two findings failed to find support. In total, 40% of the original findings failed at least one major replication criterion. Potential ways to implement and incentivize pre-publication independent replication on a large scale are discussed.",

keywords = "Crowdsourcing science, Meta-science, Methodology, Replication, Reproducibility, Research transparency",

author = "Martin Schweinsberg and Nikhil Madan and Michelangelo Vianello and Sommer, {S. Amy} and Jennifer Jordan and Warren Tierney and Eli Awtrey and Zhu, {Luke Lei} and Daniel Diermeier and Heinze, {Justin E.} and Malavika Srinivasan and David Tannenbaum and Eliza Bivolaru and Jason Dana and Davis-Stober, {Clintin P.} and {du Plessis}, Christilene and Gronau, {Quentin F.} and Hafenbrack, {Andrew C.} and Liao, {Eko Yi} and Alexander Ly and Maarten Marsman and Toshio Murase and Israr Qureshi and Michael Schaerer and Nico Thornley and Tworek, {Christina M.} and Eric-Jan Wagenmakers and Lynn Wong and Tabitha Anderson and Bauman, {Christopher W.} and Bedwell, {Wendy L.} and Victoria Brescoll and Andrew Canavan and Chandler, {Jesse J.} and Erik Cheries and Sapna Cheryan and Felix Cheung and Andrei Cimpian and Clark, {Mark A.} and Diana Cordon and Fiery Cushman and Ditto, {Peter H.} and Thomas Donahue and Frick, {Sarah E.} and Monica Gamez-Djokic and Grady, {Rebecca Hofstein} and Jesse Graham and Jun Gu and Adam Hahn and Hanson, {Brittany E.} and Hartwich, {Nicole J.} and Kristie Hein and Yoel Inbar and Lily Jiang and Tehlyr Kellogg and Kennedy, {Deanna M.} and Nicole Legate and Luoma, {Timo P.} and Heidi Maibuecher and Peter Meindl and Jennifer Miles and Alexandra Mislin and Molden, {Daniel C.} and Matt Motyl and George Newman and Ngo, {Hoai Huong} and Harvey Packham and Ramsay, {Philip S.} and Ray, {Jennifer L.} and Sackett, {Aaron M.} and Sellier, {Anne Laure} and Tatiana Sokolova and Walter Sowden and Daniel Storage and Xiaomin Sun and {Van Bavel}, {Jay J.} and Washburn, {Anthony N.} and Cong Wei and Erik Wetter and Wilson, {Carlos T.} and Sophie-Charlotte Darroux and Uhlmann, {Eric Luis}",

note = "Copyright {\textcopyright} 2015 The Authors. Published by Elsevier Inc.. Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.",

year = "2016",

month = sep,

day = "1",

doi = "10.1016/j.jesp.2015.10.001",

language = "English",

volume = "66",

pages = "55--67",

journal = "Journal of Experimental Social Psychology",

issn = "0022-1031",

publisher = "Elsevier",

}

Schweinsberg, M, Madan, N, Vianello, M, Sommer, SA, Jordan, J, Tierney, W, Awtrey, E, Zhu, LL, Diermeier, D, Heinze, JE, Srinivasan, M, Tannenbaum, D, Bivolaru, E, Dana, J, Davis-Stober, CP, du Plessis, C, Gronau, QF, Hafenbrack, AC, Liao, EY, Ly, A, Marsman, M, Murase, T, Qureshi, I, Schaerer, M, Thornley, N, Tworek, CM, Wagenmakers, E-J, Wong, L, Anderson, T, Bauman, CW, Bedwell, WL, Brescoll, V, Canavan, A, Chandler, JJ, Cheries, E, Cheryan, S, Cheung, F, Cimpian, A, Clark, MA, Cordon, D, Cushman, F, Ditto, PH, Donahue, T, Frick, SE, Gamez-Djokic, M, Grady, RH, Graham, J, Gu, J, Hahn, A, Hanson, BE, Hartwich, NJ, Hein, K, Inbar, Y, Jiang, L, Kellogg, T, Kennedy, DM, Legate, N, Luoma, TP, Maibuecher, H, Meindl, P, Miles, J, Mislin, A, Molden, DC, Motyl, M, Newman, G, Ngo, HH, Packham, H, Ramsay, PS, Ray, JL, Sackett, AM, Sellier, AL, Sokolova, T, Sowden, W, Storage, D, Sun, X, Van Bavel, JJ, Washburn, AN, Wei, C, Wetter, E, Wilson, CT, Darroux, S-C & Uhlmann, EL 2016, 'The pipeline project: Pre-publication independent replications of a single laboratory's research pipeline', Journal of Experimental Social Psychology, vol. 66, pp. 55-67. https://doi.org/10.1016/j.jesp.2015.10.001

TY - JOUR

T1 - The pipeline project

T2 - Pre-publication independent replications of a single laboratory's research pipeline

AU - Schweinsberg, Martin

AU - Madan, Nikhil

AU - Vianello, Michelangelo

AU - Sommer, S. Amy

AU - Jordan, Jennifer

AU - Tierney, Warren

AU - Awtrey, Eli

AU - Zhu, Luke Lei

AU - Diermeier, Daniel

AU - Heinze, Justin E.

AU - Srinivasan, Malavika

AU - Tannenbaum, David

AU - Bivolaru, Eliza

AU - Dana, Jason

AU - Davis-Stober, Clintin P.

AU - du Plessis, Christilene

AU - Gronau, Quentin F.

AU - Hafenbrack, Andrew C.

AU - Liao, Eko Yi

AU - Ly, Alexander

AU - Marsman, Maarten

AU - Murase, Toshio

AU - Qureshi, Israr

AU - Schaerer, Michael

AU - Thornley, Nico

AU - Tworek, Christina M.

AU - Wagenmakers, Eric-Jan

AU - Wong, Lynn

AU - Anderson, Tabitha

AU - Bauman, Christopher W.

AU - Bedwell, Wendy L.

AU - Brescoll, Victoria

AU - Canavan, Andrew

AU - Chandler, Jesse J.

AU - Cheries, Erik

AU - Cheryan, Sapna

AU - Cheung, Felix

AU - Cimpian, Andrei

AU - Clark, Mark A.

AU - Cordon, Diana

AU - Cushman, Fiery

AU - Ditto, Peter H.

AU - Donahue, Thomas

AU - Frick, Sarah E.

AU - Gamez-Djokic, Monica

AU - Grady, Rebecca Hofstein

AU - Graham, Jesse

AU - Gu, Jun

AU - Hahn, Adam

AU - Hanson, Brittany E.

AU - Hartwich, Nicole J.

AU - Hein, Kristie

AU - Inbar, Yoel

AU - Jiang, Lily

AU - Kellogg, Tehlyr

AU - Kennedy, Deanna M.

AU - Legate, Nicole

AU - Luoma, Timo P.

AU - Maibuecher, Heidi

AU - Meindl, Peter

AU - Miles, Jennifer

AU - Mislin, Alexandra

AU - Molden, Daniel C.

AU - Motyl, Matt

AU - Newman, George

AU - Ngo, Hoai Huong

AU - Packham, Harvey

AU - Ramsay, Philip S.

AU - Ray, Jennifer L.

AU - Sackett, Aaron M.

AU - Sellier, Anne Laure

AU - Sokolova, Tatiana

AU - Sowden, Walter

AU - Storage, Daniel

AU - Sun, Xiaomin

AU - Van Bavel, Jay J.

AU - Washburn, Anthony N.

AU - Wei, Cong

AU - Wetter, Erik

AU - Wilson, Carlos T.

AU - Darroux, Sophie-Charlotte

AU - Uhlmann, Eric Luis

N1 - Copyright © 2015 The Authors. Published by Elsevier Inc.. Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.

PY - 2016/9/1

Y1 - 2016/9/1

N2 - This crowdsourced project introduces a collaborative approach to improving the reproducibility of scientific research, in which findings are replicated in qualified independent laboratories before (rather than after) they are published. Our goal is to establish a non-adversarial replication process with highly informative final results. To illustrate the Pre-Publication Independent Replication (PPIR) approach, 25 research groups conducted replications of all ten moral judgment effects which the last author and his collaborators had “in the pipeline” as of August 2014. Six findings replicated according to all replication criteria, one finding replicated but with a significantly smaller effect size than the original, one finding replicated consistently in the original culture but not outside of it, and two findings failed to find support. In total, 40% of the original findings failed at least one major replication criterion. Potential ways to implement and incentivize pre-publication independent replication on a large scale are discussed.

AB - This crowdsourced project introduces a collaborative approach to improving the reproducibility of scientific research, in which findings are replicated in qualified independent laboratories before (rather than after) they are published. Our goal is to establish a non-adversarial replication process with highly informative final results. To illustrate the Pre-Publication Independent Replication (PPIR) approach, 25 research groups conducted replications of all ten moral judgment effects which the last author and his collaborators had “in the pipeline” as of August 2014. Six findings replicated according to all replication criteria, one finding replicated but with a significantly smaller effect size than the original, one finding replicated consistently in the original culture but not outside of it, and two findings failed to find support. In total, 40% of the original findings failed at least one major replication criterion. Potential ways to implement and incentivize pre-publication independent replication on a large scale are discussed.

KW - Crowdsourcing science

KW - Meta-science

KW - Methodology

KW - Replication

KW - Reproducibility

KW - Research transparency

UR - http://www.scopus.com/inward/record.url?scp=84977144634&partnerID=8YFLogxK

U2 - 10.1016/j.jesp.2015.10.001

DO - 10.1016/j.jesp.2015.10.001

M3 - Article

AN - SCOPUS:84977144634

SN - 0022-1031

VL - 66

SP - 55

EP - 67

JO - Journal of Experimental Social Psychology

JF - Journal of Experimental Social Psychology

ER -

The pipeline project: Pre-publication independent replications of a single laboratory's research pipeline

Abstract

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this