Examining the generalizability of research findings from archival data

Andrew Delios; Elena Giulia Clemente; Tao Wu; Hongbin Tan; Yong Wang; Michael Gordon; Domenico Viganola; Zhaowei Chen; Anna Dreber; Magnus Johannesson; Thomas Pfeiffer; Generalizability Tests Forecasting Collaboration; Eric Luis Uhlmann

doi:10.1073/pnas.2120377119

Examining the generalizability of research findings from archival data

Andrew Delios^*, Elena Giulia Clemente, Tao Wu, Hongbin Tan, Yong Wang, Michael Gordon, Domenico Viganola, Zhaowei Chen, Anna Dreber, Magnus Johannesson, Thomas Pfeiffer, Generalizability Tests Forecasting Collaboration, Eric Luis Uhlmann^*

^*Corresponding author for this work

Research output: Contribution to journal › Article › peer-review

21 Citations (Scopus)

45 Downloads (Pure)

Abstract

This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability—for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.

Original language	English
Article number	e2120377119
Pages (from-to)	1-9
Number of pages	9
Journal	Proceedings of the National Academy of Sciences of the United States of America
Volume	119
Issue number	30
DOIs	https://doi.org/10.1073/pnas.2120377119
Publication status	Published - 19 Jul 2022

Bibliographical note

Copyright the Author(s) 2022. Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.

Keywords

research reliability
generalizability
archival data
reproducibility
context sensitivity

Access to Document

10.1073/pnas.2120377119Licence: CC BY-NC-ND

Publisher version (open access)Final published version, 850 KBLicence: CC BY-NC-ND

Cite this

Delios, A., Clemente, E. G., Wu, T., Tan, H., Wang, Y., Gordon, M., Viganola, D., Chen, Z., Dreber, A., Johannesson, M., Pfeiffer, T., Generalizability Tests Forecasting Collaboration, & Uhlmann, E. L. (2022). Examining the generalizability of research findings from archival data. Proceedings of the National Academy of Sciences of the United States of America, 119(30), 1-9. Article e2120377119. https://doi.org/10.1073/pnas.2120377119

@article{d5b1ce2608a34693bf7b663e42c5fa74,

title = "Examining the generalizability of research findings from archival data",

abstract = "This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability—for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.",

keywords = "research reliability, generalizability, archival data, reproducibility, context sensitivity",

author = "Andrew Delios and Clemente, {Elena Giulia} and Tao Wu and Hongbin Tan and Yong Wang and Michael Gordon and Domenico Viganola and Zhaowei Chen and Anna Dreber and Magnus Johannesson and Thomas Pfeiffer and {Generalizability Tests Forecasting Collaboration} and {Abd Al-Aziz}, {Ahmad M.} and Abraham, {Ajay T.} and Jais Trojan and Mat{\'u}{\v s} Adamkovi{\v c} and Elena Agadullina and Jungsoo Ahn and Cinla Akinci and Handan Akkas and David Albrecht and Shilaan Alzahawi and Marcio Amaral-Baptista and Rahul Anand and Ang, {Kevin Francis U.} and Frederik Anseel and Aruta, {John Jamir Benzon R.} and Mujeeba Ashraf and Baker, {Bradley J.} and Xueqi Bao and Ernest Baskin and Hanoku Bathula and Bauman, {Christopher W.} and Jozef Bavolar and Secil Bayraktar and Beckman, {Stephanie E.} and Benjamin, {Aaron S.} and Brown, {Stephanie E. V.} and Jeffrey Buckley and {Buitrago R.}, {Ricardo E.} and Bution, {Jefferson L.} and Nick Byrd and Clara Carrera and Caruso, {Eugene M.} and Minxia Chen and Lin Chen and Cicerali, {Eyyub Ensari} and Cohen, {Eric D.} and Marcus Crede and Jamie Cummins and Linus Dahlander and Daniels, {David P.} and Daskalo, {Lea Liat} and Dawson, {Ian G. J.} and Erik Dietl and Artur Domurat and Jacinta Dsilva and {du Plessis}, Christilene and Dubrov, {Dmitrii I.} and Sarah Edris and Elbaek, {Christian T.} and Elsherif, {Mahmoud M.} and Evans, {Thomas R.} and Fellenz, {Martin R.} and Susann Fiedler and Mustafa Firat and Raquel Freitag and Furrer, {R{\'e}my A.} and Richa Gautam and Gautam, {Dhruba Kumar} and Brian Gearin and Stephan Gerschewski and Omid Ghasemi and Zohreh Ghasemi and Anindya Ghosh and Cinzia Giani and Goldberg, {Matthew H.} and Manisha Goswami and Lorenz Graf-Vlachy and Griffith, {Jennifer A.} and Dmitry Grigoryev and Jingyang Gu and Rajeshwari H and Hadida, {Allegre L.} and Hafenbrack, {Andrew C.} and Sebastian Hafenbr{\"a}dl and Hammersley, {Jonathan J.} and Hyemin Han and Harman, {Jason L.} and Andree Hartanto and Henkel, {Alexander P.} and Yen-Chen Ho and Holding, {Benjamin C.} and Felix Holzmeister and Alexandra Horobet and Huang, {Tina S.-T.} and Yiming Huang and Huntsinger, {Jeffrey R.} and Katarzyna Idzikowska and Hirotaka Imada and Rabia Imran and Ingels, {Michael J.} and Bastian Jaeger and Janssen, {Steve M. J.} and Fanli Jia and Alfredo Jim{\'e}nez and Jin, {Jason Lu} and Niklas Johannes and Daniel Jolles and Bibiana Jozefiakova and Pavol Ka{\v c}m{\'a}r and Tamara Kalandadze and Kyriaki Kalimeri and Polly Kang and Jaroslaw Kantorowicz and Didar Karadağ and Hamid Karimi-Rouzbahani and Kee, {Daisy Mui Hung} and Lucas Keller and Haider Khan and Mikael Knutsson and Olga Kombeiz and Aleksey Korniychuk and Marta Kowal and Johannes Leder and Liang, {Liang Wenhao} and Liew, {Taegyeong (Tae)} and Fangwen Lin and Chengwei Liu and Bin Liu and Longo, {Maria Cristina} and Andrey Lovakov and Low, {Mei Peng} and Lucas, {Gerardus J. M.} and Oliver Lukason and Ly, {Albert L.} and Zhuoran Ma and Alexander Mafael and Mahar, {Elizabeth A.} and Soheil Mahmoudkalayeh and David Manheim and Alfred Marcus and Marsh, {Melvin S.} and Martin, {Jolie M.} and Martinez, {Luis E.} and Mario Martinoli and Marcel Marton{\v c}ik and Masters-Waage, {Theodore C.} and Rui Mata and Hamid Mazloomi and McCarthy, {Randy J.} and Philip Millroth and Mahima Mishra and Supriti Mishra and Alexander Mohr and David Moreau and Annalisa Myer and Amos Nadler and {Sudhir Nair}, {Peter B.} and Gustav Nilsonne and Pawe{\l} Niszczota and Aoife O'Mahony and Marc Oberhauser and Tomasz Obloj and Orhan, {Mehmet A.} and Oswald, {Flora E.} and Tobias Otterbring and Otto, {Philipp E.} and Ivar Padr{\'o}n-Hern{\'a}ndez and Pan, {Alan J.} and Mariola Paruzel-Czachura and Gerit Pfuhl and Angelo Pirrone and Simon Porcher and John Protzko and Constantin Prox and Shelly Qi and Rima-Maria Rahal and Rahman, {Md. Shahinoor} and Reina, {Michelle L.} and Satyanarayana Rentala and Zahid Riaz and Ivan Ropovik and Lukas R{\"o}seler and Robert Ross and Uhlmann, {Eric Luis}",

note = "Copyright the Author(s) 2022. Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.",

year = "2022",

month = jul,

day = "19",

doi = "10.1073/pnas.2120377119",

language = "English",

volume = "119",

pages = "1--9",

journal = "Proceedings of the National Academy of Sciences of the United States of America",

issn = "0027-8424",

publisher = "National Academy of Sciences",

number = "30",

}

Delios, A, Clemente, EG, Wu, T, Tan, H, Wang, Y, Gordon, M, Viganola, D, Chen, Z, Dreber, A, Johannesson, M, Pfeiffer, T, Generalizability Tests Forecasting Collaboration & Uhlmann, EL 2022, 'Examining the generalizability of research findings from archival data', Proceedings of the National Academy of Sciences of the United States of America, vol. 119, no. 30, e2120377119, pp. 1-9. https://doi.org/10.1073/pnas.2120377119

TY - JOUR

T1 - Examining the generalizability of research findings from archival data

AU - Delios, Andrew

AU - Clemente, Elena Giulia

AU - Wu, Tao

AU - Tan, Hongbin

AU - Wang, Yong

AU - Gordon, Michael

AU - Viganola, Domenico

AU - Chen, Zhaowei

AU - Dreber, Anna

AU - Johannesson, Magnus

AU - Pfeiffer, Thomas

AU - Generalizability Tests Forecasting Collaboration

AU - Abd Al-Aziz, Ahmad M.

AU - Abraham, Ajay T.

AU - Trojan, Jais

AU - Adamkovič, Matúš

AU - Agadullina, Elena

AU - Ahn, Jungsoo

AU - Akinci, Cinla

AU - Akkas, Handan

AU - Albrecht, David

AU - Alzahawi, Shilaan

AU - Amaral-Baptista, Marcio

AU - Anand, Rahul

AU - Ang, Kevin Francis U.

AU - Anseel, Frederik

AU - Aruta, John Jamir Benzon R.

AU - Ashraf, Mujeeba

AU - Baker, Bradley J.

AU - Bao, Xueqi

AU - Baskin, Ernest

AU - Bathula, Hanoku

AU - Bauman, Christopher W.

AU - Bavolar, Jozef

AU - Bayraktar, Secil

AU - Beckman, Stephanie E.

AU - Benjamin, Aaron S.

AU - Brown, Stephanie E. V.

AU - Buckley, Jeffrey

AU - Buitrago R., Ricardo E.

AU - Bution, Jefferson L.

AU - Byrd, Nick

AU - Carrera, Clara

AU - Caruso, Eugene M.

AU - Chen, Minxia

AU - Chen, Lin

AU - Cicerali, Eyyub Ensari

AU - Cohen, Eric D.

AU - Crede, Marcus

AU - Cummins, Jamie

AU - Dahlander, Linus

AU - Daniels, David P.

AU - Daskalo, Lea Liat

AU - Dawson, Ian G. J.

AU - Dietl, Erik

AU - Domurat, Artur

AU - Dsilva, Jacinta

AU - du Plessis, Christilene

AU - Dubrov, Dmitrii I.

AU - Edris, Sarah

AU - Elbaek, Christian T.

AU - Elsherif, Mahmoud M.

AU - Evans, Thomas R.

AU - Fellenz, Martin R.

AU - Fiedler, Susann

AU - Firat, Mustafa

AU - Freitag, Raquel

AU - Furrer, Rémy A.

AU - Gautam, Richa

AU - Gautam, Dhruba Kumar

AU - Gearin, Brian

AU - Gerschewski, Stephan

AU - Ghasemi, Omid

AU - Ghasemi, Zohreh

AU - Ghosh, Anindya

AU - Giani, Cinzia

AU - Goldberg, Matthew H.

AU - Goswami, Manisha

AU - Graf-Vlachy, Lorenz

AU - Griffith, Jennifer A.

AU - Grigoryev, Dmitry

AU - Gu, Jingyang

AU - H, Rajeshwari

AU - Hadida, Allegre L.

AU - Hafenbrack, Andrew C.

AU - Hafenbrädl, Sebastian

AU - Hammersley, Jonathan J.

AU - Han, Hyemin

AU - Harman, Jason L.

AU - Hartanto, Andree

AU - Henkel, Alexander P.

AU - Ho, Yen-Chen

AU - Holding, Benjamin C.

AU - Holzmeister, Felix

AU - Horobet, Alexandra

AU - Huang, Tina S.-T.

AU - Huang, Yiming

AU - Huntsinger, Jeffrey R.

AU - Idzikowska, Katarzyna

AU - Imada, Hirotaka

AU - Imran, Rabia

AU - Ingels, Michael J.

AU - Jaeger, Bastian

AU - Janssen, Steve M. J.

AU - Jia, Fanli

AU - Jiménez, Alfredo

AU - Jin, Jason Lu

AU - Johannes, Niklas

AU - Jolles, Daniel

AU - Jozefiakova, Bibiana

AU - Kačmár, Pavol

AU - Kalandadze, Tamara

AU - Kalimeri, Kyriaki

AU - Kang, Polly

AU - Kantorowicz, Jaroslaw

AU - Karadağ, Didar

AU - Karimi-Rouzbahani, Hamid

AU - Kee, Daisy Mui Hung

AU - Keller, Lucas

AU - Khan, Haider

AU - Knutsson, Mikael

AU - Kombeiz, Olga

AU - Korniychuk, Aleksey

AU - Kowal, Marta

AU - Leder, Johannes

AU - Liang, Liang Wenhao

AU - Liew, Taegyeong (Tae)

AU - Lin, Fangwen

AU - Liu, Chengwei

AU - Liu, Bin

AU - Longo, Maria Cristina

AU - Lovakov, Andrey

AU - Low, Mei Peng

AU - Lucas, Gerardus J. M.

AU - Lukason, Oliver

AU - Ly, Albert L.

AU - Ma, Zhuoran

AU - Mafael, Alexander

AU - Mahar, Elizabeth A.

AU - Mahmoudkalayeh, Soheil

AU - Manheim, David

AU - Marcus, Alfred

AU - Marsh, Melvin S.

AU - Martin, Jolie M.

AU - Martinez, Luis E.

AU - Martinoli, Mario

AU - Martončik, Marcel

AU - Masters-Waage, Theodore C.

AU - Mata, Rui

AU - Mazloomi, Hamid

AU - McCarthy, Randy J.

AU - Millroth, Philip

AU - Mishra, Mahima

AU - Mishra, Supriti

AU - Mohr, Alexander

AU - Moreau, David

AU - Myer, Annalisa

AU - Nadler, Amos

AU - Sudhir Nair, Peter B.

AU - Nilsonne, Gustav

AU - Niszczota, Paweł

AU - O'Mahony, Aoife

AU - Oberhauser, Marc

AU - Obloj, Tomasz

AU - Orhan, Mehmet A.

AU - Oswald, Flora E.

AU - Otterbring, Tobias

AU - Otto, Philipp E.

AU - Padrón-Hernández, Ivar

AU - Pan, Alan J.

AU - Paruzel-Czachura, Mariola

AU - Pfuhl, Gerit

AU - Pirrone, Angelo

AU - Porcher, Simon

AU - Protzko, John

AU - Prox, Constantin

AU - Qi, Shelly

AU - Rahal, Rima-Maria

AU - Rahman, Md. Shahinoor

AU - Reina, Michelle L.

AU - Rentala, Satyanarayana

AU - Riaz, Zahid

AU - Ropovik, Ivan

AU - Röseler, Lukas

AU - Ross, Robert

AU - Uhlmann, Eric Luis

N1 - Copyright the Author(s) 2022. Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.

PY - 2022/7/19

Y1 - 2022/7/19

N2 - This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability—for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.

AB - This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability—for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.

KW - research reliability

KW - generalizability

KW - archival data

KW - reproducibility

KW - context sensitivity

UR - http://www.scopus.com/inward/record.url?scp=85134491120&partnerID=8YFLogxK

U2 - 10.1073/pnas.2120377119

DO - 10.1073/pnas.2120377119

M3 - Article

C2 - 35858443

SN - 0027-8424

VL - 119

SP - 1

EP - 9

JO - Proceedings of the National Academy of Sciences of the United States of America

JF - Proceedings of the National Academy of Sciences of the United States of America

IS - 30

M1 - e2120377119

ER -

Examining the generalizability of research findings from archival data

Abstract

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this