Does health informatics have a replication crisis?

Research output: Contribution to journalArticleResearchpeer-review

Abstract

Objective: Many research fields, including psychology and basic medical sciences, struggle with poor reproducibility of reported studies. Biomedical and health informatics is unlikely to be immune to these challenges. This paper explores replication in informatics and the unique challenges the discipline faces. Methods: Narrative review of recent literature on research replication challenges. Results: While there is growing interest in re-analysis of existing data, experimental replication studies appear uncommon in informatics. Context effects are a particular challenge as they make ensuring replication fidelity difficult, and the same intervention will never quite reproduce the same result in different settings. Replication studies take many forms, trading-off testing validity of past findings against testing generalizability. Exact and partial replication designs emphasize testing validity while quasi and conceptual studies test generalizability of an underlying model or hypothesis with different methods or in a different setting. Conclusions: The cost of poor replication is a weakening in the quality of published research and the evidence-based foundation of health informatics. The benefits of replication include increased rigor in research, and the development of evaluation methods that distinguish the impact of context and the nonreproducibility of research. Taking replication seriously is essential if biomedical and health informatics is to be an evidence-based discipline.
LanguageEnglish
Pages963-968
Number of pages6
JournalJournal of the American Medical Informatics Association : JAMIA
Volume25
Issue number8
DOIs
Publication statusPublished - Aug 2018

Fingerprint

Informatics
Health
Research
Medical Psychology
Costs and Cost Analysis

Bibliographical note

Copyright the Author(s) 2018. Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.

Keywords

  • evaluation
  • replication
  • evidence-based informatics
  • research bias

Cite this

@article{748ade3178b2479c81f6565dcb3abaaa,
title = "Does health informatics have a replication crisis?",
abstract = "Objective: Many research fields, including psychology and basic medical sciences, struggle with poor reproducibility of reported studies. Biomedical and health informatics is unlikely to be immune to these challenges. This paper explores replication in informatics and the unique challenges the discipline faces. Methods: Narrative review of recent literature on research replication challenges. Results: While there is growing interest in re-analysis of existing data, experimental replication studies appear uncommon in informatics. Context effects are a particular challenge as they make ensuring replication fidelity difficult, and the same intervention will never quite reproduce the same result in different settings. Replication studies take many forms, trading-off testing validity of past findings against testing generalizability. Exact and partial replication designs emphasize testing validity while quasi and conceptual studies test generalizability of an underlying model or hypothesis with different methods or in a different setting. Conclusions: The cost of poor replication is a weakening in the quality of published research and the evidence-based foundation of health informatics. The benefits of replication include increased rigor in research, and the development of evaluation methods that distinguish the impact of context and the nonreproducibility of research. Taking replication seriously is essential if biomedical and health informatics is to be an evidence-based discipline.",
keywords = "evaluation, replication, evidence-based informatics, research bias",
author = "Enrico Coiera and Elske Ammenwerth and Andrew Georgiou and Farah Magrabi",
note = "Copyright the Author(s) 2018. Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.",
year = "2018",
month = "8",
doi = "10.1093/jamia/ocy028",
language = "English",
volume = "25",
pages = "963--968",
journal = "Journal of the American Medical Informatics Association : JAMIA",
issn = "1067-5027",
publisher = "Oxford University Press",
number = "8",

}

Does health informatics have a replication crisis? / Coiera, Enrico; Ammenwerth, Elske; Georgiou, Andrew; Magrabi, Farah.

In: Journal of the American Medical Informatics Association : JAMIA, Vol. 25, No. 8, 08.2018, p. 963-968.

Research output: Contribution to journalArticleResearchpeer-review

TY - JOUR

T1 - Does health informatics have a replication crisis?

AU - Coiera, Enrico

AU - Ammenwerth, Elske

AU - Georgiou, Andrew

AU - Magrabi, Farah

N1 - Copyright the Author(s) 2018. Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.

PY - 2018/8

Y1 - 2018/8

N2 - Objective: Many research fields, including psychology and basic medical sciences, struggle with poor reproducibility of reported studies. Biomedical and health informatics is unlikely to be immune to these challenges. This paper explores replication in informatics and the unique challenges the discipline faces. Methods: Narrative review of recent literature on research replication challenges. Results: While there is growing interest in re-analysis of existing data, experimental replication studies appear uncommon in informatics. Context effects are a particular challenge as they make ensuring replication fidelity difficult, and the same intervention will never quite reproduce the same result in different settings. Replication studies take many forms, trading-off testing validity of past findings against testing generalizability. Exact and partial replication designs emphasize testing validity while quasi and conceptual studies test generalizability of an underlying model or hypothesis with different methods or in a different setting. Conclusions: The cost of poor replication is a weakening in the quality of published research and the evidence-based foundation of health informatics. The benefits of replication include increased rigor in research, and the development of evaluation methods that distinguish the impact of context and the nonreproducibility of research. Taking replication seriously is essential if biomedical and health informatics is to be an evidence-based discipline.

AB - Objective: Many research fields, including psychology and basic medical sciences, struggle with poor reproducibility of reported studies. Biomedical and health informatics is unlikely to be immune to these challenges. This paper explores replication in informatics and the unique challenges the discipline faces. Methods: Narrative review of recent literature on research replication challenges. Results: While there is growing interest in re-analysis of existing data, experimental replication studies appear uncommon in informatics. Context effects are a particular challenge as they make ensuring replication fidelity difficult, and the same intervention will never quite reproduce the same result in different settings. Replication studies take many forms, trading-off testing validity of past findings against testing generalizability. Exact and partial replication designs emphasize testing validity while quasi and conceptual studies test generalizability of an underlying model or hypothesis with different methods or in a different setting. Conclusions: The cost of poor replication is a weakening in the quality of published research and the evidence-based foundation of health informatics. The benefits of replication include increased rigor in research, and the development of evaluation methods that distinguish the impact of context and the nonreproducibility of research. Taking replication seriously is essential if biomedical and health informatics is to be an evidence-based discipline.

KW - evaluation

KW - replication

KW - evidence-based informatics

KW - research bias

UR - http://www.scopus.com/inward/record.url?scp=85055022836&partnerID=8YFLogxK

U2 - 10.1093/jamia/ocy028

DO - 10.1093/jamia/ocy028

M3 - Article

VL - 25

SP - 963

EP - 968

JO - Journal of the American Medical Informatics Association : JAMIA

T2 - Journal of the American Medical Informatics Association : JAMIA

JF - Journal of the American Medical Informatics Association : JAMIA

SN - 1067-5027

IS - 8

ER -