Replication studies in the clinical decision support literature – frequency, fidelity and impact

Enrico Coiera, Huong Ly Tong

Research output: Contribution to journalArticlepeer-review

Abstract

Objective: To assess the frequency, fidelity and impact of replication studies in the clinical decision support literature (CDSS).

Materials and Methods: A PRISMA compliant review identified CDSS replications across 28 health and biomedical informatics journals. Included papers were assessed for fidelity to the original study using five categories: Identical, Substitutable, In-class, Augmented and Out-of-class, and seven IMPISCO domains: Investigators (I), Method (M), Population (P), Intervention (I), Setting (S), Comparator (C), Outcome (O). A fidelity score and heatmap were generated using the ratings.

Results: From 4063 publications matching search criteria for CDSS research, 404 (9.9%) matched replication search criteria. Only 12/4063 (0.3%) were ultimately identified as replications. Six papers replicated but could not reproduce the results of the Han et al. (2005) CPOE study showing mortality increase, and over time changed from truth testing to generalizing this result. Other replications successfully tested variants of CDSS technology (2/12) or validated measurement instruments (4/12).

Discussion: A replication rate of 3 in a thousand studies is low even by the low rates in other disciplines. Several new reporting methods were developed for this study including the IMPISCO framework, fidelity scores and fidelity heatmaps. A reporting structure for clearly identifying replication research is also proposed.

Conclusion: There is an urgent need to better characterise which core CDSS principles require replication, identify past replication data and conduct missing replication studies. Attention to replication should improve the efficiency and effectiveness of CDSS research and avoiding potentially harmful trial and error technology deployment.
Original languageEnglish
JournalJournal of the American Medical Informatics Association : JAMIA
Publication statusAccepted/In press - 28 Feb 2021

Fingerprint Dive into the research topics of 'Replication studies in the clinical decision support literature – frequency, fidelity and impact'. Together they form a unique fingerprint.

Cite this