TY - JOUR
T1 - Replication studies in the clinical decision support literature
T2 - frequency, fidelity, and impact
AU - Coiera, Enrico
AU - Tong, Huong Ly
PY - 2021/9/1
Y1 - 2021/9/1
N2 - Objective: To assess the frequency, fidelity, and impact of replication studies in the clinical decision support system (CDSS) literature. Materials and Methods: a PRISMa-compliant review identified CDSS replications across 28 health and biomedical informatics journals. Included articles were assessed for fidelity to the original study using 5 categories: Identical, Substitutable, In-class, augmented, and Out-of-class; and 7 IMPISCO domains: Investigators (I), Method (M), Population (P), Intervention (I), Setting (S), Comparator (C), and Outcome (O). a fidelity score and heat map were generated using the ratings. Results: From 4063 publications matching search criteria for CDSS research, only 12/4063 (0.3%) were ultimately identified as replications. Six articles replicated but could not reproduce the results of the Han et al (2005) CPOe study showing mortality increase and, over time, changed from truth testing to generalizing this result. Other replications successfully tested variants of CDSS technology (2/12) or validated measurement instruments (4/12). Discussion: a replication rate of 3 in a thousand studies is low even by the low rates in other disciplines. Several new reporting methods were developed for this study, including the IMPISCO framework, fidelity scores, and fidelity heat maps. a reporting structure for clearly identifying replication research is also proposed. Conclusion: There is an urgent need to better characterize which core CDSS principles require replication, identify past replication data, and conduct missing replication studies. attention to replication should improve the efficiency and effectiveness of CDSS research and avoiding potentially harmful trial and error technology deployment.
AB - Objective: To assess the frequency, fidelity, and impact of replication studies in the clinical decision support system (CDSS) literature. Materials and Methods: a PRISMa-compliant review identified CDSS replications across 28 health and biomedical informatics journals. Included articles were assessed for fidelity to the original study using 5 categories: Identical, Substitutable, In-class, augmented, and Out-of-class; and 7 IMPISCO domains: Investigators (I), Method (M), Population (P), Intervention (I), Setting (S), Comparator (C), and Outcome (O). a fidelity score and heat map were generated using the ratings. Results: From 4063 publications matching search criteria for CDSS research, only 12/4063 (0.3%) were ultimately identified as replications. Six articles replicated but could not reproduce the results of the Han et al (2005) CPOe study showing mortality increase and, over time, changed from truth testing to generalizing this result. Other replications successfully tested variants of CDSS technology (2/12) or validated measurement instruments (4/12). Discussion: a replication rate of 3 in a thousand studies is low even by the low rates in other disciplines. Several new reporting methods were developed for this study, including the IMPISCO framework, fidelity scores, and fidelity heat maps. a reporting structure for clearly identifying replication research is also proposed. Conclusion: There is an urgent need to better characterize which core CDSS principles require replication, identify past replication data, and conduct missing replication studies. attention to replication should improve the efficiency and effectiveness of CDSS research and avoiding potentially harmful trial and error technology deployment.
UR - http://www.scopus.com/inward/record.url?scp=85114161260&partnerID=8YFLogxK
U2 - 10.1093/jamia/ocab049
DO - 10.1093/jamia/ocab049
M3 - Article
C2 - 34226931
AN - SCOPUS:85114161260
SN - 1067-5027
VL - 28
SP - 1815
EP - 1825
JO - Journal of the American Medical Informatics Association : JAMIA
JF - Journal of the American Medical Informatics Association : JAMIA
IS - 9
ER -