A review of measurement practice in studies of clinical decision support systems 1998-2017

Philip J. Scott, Angela W. Brown, Taiwo Adedeji, Jeremy C. Wyatt, Andrew Georgiou, Eric L. Eisenstein, Charles P. Friedman

Research output: Contribution to journalReview articlepeer-review

13 Citations (Scopus)
30 Downloads (Pure)


Objective: To assess measurement practice in clinical decision support evaluation studies. Materials and methods: We identified empirical studies evaluating clinical decision support systems published from 1998 to 2017. We reviewed titles, abstracts, and full paper contents for evidence of attention to measurement validity, reliability, or reuse. We used Friedman and Wyatt's typology to categorize the studies. Results: There were 391 studies that met the inclusion criteria. Study types in this cohort were primarily field user effect studies (n = 210) or problem impact studies (n = 150). Of those, 280 studies (72%) had no evidence of attention to measurement methodology, and 111 (28%) had some evidence with 33 (8%) offering validity evidence; 45 (12%) offering reliability evidence; and 61 (16%) reporting measurement artefact reuse. Discussion: Only 5 studies offered validity assessment within the study. Valid measures were predominantly observed in problem impact studies with the majority of measures being clinical or patient reported outcomes with validity measured elsewhere. Conclusion: Measurement methodology is frequently ignored in empirical studies of clinical decision support systems and particularly so in field user effect studies. Authors may in fact be attending to measurement considerations and not reporting this or employing methods of unknown validity and reliability in their studies. In the latter case, reported study results may be biased and effect sizes misleading. We argue that replication studies to strengthen the evidence base require greater attention to measurement practice in health informatics research.

Original languageEnglish
Pages (from-to)1120-1128
Number of pages9
JournalJournal of the American Medical Informatics Association : JAMIA
Issue number10
Publication statusPublished - 1 Oct 2019

Bibliographical note

Copyright the Author(s) 2019. Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.


  • clinical decision support systems
  • health informatics
  • measurement
  • reliability
  • validity


Dive into the research topics of 'A review of measurement practice in studies of clinical decision support systems 1998-2017'. Together they form a unique fingerprint.

Cite this