Evaluating the efficiency and safety of speech recognition within a commercial electronic health record system: a replication study

Tobias Hodgson, Farah Magrabi, Enrico Coiera

Research output: Contribution to journalArticleResearchpeer-review

Abstract

Objective: To conduct a replication study to validate previously identified significant risks and inefficiencies associated with the use of speech recognition (SR) for documentation within an electronic health record (EHR) system. Methods: Thirty-five emergency department clinicians undertook randomly allocated clinical documentation tasks using keyboard and mouse (KBM) or SR using a commercial EHR system. The experiment design, setting, and tasks (E2) replicated an earlier study (E1), while technical integration issues that may have led to poorer SR performance were addressed. Results: Complex tasks were significantly slower to complete using SR (16.94%) than KBM (KBM: 191.9 s, SR: 224.4 s; p = 0.009; CI, 11.9-48.3), replicating task completion times observed in the earlier experiment. Errors (non-typographical) were significantly higher with SR compared with KBM for both simple (KBM: 3, SR: 84; p < 0.001; CI, 1.5- 2.5) and complex tasks (KBM: 23, SR: 53; p = 0.001; CI, 0.5-1.0), again replicating earlier results (E1: 170, E2: 163; p = 0.660; CI, 0.0-0.0). Typographical errors were reduced significantly in the new study (E1: 465, E2: 150; p < 0.001; CI, 2.0-3.0). Discussion: The results of this study replicate those reported earlier. The use of SR for clinical documentation within an EHR system appears to be consistently associated with decreased time efficiencies and increased errors. Modifications implemented to optimize SR integration in the EHR seem to have resulted in minor improvements that did not fundamentally change overall results. Conclusion: This replication study adds further evidence for the poor performance of SR-assisted clinical documentation within an EHR. Replication studies remain rare in informatics literature, especially where study results are unexpected or have significant implication; such studies are clearly needed to avoid overdependence on the results of a single study.

LanguageEnglish
Pages326-335
Number of pages10
JournalApplied clinical informatics
Volume9
Issue number2
DOIs
Publication statusPublished - 1 Apr 2018

Fingerprint

Electronic Health Records
Speech recognition
Health
Safety
Documentation
Informatics
Hospital Emergency Service
Experiments

Keywords

  • Electronic health record
  • Integration
  • Medical errors
  • Patient safety
  • Speech recognition

Cite this

@article{db56cd14a21a47b18d873befa3d24f48,
title = "Evaluating the efficiency and safety of speech recognition within a commercial electronic health record system: a replication study",
abstract = "Objective: To conduct a replication study to validate previously identified significant risks and inefficiencies associated with the use of speech recognition (SR) for documentation within an electronic health record (EHR) system. Methods: Thirty-five emergency department clinicians undertook randomly allocated clinical documentation tasks using keyboard and mouse (KBM) or SR using a commercial EHR system. The experiment design, setting, and tasks (E2) replicated an earlier study (E1), while technical integration issues that may have led to poorer SR performance were addressed. Results: Complex tasks were significantly slower to complete using SR (16.94{\%}) than KBM (KBM: 191.9 s, SR: 224.4 s; p = 0.009; CI, 11.9-48.3), replicating task completion times observed in the earlier experiment. Errors (non-typographical) were significantly higher with SR compared with KBM for both simple (KBM: 3, SR: 84; p < 0.001; CI, 1.5- 2.5) and complex tasks (KBM: 23, SR: 53; p = 0.001; CI, 0.5-1.0), again replicating earlier results (E1: 170, E2: 163; p = 0.660; CI, 0.0-0.0). Typographical errors were reduced significantly in the new study (E1: 465, E2: 150; p < 0.001; CI, 2.0-3.0). Discussion: The results of this study replicate those reported earlier. The use of SR for clinical documentation within an EHR system appears to be consistently associated with decreased time efficiencies and increased errors. Modifications implemented to optimize SR integration in the EHR seem to have resulted in minor improvements that did not fundamentally change overall results. Conclusion: This replication study adds further evidence for the poor performance of SR-assisted clinical documentation within an EHR. Replication studies remain rare in informatics literature, especially where study results are unexpected or have significant implication; such studies are clearly needed to avoid overdependence on the results of a single study.",
keywords = "Electronic health record, Integration, Medical errors, Patient safety, Speech recognition",
author = "Tobias Hodgson and Farah Magrabi and Enrico Coiera",
year = "2018",
month = "4",
day = "1",
doi = "10.1055/s-0038-1649509",
language = "English",
volume = "9",
pages = "326--335",
journal = "Applied clinical informatics",
issn = "1869-0327",
publisher = "Schattauer GmbH",
number = "2",

}

Evaluating the efficiency and safety of speech recognition within a commercial electronic health record system : a replication study. / Hodgson, Tobias; Magrabi, Farah; Coiera, Enrico.

In: Applied clinical informatics, Vol. 9, No. 2, 01.04.2018, p. 326-335.

Research output: Contribution to journalArticleResearchpeer-review

TY - JOUR

T1 - Evaluating the efficiency and safety of speech recognition within a commercial electronic health record system

T2 - Applied clinical informatics

AU - Hodgson, Tobias

AU - Magrabi, Farah

AU - Coiera, Enrico

PY - 2018/4/1

Y1 - 2018/4/1

N2 - Objective: To conduct a replication study to validate previously identified significant risks and inefficiencies associated with the use of speech recognition (SR) for documentation within an electronic health record (EHR) system. Methods: Thirty-five emergency department clinicians undertook randomly allocated clinical documentation tasks using keyboard and mouse (KBM) or SR using a commercial EHR system. The experiment design, setting, and tasks (E2) replicated an earlier study (E1), while technical integration issues that may have led to poorer SR performance were addressed. Results: Complex tasks were significantly slower to complete using SR (16.94%) than KBM (KBM: 191.9 s, SR: 224.4 s; p = 0.009; CI, 11.9-48.3), replicating task completion times observed in the earlier experiment. Errors (non-typographical) were significantly higher with SR compared with KBM for both simple (KBM: 3, SR: 84; p < 0.001; CI, 1.5- 2.5) and complex tasks (KBM: 23, SR: 53; p = 0.001; CI, 0.5-1.0), again replicating earlier results (E1: 170, E2: 163; p = 0.660; CI, 0.0-0.0). Typographical errors were reduced significantly in the new study (E1: 465, E2: 150; p < 0.001; CI, 2.0-3.0). Discussion: The results of this study replicate those reported earlier. The use of SR for clinical documentation within an EHR system appears to be consistently associated with decreased time efficiencies and increased errors. Modifications implemented to optimize SR integration in the EHR seem to have resulted in minor improvements that did not fundamentally change overall results. Conclusion: This replication study adds further evidence for the poor performance of SR-assisted clinical documentation within an EHR. Replication studies remain rare in informatics literature, especially where study results are unexpected or have significant implication; such studies are clearly needed to avoid overdependence on the results of a single study.

AB - Objective: To conduct a replication study to validate previously identified significant risks and inefficiencies associated with the use of speech recognition (SR) for documentation within an electronic health record (EHR) system. Methods: Thirty-five emergency department clinicians undertook randomly allocated clinical documentation tasks using keyboard and mouse (KBM) or SR using a commercial EHR system. The experiment design, setting, and tasks (E2) replicated an earlier study (E1), while technical integration issues that may have led to poorer SR performance were addressed. Results: Complex tasks were significantly slower to complete using SR (16.94%) than KBM (KBM: 191.9 s, SR: 224.4 s; p = 0.009; CI, 11.9-48.3), replicating task completion times observed in the earlier experiment. Errors (non-typographical) were significantly higher with SR compared with KBM for both simple (KBM: 3, SR: 84; p < 0.001; CI, 1.5- 2.5) and complex tasks (KBM: 23, SR: 53; p = 0.001; CI, 0.5-1.0), again replicating earlier results (E1: 170, E2: 163; p = 0.660; CI, 0.0-0.0). Typographical errors were reduced significantly in the new study (E1: 465, E2: 150; p < 0.001; CI, 2.0-3.0). Discussion: The results of this study replicate those reported earlier. The use of SR for clinical documentation within an EHR system appears to be consistently associated with decreased time efficiencies and increased errors. Modifications implemented to optimize SR integration in the EHR seem to have resulted in minor improvements that did not fundamentally change overall results. Conclusion: This replication study adds further evidence for the poor performance of SR-assisted clinical documentation within an EHR. Replication studies remain rare in informatics literature, especially where study results are unexpected or have significant implication; such studies are clearly needed to avoid overdependence on the results of a single study.

KW - Electronic health record

KW - Integration

KW - Medical errors

KW - Patient safety

KW - Speech recognition

UR - http://www.scopus.com/inward/record.url?scp=85054421414&partnerID=8YFLogxK

UR - http://purl.org/au-research/grants/nhmrc/1032664

U2 - 10.1055/s-0038-1649509

DO - 10.1055/s-0038-1649509

M3 - Article

VL - 9

SP - 326

EP - 335

JO - Applied clinical informatics

JF - Applied clinical informatics

SN - 1869-0327

IS - 2

ER -