Machine learning in clinical practice: evaluation of an artificial intelligence tool after implementation

Hamed Akhlaghi, Sam Freeman, Cynthia Vari, Bede McKenna, George Braitberg, Jonathan Karro, Bahman Tahayori

Research output: Contribution to journalArticlepeer-review

1 Citation (Scopus)
31 Downloads (Pure)


Objective: Artificial intelligence (AI) has gradually found its way into healthcare, and its future integration into clinical practice is inevitable. In the present study, we evaluate the accuracy of a novel AI algorithm designed to predict admission based on a triage note after clinical implementation. This is the first of such studies to investigate real‐time AI performance in the emergency setting. Methods: The novel AI algorithm that predicts admission using a triage note was translated into clinical practice and integrated within St Vincent's Hospital Melbourne's electronic emergency patient management system. The data were collected from 1 January 2021 to 17 August 2022 to evaluate the diagnostic accuracy of the AI system after implementation. Results: A total of 77 125 ED presentations were included. The live AI algorithm has a sensitivity of 73.1% (95% confidence interval 72.5–73.8), specificity of 74.3% (73.9–74.7), positive predictive value of 50% (49.6–50.4) and negative predictive value of 88.7% (88.5–89) with a total accuracy of 74% (73.7–74.3). The accuracy of the system was at the lowest for admission to psychiatric units (34%) and at the highest for gastroenterology and medical admission (84% and 80%, respectively). Conclusion: Our study showed the diagnostic evaluation of a real‐time AI clinical decision‐support tool became less accurate than the original. Although real‐time sensitivity and specificity of the AI tool was still acceptable as a decision‐support tool in the ED, we propose that continuous training and evaluation of AI‐enabled clinical support tools in healthcare are conducted to ensure consistent accuracy and performance to prevent inadvertent consequences.
Original languageEnglish
Pages (from-to)118-124
Number of pages7
JournalEMA - Emergency Medicine Australasia
Issue number1
Early online date28 Sept 2023
Publication statusPublished - Feb 2024
Externally publishedYes

Bibliographical note

Copyright the Author(s) 2023. Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.


  • artificial intelligence
  • emergency department
  • machine learning
  • research translation
  • triage note


Dive into the research topics of 'Machine learning in clinical practice: evaluation of an artificial intelligence tool after implementation'. Together they form a unique fingerprint.

Cite this