Evaluation framework to guide implementation of AI systems into healthcare settings

Sandeep Reddy*, Wendy Rogers, Ville-Petteri Makinen, Enrico Coiera, Pieta Brown, Markus Wenzel, Eva Weicken, Saba Ansari, Piyush Mathur, Aaron Casey, Blair Kelly

*Corresponding author for this work

Research output: Contribution to journalReview articlepeer-review

2 Downloads (Pure)

Abstract

Objectives: To date, many artificial intelligence (AI) systems have been developed in healthcare, but adoption has been limited. This may be due to inappropriate or incomplete evaluation and a lack of internationally recognised AI standards on evaluation. To have confidence in the generalisability of AI systems in healthcare and to enable their integration into workflows, there is a need for a practical yet comprehensive instrument to assess the translational aspects of the available AI systems. Currently available evaluation frameworks for AI in healthcare focus on the reporting and regulatory aspects but have little guidance regarding assessment of the translational aspects of the AI systems like the functional, utility and ethical components.

Methods: To address this gap and create a framework that assesses real-world systems, an international team has developed a translationally focused evaluation framework termed € Translational Evaluation of Healthcare AI (TEHAI)'. A critical review of literature assessed existing evaluation and reporting frameworks and gaps. Next, using health technology evaluation and translational principles, reporting components were identified for consideration. These were independently reviewed for consensus inclusion in a final framework by an international panel of eight expert.

Results: TEHAI includes three main components: capability, utility and adoption. The emphasis on translational and ethical features of the model development and deployment distinguishes TEHAI from other evaluation instruments. In specific, the evaluation components can be applied at any stage of the development and deployment of the AI system.

Discussion: One major limitation of existing reporting or evaluation frameworks is their narrow focus. TEHAI, because of its strong foundation in translation research models and an emphasis on safety, translational value and generalisability, not only has a theoretical basis but also practical application to assessing real-world systems.

Conclusion: The translational research theoretic approach used to develop TEHAI should see it having application not just for evaluation of clinical AI in research settings, but more broadly to guide evaluation of working clinical systems.

Original languageEnglish
Article numbere100444
Pages (from-to)1-7
Number of pages7
JournalBMJ Health and Care Informatics
Volume28
Issue number1
DOIs
Publication statusPublished - 12 Oct 2021

Bibliographical note

Copyright the Author(s) 2021. Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.

Keywords

  • artificial intelligence
  • data science
  • health services research
  • informatics
  • machine learning

Fingerprint

Dive into the research topics of 'Evaluation framework to guide implementation of AI systems into healthcare settings'. Together they form a unique fingerprint.

Cite this