Conversational agents in healthcare: a systematic review

Research output: Contribution to journalArticleResearchpeer-review

Abstract

Objective: Our objective was to review the characteristics, current applications, and evaluation measures of conversational agents with unconstrained natural language input capabilities used for health-related purposes.

Methods: We searched PubMed, Embase, CINAHL, PsycInfo, and ACM Digital using a predefined search strategy. Studies were included if they focused on consumers or healthcare professionals; involved a conversational agent using any unconstrained natural language input; and reported evaluation measures resulting from user interaction with the system. Studies were screened by independent reviewers and Cohen’s kappa measured inter-coder agreement.

Results: The database search retrieved 1513 citations; 17 articles (14 different conversational agents) met the inclusion criteria. Dialogue management strategies were mostly finite-state and frame-based (6 and 7 conversational agents, respectively); agent-based strategies were present in one type of system. Two studies were randomized controlled trials (RCTs), 1 was cross-sectional, and the remaining were quasi-experimental. Half of the conversational agents supported consumers with health tasks such as self-care. The only RCT evaluating the efficacy of a conversational agent found a significant effect in reducing depression symptoms (effect size d = 0.44, p = .04). Patient safety was rarely evaluated in the included studies.

Conclusions: The use of conversational agents with unconstrained natural language input capabilities for health-related purposes is an emerging field of research, where the few published studies were mainly quasi-experimental, and rarely evaluated efficacy or safety. Future studies would benefit from more robust experimental designs and standardized reporting.

Protocol Registration: The protocol for this systematic review is registered at PROSPERO with the number CRD42017065917.
LanguageEnglish
Pages1248–1258
Number of pages11
JournalJournal of the American Medical Informatics Association
Volume25
Issue number9
Early online date13 Jul 2018
DOIs
Publication statusPublished - 1 Sep 2018

Fingerprint

Language
Delivery of Health Care
Health
Randomized Controlled Trials
Patient Safety
Self Care
PubMed
Research Design
Databases
Depression
Safety
Research

Bibliographical note

Copyright the Author(s) 2018. Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.

Keywords

  • artificial intelligence [Mesh]
  • medical informatics [Mesh]
  • conversational agent
  • dialogue system

Cite this

@article{241aa62a12e54ceea9fe01363267cb4b,
title = "Conversational agents in healthcare: a systematic review",
abstract = "Objective: Our objective was to review the characteristics, current applications, and evaluation measures of conversational agents with unconstrained natural language input capabilities used for health-related purposes.Methods: We searched PubMed, Embase, CINAHL, PsycInfo, and ACM Digital using a predefined search strategy. Studies were included if they focused on consumers or healthcare professionals; involved a conversational agent using any unconstrained natural language input; and reported evaluation measures resulting from user interaction with the system. Studies were screened by independent reviewers and Cohen’s kappa measured inter-coder agreement.Results: The database search retrieved 1513 citations; 17 articles (14 different conversational agents) met the inclusion criteria. Dialogue management strategies were mostly finite-state and frame-based (6 and 7 conversational agents, respectively); agent-based strategies were present in one type of system. Two studies were randomized controlled trials (RCTs), 1 was cross-sectional, and the remaining were quasi-experimental. Half of the conversational agents supported consumers with health tasks such as self-care. The only RCT evaluating the efficacy of a conversational agent found a significant effect in reducing depression symptoms (effect size d = 0.44, p = .04). Patient safety was rarely evaluated in the included studies.Conclusions: The use of conversational agents with unconstrained natural language input capabilities for health-related purposes is an emerging field of research, where the few published studies were mainly quasi-experimental, and rarely evaluated efficacy or safety. Future studies would benefit from more robust experimental designs and standardized reporting.Protocol Registration: The protocol for this systematic review is registered at PROSPERO with the number CRD42017065917.",
keywords = "artificial intelligence [Mesh], medical informatics [Mesh], conversational agent, dialogue system",
author = "{Laranjo da Silva}, Liliana and Dunn, {Adam G.} and Tong, {Huong Ly} and Kocaballi, {Ahmet Baki} and Jessica Chen and Rabia Bashir and Didi Surian and Blanca Gallego and Farah Magrabi and Lau, {Annie Y. S.} and Enrico Coiera",
note = "Copyright the Author(s) 2018. Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.",
year = "2018",
month = "9",
day = "1",
doi = "10.1093/jamia/ocy072",
language = "English",
volume = "25",
pages = "1248–1258",
journal = "Journal of the American Medical Informatics Association : JAMIA",
issn = "1067-5027",
publisher = "Oxford University Press",
number = "9",

}

TY - JOUR

T1 - Conversational agents in healthcare

T2 - Journal of the American Medical Informatics Association : JAMIA

AU - Laranjo da Silva, Liliana

AU - Dunn, Adam G.

AU - Tong, Huong Ly

AU - Kocaballi, Ahmet Baki

AU - Chen, Jessica

AU - Bashir, Rabia

AU - Surian, Didi

AU - Gallego, Blanca

AU - Magrabi, Farah

AU - Lau, Annie Y. S.

AU - Coiera, Enrico

N1 - Copyright the Author(s) 2018. Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.

PY - 2018/9/1

Y1 - 2018/9/1

N2 - Objective: Our objective was to review the characteristics, current applications, and evaluation measures of conversational agents with unconstrained natural language input capabilities used for health-related purposes.Methods: We searched PubMed, Embase, CINAHL, PsycInfo, and ACM Digital using a predefined search strategy. Studies were included if they focused on consumers or healthcare professionals; involved a conversational agent using any unconstrained natural language input; and reported evaluation measures resulting from user interaction with the system. Studies were screened by independent reviewers and Cohen’s kappa measured inter-coder agreement.Results: The database search retrieved 1513 citations; 17 articles (14 different conversational agents) met the inclusion criteria. Dialogue management strategies were mostly finite-state and frame-based (6 and 7 conversational agents, respectively); agent-based strategies were present in one type of system. Two studies were randomized controlled trials (RCTs), 1 was cross-sectional, and the remaining were quasi-experimental. Half of the conversational agents supported consumers with health tasks such as self-care. The only RCT evaluating the efficacy of a conversational agent found a significant effect in reducing depression symptoms (effect size d = 0.44, p = .04). Patient safety was rarely evaluated in the included studies.Conclusions: The use of conversational agents with unconstrained natural language input capabilities for health-related purposes is an emerging field of research, where the few published studies were mainly quasi-experimental, and rarely evaluated efficacy or safety. Future studies would benefit from more robust experimental designs and standardized reporting.Protocol Registration: The protocol for this systematic review is registered at PROSPERO with the number CRD42017065917.

AB - Objective: Our objective was to review the characteristics, current applications, and evaluation measures of conversational agents with unconstrained natural language input capabilities used for health-related purposes.Methods: We searched PubMed, Embase, CINAHL, PsycInfo, and ACM Digital using a predefined search strategy. Studies were included if they focused on consumers or healthcare professionals; involved a conversational agent using any unconstrained natural language input; and reported evaluation measures resulting from user interaction with the system. Studies were screened by independent reviewers and Cohen’s kappa measured inter-coder agreement.Results: The database search retrieved 1513 citations; 17 articles (14 different conversational agents) met the inclusion criteria. Dialogue management strategies were mostly finite-state and frame-based (6 and 7 conversational agents, respectively); agent-based strategies were present in one type of system. Two studies were randomized controlled trials (RCTs), 1 was cross-sectional, and the remaining were quasi-experimental. Half of the conversational agents supported consumers with health tasks such as self-care. The only RCT evaluating the efficacy of a conversational agent found a significant effect in reducing depression symptoms (effect size d = 0.44, p = .04). Patient safety was rarely evaluated in the included studies.Conclusions: The use of conversational agents with unconstrained natural language input capabilities for health-related purposes is an emerging field of research, where the few published studies were mainly quasi-experimental, and rarely evaluated efficacy or safety. Future studies would benefit from more robust experimental designs and standardized reporting.Protocol Registration: The protocol for this systematic review is registered at PROSPERO with the number CRD42017065917.

KW - artificial intelligence [Mesh]

KW - medical informatics [Mesh]

KW - conversational agent

KW - dialogue system

UR - http://purl.org/au-research/grants/nhmrc/1134919

UR - http://purl.org/au-research/grants/nhmrc/1054146

UR - http://www.scopus.com/inward/record.url?scp=85055086455&partnerID=8YFLogxK

U2 - 10.1093/jamia/ocy072

DO - 10.1093/jamia/ocy072

M3 - Article

VL - 25

SP - 1248

EP - 1258

JO - Journal of the American Medical Informatics Association : JAMIA

JF - Journal of the American Medical Informatics Association : JAMIA

SN - 1067-5027

IS - 9

ER -