Understanding and measuring user experience in conversational interfaces

Research output: Contribution to journalArticleResearchpeer-review

Abstract

Although various methods have been developed to evaluate conversational interfaces, there has been a lack of methods specifically focusing on evaluating user experience. This paper reviews the understandings of user experience (UX) in conversational interfaces literature and examines the six questionnaires commonly used for evaluating conversational systems in order to assess the potential suitability of these questionnaires to measure different UX dimensions in that context. The method to examine the questionnaires involved developing an assessment framework for main UX dimensions with relevant attributes and coding the items in the questionnaires according to the framework. The results show that (i) the understandings of UX notably differed in literature; (ii) four questionnaires included assessment items, in varying extents, to measure hedonic, aesthetic and pragmatic dimensions of UX; (iii) while the dimension of affect was covered by two questionnaires, playfulness, motivation, and frustration dimensions were covered by one questionnaire only. The largest coverage of UX dimensions has been provided by the Subjective Assessment of Speech System Interfaces (SASSI). We recommend using multiple questionnaires to obtain a more complete measurement of user experience or improve the assessment of a particular UX dimension.
LanguageEnglish
Pages192-207
Number of pages16
JournalInteracting with Computers
Volume31
Issue number2
Early online date13 May 2019
DOIs
Publication statusPublished - 2019

Keywords

  • Human computer interaction
  • user experience
  • Natural language interfaces
  • Interaction design
  • conversational agents

Cite this

@article{2b1465ea80344aa18e5883b72bf1f337,
title = "Understanding and measuring user experience in conversational interfaces",
abstract = "Although various methods have been developed to evaluate conversational interfaces, there has been a lack of methods specifically focusing on evaluating user experience. This paper reviews the understandings of user experience (UX) in conversational interfaces literature and examines the six questionnaires commonly used for evaluating conversational systems in order to assess the potential suitability of these questionnaires to measure different UX dimensions in that context. The method to examine the questionnaires involved developing an assessment framework for main UX dimensions with relevant attributes and coding the items in the questionnaires according to the framework. The results show that (i) the understandings of UX notably differed in literature; (ii) four questionnaires included assessment items, in varying extents, to measure hedonic, aesthetic and pragmatic dimensions of UX; (iii) while the dimension of affect was covered by two questionnaires, playfulness, motivation, and frustration dimensions were covered by one questionnaire only. The largest coverage of UX dimensions has been provided by the Subjective Assessment of Speech System Interfaces (SASSI). We recommend using multiple questionnaires to obtain a more complete measurement of user experience or improve the assessment of a particular UX dimension.",
keywords = "Human computer interaction, user experience, Natural language interfaces, Interaction design, conversational agents",
author = "Kocaballi, {Ahmet Baki} and Liliana Laranjo and Enrico Coiera",
year = "2019",
doi = "10.1093/iwc/iwz015",
language = "English",
volume = "31",
pages = "192--207",
journal = "Interacting with Computers",
issn = "0953-5438",
publisher = "Oxford University Press",
number = "2",

}

Understanding and measuring user experience in conversational interfaces. / Kocaballi, Ahmet Baki; Laranjo, Liliana; Coiera, Enrico.

In: Interacting with Computers, Vol. 31, No. 2, 2019, p. 192-207.

Research output: Contribution to journalArticleResearchpeer-review

TY - JOUR

T1 - Understanding and measuring user experience in conversational interfaces

AU - Kocaballi, Ahmet Baki

AU - Laranjo, Liliana

AU - Coiera, Enrico

PY - 2019

Y1 - 2019

N2 - Although various methods have been developed to evaluate conversational interfaces, there has been a lack of methods specifically focusing on evaluating user experience. This paper reviews the understandings of user experience (UX) in conversational interfaces literature and examines the six questionnaires commonly used for evaluating conversational systems in order to assess the potential suitability of these questionnaires to measure different UX dimensions in that context. The method to examine the questionnaires involved developing an assessment framework for main UX dimensions with relevant attributes and coding the items in the questionnaires according to the framework. The results show that (i) the understandings of UX notably differed in literature; (ii) four questionnaires included assessment items, in varying extents, to measure hedonic, aesthetic and pragmatic dimensions of UX; (iii) while the dimension of affect was covered by two questionnaires, playfulness, motivation, and frustration dimensions were covered by one questionnaire only. The largest coverage of UX dimensions has been provided by the Subjective Assessment of Speech System Interfaces (SASSI). We recommend using multiple questionnaires to obtain a more complete measurement of user experience or improve the assessment of a particular UX dimension.

AB - Although various methods have been developed to evaluate conversational interfaces, there has been a lack of methods specifically focusing on evaluating user experience. This paper reviews the understandings of user experience (UX) in conversational interfaces literature and examines the six questionnaires commonly used for evaluating conversational systems in order to assess the potential suitability of these questionnaires to measure different UX dimensions in that context. The method to examine the questionnaires involved developing an assessment framework for main UX dimensions with relevant attributes and coding the items in the questionnaires according to the framework. The results show that (i) the understandings of UX notably differed in literature; (ii) four questionnaires included assessment items, in varying extents, to measure hedonic, aesthetic and pragmatic dimensions of UX; (iii) while the dimension of affect was covered by two questionnaires, playfulness, motivation, and frustration dimensions were covered by one questionnaire only. The largest coverage of UX dimensions has been provided by the Subjective Assessment of Speech System Interfaces (SASSI). We recommend using multiple questionnaires to obtain a more complete measurement of user experience or improve the assessment of a particular UX dimension.

KW - Human computer interaction

KW - user experience

KW - Natural language interfaces

KW - Interaction design

KW - conversational agents

U2 - 10.1093/iwc/iwz015

DO - 10.1093/iwc/iwz015

M3 - Article

VL - 31

SP - 192

EP - 207

JO - Interacting with Computers

T2 - Interacting with Computers

JF - Interacting with Computers

SN - 0953-5438

IS - 2

ER -