Measuring user experience in conversational interfaces: a comparison of six questionnaires

A. Baki Kocaballi; Liliana Laranjo da Silva; Enrico Coiera

doi:10.14236/ewic/HCI2018.21

Measuring user experience in conversational interfaces

a comparison of six questionnaires

A. Baki Kocaballi, Liliana Laranjo da Silva, Enrico Coiera

Research output: Chapter in Book/Report/Conference proceeding › Conference proceeding contribution

6 Citations (Scopus)

Abstract

User experience (UX) has become an important aspect in the evaluation of interactive systems. In parallel, conversational interfaces have been increasingly used in many work and everyday settings. Although there have been various methods developed to evaluate conversational interfaces, there has been a lack of methods specifically focusing on evaluating user experience. This study reviews the six main questionnaires for evaluating conversational systems in order to assess the potential suitability of these questionnaires to measure various UX dimensions. We found that (i) four questionnaires included assessment items, in varying extents, to measure hedonic, aesthetic and pragmatic dimensions of UX; (ii) two questionnaires assessed affect, and one assessed frustration dimension; and, (iii) enchantment, playfulness and motivation dimensions have not been covered sufficiently by any questionnaires. We recommend using multiple questionnaires to obtain a more complete measurement of user experience or improve the assessment of a particular UX dimension.

Original language	English
Title of host publication	Proceedings of 32nd BCS Human Computer Interaction Conference (HCI 2018)
Editors	Raymond Bond, Maurice Mulvenna, Jonathan Wallace, Michaela Black
Place of Publication	Belfast
Publisher	BCS, The Chartered Institute for IT
Pages	1-12
Number of pages	12
DOIs	https://doi.org/10.14236/ewic/HCI2018.21
Publication status	Published - Nov 2018
Event	32nd International BCS Human Computer Interaction Conference (HCI 2018) - Belfast, Ireland Duration: 4 Jul 2018 → 6 Jul 2018

Conference

Conference	32nd International BCS Human Computer Interaction Conference (HCI 2018)
Country	Ireland
City	Belfast
Period	4/07/18 → 6/07/18

Keywords

User Experience
conversational agents
dialogue systems
evaluation
voice interfaces
standardised questionnaires

Cite this

Kocaballi, A. B., Laranjo da Silva, L., & Coiera, E. (2018). Measuring user experience in conversational interfaces: a comparison of six questionnaires. In R. Bond, M. Mulvenna, J. Wallace, & M. Black (Eds.), Proceedings of 32nd BCS Human Computer Interaction Conference (HCI 2018) (pp. 1-12). Belfast: BCS, The Chartered Institute for IT. https://doi.org/10.14236/ewic/HCI2018.21

Kocaballi, A. Baki ; Laranjo da Silva, Liliana ; Coiera, Enrico. / Measuring user experience in conversational interfaces : a comparison of six questionnaires. Proceedings of 32nd BCS Human Computer Interaction Conference (HCI 2018). editor / Raymond Bond ; Maurice Mulvenna ; Jonathan Wallace ; Michaela Black. Belfast : BCS, The Chartered Institute for IT, 2018. pp. 1-12

@inproceedings{43f4c43df9a743e08c4f45e81eb9f04e,

title = "Measuring user experience in conversational interfaces: a comparison of six questionnaires",

abstract = "User experience (UX) has become an important aspect in the evaluation of interactive systems. In parallel, conversational interfaces have been increasingly used in many work and everyday settings. Although there have been various methods developed to evaluate conversational interfaces, there has been a lack of methods specifically focusing on evaluating user experience. This study reviews the six main questionnaires for evaluating conversational systems in order to assess the potential suitability of these questionnaires to measure various UX dimensions. We found that (i) four questionnaires included assessment items, in varying extents, to measure hedonic, aesthetic and pragmatic dimensions of UX; (ii) two questionnaires assessed affect, and one assessed frustration dimension; and, (iii) enchantment, playfulness and motivation dimensions have not been covered sufficiently by any questionnaires. We recommend using multiple questionnaires to obtain a more complete measurement of user experience or improve the assessment of a particular UX dimension.",

keywords = "User Experience, conversational agents, dialogue systems, evaluation, voice interfaces, standardised questionnaires",

author = "Kocaballi, {A. Baki} and {Laranjo da Silva}, Liliana and Enrico Coiera",

year = "2018",

month = "11",

doi = "10.14236/ewic/HCI2018.21",

language = "English",

pages = "1--12",

editor = "Raymond Bond and Maurice Mulvenna and Jonathan Wallace and Michaela Black",

booktitle = "Proceedings of 32nd BCS Human Computer Interaction Conference (HCI 2018)",

publisher = "BCS, The Chartered Institute for IT",

address = "United Kingdom",

}

Kocaballi, AB , Laranjo da Silva, L & Coiera, E 2018, Measuring user experience in conversational interfaces: a comparison of six questionnaires. in R Bond, M Mulvenna, J Wallace & M Black (eds), Proceedings of 32nd BCS Human Computer Interaction Conference (HCI 2018). BCS, The Chartered Institute for IT, Belfast, pp. 1-12, 32nd International BCS Human Computer Interaction Conference (HCI 2018), Belfast, Ireland, 4/07/18. https://doi.org/10.14236/ewic/HCI2018.21

Measuring user experience in conversational interfaces : a comparison of six questionnaires. / Kocaballi, A. Baki ; Laranjo da Silva, Liliana ; Coiera, Enrico.

Proceedings of 32nd BCS Human Computer Interaction Conference (HCI 2018). ed. / Raymond Bond; Maurice Mulvenna; Jonathan Wallace; Michaela Black. Belfast : BCS, The Chartered Institute for IT, 2018. p. 1-12.

Research output: Chapter in Book/Report/Conference proceeding › Conference proceeding contribution

TY - GEN

T1 - Measuring user experience in conversational interfaces

T2 - a comparison of six questionnaires

AU - Kocaballi, A. Baki

AU - Laranjo da Silva, Liliana

AU - Coiera, Enrico

PY - 2018/11

Y1 - 2018/11

N2 - User experience (UX) has become an important aspect in the evaluation of interactive systems. In parallel, conversational interfaces have been increasingly used in many work and everyday settings. Although there have been various methods developed to evaluate conversational interfaces, there has been a lack of methods specifically focusing on evaluating user experience. This study reviews the six main questionnaires for evaluating conversational systems in order to assess the potential suitability of these questionnaires to measure various UX dimensions. We found that (i) four questionnaires included assessment items, in varying extents, to measure hedonic, aesthetic and pragmatic dimensions of UX; (ii) two questionnaires assessed affect, and one assessed frustration dimension; and, (iii) enchantment, playfulness and motivation dimensions have not been covered sufficiently by any questionnaires. We recommend using multiple questionnaires to obtain a more complete measurement of user experience or improve the assessment of a particular UX dimension.

AB - User experience (UX) has become an important aspect in the evaluation of interactive systems. In parallel, conversational interfaces have been increasingly used in many work and everyday settings. Although there have been various methods developed to evaluate conversational interfaces, there has been a lack of methods specifically focusing on evaluating user experience. This study reviews the six main questionnaires for evaluating conversational systems in order to assess the potential suitability of these questionnaires to measure various UX dimensions. We found that (i) four questionnaires included assessment items, in varying extents, to measure hedonic, aesthetic and pragmatic dimensions of UX; (ii) two questionnaires assessed affect, and one assessed frustration dimension; and, (iii) enchantment, playfulness and motivation dimensions have not been covered sufficiently by any questionnaires. We recommend using multiple questionnaires to obtain a more complete measurement of user experience or improve the assessment of a particular UX dimension.

KW - User Experience

KW - conversational agents

KW - dialogue systems

KW - evaluation

KW - voice interfaces

KW - standardised questionnaires

UR - http://hci2018.bcs.org/prelim_proceedings/

UR - http://www.scopus.com/inward/record.url?scp=85058339544&partnerID=8YFLogxK

U2 - 10.14236/ewic/HCI2018.21

DO - 10.14236/ewic/HCI2018.21

M3 - Conference proceeding contribution

SP - 1

EP - 12

BT - Proceedings of 32nd BCS Human Computer Interaction Conference (HCI 2018)

A2 - Bond, Raymond

A2 - Mulvenna, Maurice

A2 - Wallace, Jonathan

A2 - Black, Michaela

PB - BCS, The Chartered Institute for IT

CY - Belfast

ER -

Access to Document

10.14236/ewic/HCI2018.21

https://ewic.bcs.org/category/19364Licence: Unspecified
Link to Preliminary Proceedings
Link to publication in Scopus