What are we measuring anyway?

A literature survey of questionnaires used in studies reported in the intelligent virtual agent conferences

Merijn Bruijnes, Siska Fitrianie, Deborah Richards, Amal Abdulrahman, Willem Paul Brinkman

Research output: Chapter in Book/Report/Conference proceedingConference abstract

1 Downloads (Pure)

Abstract

Research into artificial social agents aims at constructing these agents and at establishing an empirically grounded understanding of them, their interaction with humans, and how they can ultimately deliver certain outcomes in areas such as health, entertainment, and education. Key for establishing such understanding is the community’s ability to describe and replicate their observations on how users perceive and interact with their agents. In this paper, we address this ability by examining questionnaires and their constructs used in empirical studies reported in the intelligent virtual agent conference proceedings from 2013 to 2018. The literature survey shows the identification of 189 constructs used in 89 questionnaires that were reported across 81 papers. We found unexpectedly little repeated use of questionnaires as the vast majority of questionnaires (more than 76%) were only reported in a single paper. We expect that this finding will motivate joint effort by the IVA community towards creating a unified measurement instrument and in the broader AI community a renewed interest in replicability of our (user) studies.

Original languageEnglish
Title of host publicationBNAIC/BENELEARN 2019: Proceedings of the 31st Benelux Conference on Artificial Intelligence (BNAIC 2019) and the 28th Belgian Dutch Conference on Machine Learning (Benelearn 2019)
EditorsKatrien Beuls, Bart Bogaerts, Gianluca Bontempi, Pierre Geurts, Nick Harley, Bertrand Lebichot, Tom Lenaerts, Gilles Louppe, Paul Van Eecke
Place of PublicationAachen
PublisherRWTH Aachen University
Number of pages2
Publication statusPublished - 2019
Event31st Benelux Conference on Artificial Intelligence and the 28th Belgian Dutch Conference on Machine Learning, BNAIC/BENELEARN 2019 - Brussels, Belgium
Duration: 6 Nov 20198 Nov 2019

Publication series

NameCEUR Workshop Proceedings
PublisherRWTH Aachen University
Volume2491
ISSN (Print)1613-0073

Conference

Conference31st Benelux Conference on Artificial Intelligence and the 28th Belgian Dutch Conference on Machine Learning, BNAIC/BENELEARN 2019
CountryBelgium
CityBrussels
Period6/11/198/11/19

Bibliographical note

Copyright the Author(s) 2019. Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.

Fingerprint Dive into the research topics of 'What are we measuring anyway? A literature survey of questionnaires used in studies reported in the intelligent virtual agent conferences'. Together they form a unique fingerprint.

Cite this