A transformer-based user satisfaction prediction for proactive interaction mechanism in DuerOS

Wei Shen, Xiaonan He*, Chuheng Zhang, Xuyun Zhang, Jian Xie

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference proceeding contributionpeer-review


Recently, spoken dialogue systems have been widely deployed in a variety of applications, serving a huge number of end-users. A common issue is that the errors resulting from noisy utterances, semantic misunderstandings, or lack of knowledge make it hard for a real system to respond properly, possibly leading to an unsatisfactory user experience. To avoid such a case, we consider a proactive interaction mechanism where the system predicts the user satisfaction with the candidate response before giving it to the user. If the user is not likely to be satisfied according to the prediction, the system will ask the user a suitable question to determine the real intent of the user instead of providing the response directly. With such an interaction with the user, the system can give a better response to the user. Previous models that predict the user satisfaction are not applicable to DuerOS which is a large-scale commercial dialogue system. They are based on hand-crafted features and thus can hardly learn the complex patterns lying behind millions of conversations and temporal dependency in multiple turns of the conversation. Moreover, they are trained and evaluated on the benchmark datasets with adequate labels, which are expensive to obtain in a commercial dialogue system. To face these challenges, we propose a pipeline to predict the user satisfaction to help DuerOS decide whether to ask for clarification in each turn. Specifically, we propose to first generate a large number of weak labels and then train a transformer-based model to predict the user satisfaction with these weak labels. Moreover, we propose a metric, contextual user satisfaction, to evaluate the experience under the proactive interaction mechanism. At last, we deploy and evaluate our model on DuerOS, and observe a 19% relative improvement on the accuracy of user satisfaction prediction and 2.3% relative improvement on user experience.

Original languageEnglish
Title of host publicationCIKM '22
Subtitle of host publicationproceedings of the 31st ACM International Conference on Information and Knowledge Management
Place of PublicationNew York
PublisherAssociation for Computing Machinery
Number of pages10
ISBN (Electronic)9781450392365
Publication statusPublished - 17 Oct 2022
Event31st ACM International Conference on Information and Knowledge Management, CIKM 2022 - Atlanta, United States
Duration: 17 Oct 202221 Oct 2022


Conference31st ACM International Conference on Information and Knowledge Management, CIKM 2022
Country/TerritoryUnited States


  • Dialogue System
  • Transformer-based Model
  • User Satisfaction Prediction


Dive into the research topics of 'A transformer-based user satisfaction prediction for proactive interaction mechanism in DuerOS'. Together they form a unique fingerprint.

Cite this