Answering unanswered questions through semantic reformulations in Spoken QA

Pedro Faustini; Zhiyu Chen; Besnik Fetahu; Oleg Rokhlenko; Shervin Malmasi

doi:10.18653/v1/2023.acl-industry.70

Answering unanswered questions through semantic reformulations in Spoken QA

Pedro Faustini, Zhiyu Chen, Besnik Fetahu, Oleg Rokhlenko, Shervin Malmasi

School of Computing

Research output: Chapter in Book/Report/Conference proceeding › Conference proceeding contribution › peer-review

5 Citations (Scopus)

Abstract

Spoken Question Answering (QA) is a key feature of voice assistants, usually backed by multiple QA systems. Users ask questions via spontaneous speech that can contain disfluencies, errors, and informal syntax or phrasing. This is a major challenge in QA, causing unanswered questions or irrelevant answers, leading to bad user experiences. We analyze failed QA requests to identify core challenges: lexical gaps, proposition types, complex syntactic structure, and high specificity. We propose a Semantic Question Reformulation (SURF) model offering three linguistically-grounded operations (repair, syntactic reshaping, generalization) to rewrite questions to facilitate answering. Offline evaluation on 1M unanswered questions from a leading voice assistant shows that SURF significantly improves answer rates: up to 24% of previously unanswered questions obtain relevant answers (75%). Live deployment shows positive impact for millions of customers with unanswered questions; explicit relevance feedback shows high user satisfaction.

Original language	English
Title of host publication	Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics volume 5
Subtitle of host publication	Industry Track
Place of Publication	Stroudsburg
Publisher	Association for Computational Linguistics
Pages	729-743
Number of pages	15
ISBN (Electronic)	9781959429685
DOIs	https://doi.org/10.18653/v1/2023.acl-industry.70
Publication status	Published - 2023
Event	Annual Meeting of the Association for Computational Linguistics (61st : 2023) - Toronto, Canada Duration: 9 Jul 2023 → 14 Jul 2023 Conference number: 61st https://2023.aclweb.org/

Publication series

Name	Proceedings of the Annual Meeting of the Association for Computational Linguistics
Volume	5
ISSN (Print)	0736-587X

Conference

Conference	Annual Meeting of the Association for Computational Linguistics (61st : 2023)
Abbreviated title	ACL'23
Country/Territory	Canada
City	Toronto
Period	9/07/23 → 14/07/23
Internet address	https://2023.aclweb.org/

Access to Document

10.18653/v1/2023.acl-industry.70Licence: Unspecified

Cite this

Faustini, P., Chen, Z., Fetahu, B., Rokhlenko, O., & Malmasi, S. (2023). Answering unanswered questions through semantic reformulations in Spoken QA. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics volume 5: Industry Track (pp. 729-743). (Proceedings of the Annual Meeting of the Association for Computational Linguistics; Vol. 5). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.acl-industry.70

Faustini, Pedro ; Chen, Zhiyu ; Fetahu, Besnik et al. / Answering unanswered questions through semantic reformulations in Spoken QA. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics volume 5: Industry Track. Stroudsburg : Association for Computational Linguistics, 2023. pp. 729-743 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

@inproceedings{2053fdfff5ad4d329e0bb9da0c1954f5,

title = "Answering unanswered questions through semantic reformulations in Spoken QA",

abstract = "Spoken Question Answering (QA) is a key feature of voice assistants, usually backed by multiple QA systems. Users ask questions via spontaneous speech that can contain disfluencies, errors, and informal syntax or phrasing. This is a major challenge in QA, causing unanswered questions or irrelevant answers, leading to bad user experiences. We analyze failed QA requests to identify core challenges: lexical gaps, proposition types, complex syntactic structure, and high specificity. We propose a Semantic Question Reformulation (SURF) model offering three linguistically-grounded operations (repair, syntactic reshaping, generalization) to rewrite questions to facilitate answering. Offline evaluation on 1M unanswered questions from a leading voice assistant shows that SURF significantly improves answer rates: up to 24\% of previously unanswered questions obtain relevant answers (75\%). Live deployment shows positive impact for millions of customers with unanswered questions; explicit relevance feedback shows high user satisfaction.",

author = "Pedro Faustini and Zhiyu Chen and Besnik Fetahu and Oleg Rokhlenko and Shervin Malmasi",

year = "2023",

doi = "10.18653/v1/2023.acl-industry.70",

language = "English",

series = "Proceedings of the Annual Meeting of the Association for Computational Linguistics",

publisher = "Association for Computational Linguistics",

pages = "729--743",

booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics volume 5",

note = "Annual Meeting of the Association for Computational Linguistics (61st : 2023), ACL'23 ; Conference date: 09-07-2023 Through 14-07-2023",

url = "https://2023.aclweb.org/",

}

Faustini, P, Chen, Z, Fetahu, B, Rokhlenko, O & Malmasi, S 2023, Answering unanswered questions through semantic reformulations in Spoken QA. in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics volume 5: Industry Track. Proceedings of the Annual Meeting of the Association for Computational Linguistics, vol. 5, Association for Computational Linguistics, Stroudsburg, pp. 729-743, Annual Meeting of the Association for Computational Linguistics (61st : 2023), Toronto, Canada, 9/07/23. https://doi.org/10.18653/v1/2023.acl-industry.70

Answering unanswered questions through semantic reformulations in Spoken QA. / Faustini, Pedro; Chen, Zhiyu; Fetahu, Besnik et al.
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics volume 5: Industry Track. Stroudsburg: Association for Computational Linguistics, 2023. p. 729-743 (Proceedings of the Annual Meeting of the Association for Computational Linguistics; Vol. 5).

Research output: Chapter in Book/Report/Conference proceeding › Conference proceeding contribution › peer-review

TY - GEN

T1 - Answering unanswered questions through semantic reformulations in Spoken QA

AU - Faustini, Pedro

AU - Chen, Zhiyu

AU - Fetahu, Besnik

AU - Rokhlenko, Oleg

AU - Malmasi, Shervin

N1 - Conference code: 61st

PY - 2023

Y1 - 2023

N2 - Spoken Question Answering (QA) is a key feature of voice assistants, usually backed by multiple QA systems. Users ask questions via spontaneous speech that can contain disfluencies, errors, and informal syntax or phrasing. This is a major challenge in QA, causing unanswered questions or irrelevant answers, leading to bad user experiences. We analyze failed QA requests to identify core challenges: lexical gaps, proposition types, complex syntactic structure, and high specificity. We propose a Semantic Question Reformulation (SURF) model offering three linguistically-grounded operations (repair, syntactic reshaping, generalization) to rewrite questions to facilitate answering. Offline evaluation on 1M unanswered questions from a leading voice assistant shows that SURF significantly improves answer rates: up to 24% of previously unanswered questions obtain relevant answers (75%). Live deployment shows positive impact for millions of customers with unanswered questions; explicit relevance feedback shows high user satisfaction.

AB - Spoken Question Answering (QA) is a key feature of voice assistants, usually backed by multiple QA systems. Users ask questions via spontaneous speech that can contain disfluencies, errors, and informal syntax or phrasing. This is a major challenge in QA, causing unanswered questions or irrelevant answers, leading to bad user experiences. We analyze failed QA requests to identify core challenges: lexical gaps, proposition types, complex syntactic structure, and high specificity. We propose a Semantic Question Reformulation (SURF) model offering three linguistically-grounded operations (repair, syntactic reshaping, generalization) to rewrite questions to facilitate answering. Offline evaluation on 1M unanswered questions from a leading voice assistant shows that SURF significantly improves answer rates: up to 24% of previously unanswered questions obtain relevant answers (75%). Live deployment shows positive impact for millions of customers with unanswered questions; explicit relevance feedback shows high user satisfaction.

UR - https://www.scopus.com/pages/publications/85174222152

U2 - 10.18653/v1/2023.acl-industry.70

DO - 10.18653/v1/2023.acl-industry.70

M3 - Conference proceeding contribution

T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics

SP - 729

EP - 743

BT - Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics volume 5

PB - Association for Computational Linguistics

CY - Stroudsburg

T2 - Annual Meeting of the Association for Computational Linguistics (61st : 2023)

Y2 - 9 July 2023 through 14 July 2023

ER -

Faustini P, Chen Z, Fetahu B, Rokhlenko O, Malmasi S. Answering unanswered questions through semantic reformulations in Spoken QA. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics volume 5: Industry Track. Stroudsburg: Association for Computational Linguistics. 2023. p. 729-743. (Proceedings of the Annual Meeting of the Association for Computational Linguistics). doi: 10.18653/v1/2023.acl-industry.70

Answering unanswered questions through semantic reformulations in Spoken QA

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint

Cite this