Improving disfluency detection by self-training a self-attentive model

Paria Jamshid Lou; Mark Johnson

Improving disfluency detection by self-training a self-attentive model

Paria Jamshid Lou^*, Mark Johnson

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference proceeding contribution › peer-review

30 Citations (Scopus)

55 Downloads (Pure)

Abstract

Self-attentive neural syntactic parsers using contextualized word embeddings (e.g. ELMo or BERT) currently produce state-of-the-art results in joint parsing and disfluency detection in speech transcripts. Since the contextualized word embeddings are pre-trained on a large amount of unlabeled data, using additional unlabeled data to train a neural model might seem redundant. However, we show that self-training - a semi-supervised technique for incorporating unlabeled data - sets a new state-of-the-art for the self-attentive parser on disfluency detection, demonstrating that self-training provides benefits orthogonal to the pre-trained contextualized word representations. We also show that ensembling self-trained parsers provides further gains for disfluency detection.

Original language	English
Title of host publication	The 58th Annual Meeting of the Association for Computational Linguistics
Subtitle of host publication	Proceedings of the Conference
Place of Publication	Stroudsburg, PA
Publisher	Association for Computational Linguistics (ACL)
Pages	3754-3763
Number of pages	10
ISBN (Print)	9781952148255
Publication status	Published - 2020
Event	58th Annual Meeting of the Association for Computational Linguistics (ACL) - Duration: 5 Jul 2020 → 10 Jul 2020

Publication series

Name	Proceedings of the Annual Meeting of the Association for Computational Linguistics
ISSN (Print)	0736-587X

Conference

Conference	58th Annual Meeting of the Association for Computational Linguistics (ACL)
Period	5/07/20 → 10/07/20

Bibliographical note

Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.

Access to Document

Publisher version (open access)Final published version, 387 KBLicence: CC BY

https://www.aclweb.org/anthology/2020.acl-main.346

Cite this

Lou, P. J., & Johnson, M. (2020). Improving disfluency detection by self-training a self-attentive model. In The 58th Annual Meeting of the Association for Computational Linguistics: Proceedings of the Conference (pp. 3754-3763). (Proceedings of the Annual Meeting of the Association for Computational Linguistics). Association for Computational Linguistics (ACL). https://www.aclweb.org/anthology/2020.acl-main.346

Lou, Paria Jamshid ; Johnson, Mark. / Improving disfluency detection by self-training a self-attentive model. The 58th Annual Meeting of the Association for Computational Linguistics: Proceedings of the Conference. Stroudsburg, PA : Association for Computational Linguistics (ACL), 2020. pp. 3754-3763 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

@inproceedings{685feff04361493c82b15c0a837023d6,

title = "Improving disfluency detection by self-training a self-attentive model",

abstract = "Self-attentive neural syntactic parsers using contextualized word embeddings (e.g. ELMo or BERT) currently produce state-of-the-art results in joint parsing and disfluency detection in speech transcripts. Since the contextualized word embeddings are pre-trained on a large amount of unlabeled data, using additional unlabeled data to train a neural model might seem redundant. However, we show that self-training - a semi-supervised technique for incorporating unlabeled data - sets a new state-of-the-art for the self-attentive parser on disfluency detection, demonstrating that self-training provides benefits orthogonal to the pre-trained contextualized word representations. We also show that ensembling self-trained parsers provides further gains for disfluency detection.",

author = "Lou, \{Paria Jamshid\} and Mark Johnson",

note = "Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.; 58th Annual Meeting of the Association for Computational Linguistics (ACL) ; Conference date: 05-07-2020 Through 10-07-2020",

year = "2020",

language = "English",

isbn = "9781952148255",

series = "Proceedings of the Annual Meeting of the Association for Computational Linguistics",

publisher = "Association for Computational Linguistics (ACL)",

pages = "3754--3763",

booktitle = "The 58th Annual Meeting of the Association for Computational Linguistics",

}

Lou, PJ & Johnson, M 2020, Improving disfluency detection by self-training a self-attentive model. in The 58th Annual Meeting of the Association for Computational Linguistics: Proceedings of the Conference. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics (ACL), Stroudsburg, PA, pp. 3754-3763, 58th Annual Meeting of the Association for Computational Linguistics (ACL) , 5/07/20. <https://www.aclweb.org/anthology/2020.acl-main.346>

Improving disfluency detection by self-training a self-attentive model. / Lou, Paria Jamshid; Johnson, Mark.
The 58th Annual Meeting of the Association for Computational Linguistics: Proceedings of the Conference. Stroudsburg, PA: Association for Computational Linguistics (ACL), 2020. p. 3754-3763 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

Research output: Chapter in Book/Report/Conference proceeding › Conference proceeding contribution › peer-review

TY - GEN

T1 - Improving disfluency detection by self-training a self-attentive model

AU - Lou, Paria Jamshid

AU - Johnson, Mark

N1 - Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.

PY - 2020

Y1 - 2020

N2 - Self-attentive neural syntactic parsers using contextualized word embeddings (e.g. ELMo or BERT) currently produce state-of-the-art results in joint parsing and disfluency detection in speech transcripts. Since the contextualized word embeddings are pre-trained on a large amount of unlabeled data, using additional unlabeled data to train a neural model might seem redundant. However, we show that self-training - a semi-supervised technique for incorporating unlabeled data - sets a new state-of-the-art for the self-attentive parser on disfluency detection, demonstrating that self-training provides benefits orthogonal to the pre-trained contextualized word representations. We also show that ensembling self-trained parsers provides further gains for disfluency detection.

AB - Self-attentive neural syntactic parsers using contextualized word embeddings (e.g. ELMo or BERT) currently produce state-of-the-art results in joint parsing and disfluency detection in speech transcripts. Since the contextualized word embeddings are pre-trained on a large amount of unlabeled data, using additional unlabeled data to train a neural model might seem redundant. However, we show that self-training - a semi-supervised technique for incorporating unlabeled data - sets a new state-of-the-art for the self-attentive parser on disfluency detection, demonstrating that self-training provides benefits orthogonal to the pre-trained contextualized word representations. We also show that ensembling self-trained parsers provides further gains for disfluency detection.

UR - http://purl.org/au-research/grants/arc/DP160102156

UR - https://www.scopus.com/pages/publications/85108393465

M3 - Conference proceeding contribution

SN - 9781952148255

T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics

SP - 3754

EP - 3763

BT - The 58th Annual Meeting of the Association for Computational Linguistics

PB - Association for Computational Linguistics (ACL)

CY - Stroudsburg, PA

T2 - 58th Annual Meeting of the Association for Computational Linguistics (ACL)

Y2 - 5 July 2020 through 10 July 2020

ER -

Lou PJ, Johnson M. Improving disfluency detection by self-training a self-attentive model. In The 58th Annual Meeting of the Association for Computational Linguistics: Proceedings of the Conference. Stroudsburg, PA: Association for Computational Linguistics (ACL). 2020. p. 3754-3763. (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

Improving disfluency detection by self-training a self-attentive model

Abstract

Publication series

Conference

Bibliographical note

Access to Document

Other files and links

Fingerprint

Improved syntactic and semantic analysis for natural language processing

Cite this

Improving disfluency detection by self-training a self-attentive model

Abstract

Publication series

Conference

Bibliographical note

Access to Document

Other files and links

Fingerprint

Projects

Improved syntactic and semantic analysis for natural language processing

Cite this