A fast and accurate Vietnamese word segmenter

Dat Quoc Nguyen; Dai Quoc Nguyen; Thanh Vu; Mark Dras; Mark Johnson

A fast and accurate Vietnamese word segmenter

Dat Quoc Nguyen, Dai Quoc Nguyen, Thanh Vu, Mark Dras, Mark Johnson

School of Computing

Research output: Chapter in Book/Report/Conference proceeding › Conference proceeding contribution › peer-review

43 Citations (Scopus)

420 Downloads (Pure)

Abstract

We propose a novel approach to Vietnamese word segmentation. Our approach is based on the Single Classification Ripple Down Rules methodology (Compton and Jansen, 1990), where rules are stored in an exception structure and new rules are only added to correct segmentation errors given by existing rules. Experimental results on the benchmark Vietnamese treebank show that our approach outperforms previous state-of-the-art approaches JVnSegmenter, vnTokenizer, DongDu and UETsegmenter in terms of both accuracy and performance speed. Our code is open-source and available at: https://github.com/datquocnguyen/RDRsegmenter.

Original language	English
Title of host publication	LREC 2018 - 11th International Conference on Language Resources and Evaluation
Editors	Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis, Takenobu Tokunaga
Place of Publication	Luxemburg
Publisher	European Language Resources Association (ELRA)
Pages	2582-2587
Number of pages	6
ISBN (Electronic)	9791095546009
Publication status	Published - 1 Jan 2019
Event	11th International Conference on Language Resources and Evaluation, LREC 2018 - Miyazaki, Japan Duration: 7 May 2018 → 12 May 2018

Conference

Conference	11th International Conference on Language Resources and Evaluation, LREC 2018
Country/Territory	Japan
City	Miyazaki
Period	7/05/18 → 12/05/18

Bibliographical note

Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.

Keywords

Single classification ripple down rules
Vietnamese
Word segmentation

Access to Document

Publisher version (open access)Final published version, 348 KBLicence: CC BY-NC

Cite this

Nguyen, D. Q., Nguyen, D. Q., Vu, T., Dras, M., & Johnson, M. (2019). A fast and accurate Vietnamese word segmenter. In N. Calzolari, K. Choukri, C. Cieri, T. Declerck, S. Goggi, K. Hasida, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, S. Piperidis, & T. Tokunaga (Eds.), LREC 2018 - 11th International Conference on Language Resources and Evaluation (pp. 2582-2587). European Language Resources Association (ELRA).

Nguyen, Dat Quoc ; Nguyen, Dai Quoc ; Vu, Thanh et al. / A fast and accurate Vietnamese word segmenter. LREC 2018 - 11th International Conference on Language Resources and Evaluation. editor / Nicoletta Calzolari ; Khalid Choukri ; Christopher Cieri ; Thierry Declerck ; Sara Goggi ; Koiti Hasida ; Hitoshi Isahara ; Bente Maegaard ; Joseph Mariani ; Helene Mazo ; Asuncion Moreno ; Jan Odijk ; Stelios Piperidis ; Takenobu Tokunaga. Luxemburg : European Language Resources Association (ELRA), 2019. pp. 2582-2587

@inproceedings{d7ee9967c1384b6aa9da9e75ca4eb65c,

title = "A fast and accurate Vietnamese word segmenter",

abstract = "We propose a novel approach to Vietnamese word segmentation. Our approach is based on the Single Classification Ripple Down Rules methodology (Compton and Jansen, 1990), where rules are stored in an exception structure and new rules are only added to correct segmentation errors given by existing rules. Experimental results on the benchmark Vietnamese treebank show that our approach outperforms previous state-of-the-art approaches JVnSegmenter, vnTokenizer, DongDu and UETsegmenter in terms of both accuracy and performance speed. Our code is open-source and available at: https://github.com/datquocnguyen/RDRsegmenter.",

keywords = "Single classification ripple down rules, Vietnamese, Word segmentation",

author = "Nguyen, {Dat Quoc} and Nguyen, {Dai Quoc} and Thanh Vu and Mark Dras and Mark Johnson",

note = "Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.; 11th International Conference on Language Resources and Evaluation, LREC 2018 ; Conference date: 07-05-2018 Through 12-05-2018",

year = "2019",

month = jan,

day = "1",

language = "English",

pages = "2582--2587",

editor = "Nicoletta Calzolari and Khalid Choukri and Christopher Cieri and Thierry Declerck and Sara Goggi and Koiti Hasida and Hitoshi Isahara and Bente Maegaard and Joseph Mariani and Helene Mazo and Asuncion Moreno and Jan Odijk and Stelios Piperidis and Takenobu Tokunaga",

booktitle = "LREC 2018 - 11th International Conference on Language Resources and Evaluation",

publisher = "European Language Resources Association (ELRA)",

}

Nguyen, DQ, Nguyen, DQ, Vu, T, Dras, M & Johnson, M 2019, A fast and accurate Vietnamese word segmenter. in N Calzolari, K Choukri, C Cieri, T Declerck, S Goggi, K Hasida, H Isahara, B Maegaard, J Mariani, H Mazo, A Moreno, J Odijk, S Piperidis & T Tokunaga (eds), LREC 2018 - 11th International Conference on Language Resources and Evaluation. European Language Resources Association (ELRA), Luxemburg, pp. 2582-2587, 11th International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, 7/05/18.

A fast and accurate Vietnamese word segmenter. / Nguyen, Dat Quoc; Nguyen, Dai Quoc; Vu, Thanh et al.
LREC 2018 - 11th International Conference on Language Resources and Evaluation. ed. / Nicoletta Calzolari; Khalid Choukri; Christopher Cieri; Thierry Declerck; Sara Goggi; Koiti Hasida; Hitoshi Isahara; Bente Maegaard; Joseph Mariani; Helene Mazo; Asuncion Moreno; Jan Odijk; Stelios Piperidis; Takenobu Tokunaga. Luxemburg: European Language Resources Association (ELRA), 2019. p. 2582-2587.

Research output: Chapter in Book/Report/Conference proceeding › Conference proceeding contribution › peer-review

TY - GEN

T1 - A fast and accurate Vietnamese word segmenter

AU - Nguyen, Dat Quoc

AU - Nguyen, Dai Quoc

AU - Vu, Thanh

AU - Dras, Mark

AU - Johnson, Mark

N1 - Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.

PY - 2019/1/1

Y1 - 2019/1/1

N2 - We propose a novel approach to Vietnamese word segmentation. Our approach is based on the Single Classification Ripple Down Rules methodology (Compton and Jansen, 1990), where rules are stored in an exception structure and new rules are only added to correct segmentation errors given by existing rules. Experimental results on the benchmark Vietnamese treebank show that our approach outperforms previous state-of-the-art approaches JVnSegmenter, vnTokenizer, DongDu and UETsegmenter in terms of both accuracy and performance speed. Our code is open-source and available at: https://github.com/datquocnguyen/RDRsegmenter.

AB - We propose a novel approach to Vietnamese word segmentation. Our approach is based on the Single Classification Ripple Down Rules methodology (Compton and Jansen, 1990), where rules are stored in an exception structure and new rules are only added to correct segmentation errors given by existing rules. Experimental results on the benchmark Vietnamese treebank show that our approach outperforms previous state-of-the-art approaches JVnSegmenter, vnTokenizer, DongDu and UETsegmenter in terms of both accuracy and performance speed. Our code is open-source and available at: https://github.com/datquocnguyen/RDRsegmenter.

KW - Single classification ripple down rules

KW - Vietnamese

KW - Word segmentation

UR - http://www.scopus.com/inward/record.url?scp=85059886819&partnerID=8YFLogxK

UR - http://www.lrec-conf.org/proceedings/lrec2018/index.html

M3 - Conference proceeding contribution

AN - SCOPUS:85059886819

SP - 2582

EP - 2587

BT - LREC 2018 - 11th International Conference on Language Resources and Evaluation

A2 - Calzolari, Nicoletta

A2 - Choukri, Khalid

A2 - Cieri, Christopher

A2 - Declerck, Thierry

A2 - Goggi, Sara

A2 - Hasida, Koiti

A2 - Isahara, Hitoshi

A2 - Maegaard, Bente

A2 - Mariani, Joseph

A2 - Mazo, Helene

A2 - Moreno, Asuncion

A2 - Odijk, Jan

A2 - Piperidis, Stelios

A2 - Tokunaga, Takenobu

PB - European Language Resources Association (ELRA)

CY - Luxemburg

T2 - 11th International Conference on Language Resources and Evaluation, LREC 2018

Y2 - 7 May 2018 through 12 May 2018

ER -

Nguyen DQ, Nguyen DQ, Vu T, Dras M , Johnson M. A fast and accurate Vietnamese word segmenter. In Calzolari N, Choukri K, Cieri C, Declerck T, Goggi S, Hasida K, Isahara H, Maegaard B, Mariani J, Mazo H, Moreno A, Odijk J, Piperidis S, Tokunaga T, editors, LREC 2018 - 11th International Conference on Language Resources and Evaluation. Luxemburg: European Language Resources Association (ELRA). 2019. p. 2582-2587

A fast and accurate Vietnamese word segmenter

Abstract

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this