WordSeg

standardizing unsupervised word form segmentation from text

Mathieu Bernard, Roland Thiolliere, Amanda Saksida, Georgia R. Loukatou, Elin Larsen, Mark Johnson, Laia Fibla, Emmanuel Dupoux, Robert Daland, Xuan Nga Cao, Alejandrina Cristia*

*Corresponding author for this work

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

A basic task in first language acquisition likely involves discovering the boundaries between words or morphemes in input where these basic units are not overtly segmented. A number of unsupervised learning algorithms have been proposed in the last 20 years for these purposes, some of which have been implemented computationally, but whose results remain difficult to compare across papers. We created a tool that is open source, enables reproducible results, and encourages cumulative science in this domain. WordSeg has a modular architecture: It combines a set of corpora description routines, multiple algorithms varying in complexity and cognitive assumptions (including several that were not publicly available, or insufficiently documented), and a rich evaluation package. In the paper, we illustrate the use of this package by analyzing a corpus of child-directed speech in various ways, which further allows us to make recommendations for experimental design of follow-up work. Supplementary materials allow readers to reproduce every result in this paper, and detailed online instructions further enable them to go beyond what we have done. Moreover, the system can be installed within container software that ensures a stable and reliable environment. Finally, by virtue of its modular architecture and transparency, WordSeg can work as an open-source platform, to which other researchers can add their own segmentation algorithms.

Original languageEnglish
Pages (from-to)264-278
Number of pages15
JournalBehavior Research Methods
Volume52
Issue number1
Early online date1 Apr 2019
DOIs
Publication statusPublished - Feb 2020

Keywords

  • Unsupervised word discovery
  • First language acquisition
  • Natural language processing
  • Cumulative science

Fingerprint Dive into the research topics of 'WordSeg: standardizing unsupervised word form segmentation from text'. Together they form a unique fingerprint.

  • Cite this

    Bernard, M., Thiolliere, R., Saksida, A., Loukatou, G. R., Larsen, E., Johnson, M., ... Cristia, A. (2020). WordSeg: standardizing unsupervised word form segmentation from text. Behavior Research Methods, 52(1), 264-278. https://doi.org/10.3758/s13428-019-01223-3