Oracle and human baselines for native language identification

Shervin Malmasi; Joel Tetreault; Mark Dras

Oracle and human baselines for native language identification

Shervin Malmasi, Joel Tetreault, Mark Dras

School of Computing

Research output: Chapter in Book/Report/Conference proceeding › Conference proceeding contribution › peer-review

24 Citations (Scopus)

Abstract

We examine different ensemble methods, including an oracle, to estimate the upper-limit of classification accuracy for Native Language Identification (NLI). The oracle outperforms state-of-the-art systems by over 10% and results indicate that for many misclassified texts the correct class label receives a significant portion of the ensemble votes, often being the runner-up. We also present a pilot study of human performance for NLI, the first such experiment. While some participants achieve modest results on our simplified setup with 5 L1s, they did not outperform our NLI system, and this performance gap is likely to widen on the standard NLI setup.

Original language	English
Title of host publication	NAACL HLT 2015
Subtitle of host publication	The Tenth Workshop on Innovative Use of NLP for Building Educational Applications : proceedings of the workshop
Place of Publication	Red Hook, New York
Publisher	The Association for Computational Linguistics
Pages	172-178
Number of pages	7
ISBN (Print)	9781941643358
Publication status	Published - 2015
Event	Workshop on Innovative Use of NLP for Building Educational Applications (10th : 2015) - Denver, CO Duration: 4 Jun 2015 → 4 Jun 2015

Workshop

Workshop	Workshop on Innovative Use of NLP for Building Educational Applications (10th : 2015)
City	Denver, CO
Period	4/06/15 → 4/06/15

Access to Document

http://www.aclweb.org/anthology/W15-0620

Cite this

@inproceedings{c92eea09a5c74c48a90cfdc6cbd4a168,

title = "Oracle and human baselines for native language identification",

abstract = "We examine different ensemble methods, including an oracle, to estimate the upper-limit of classification accuracy for Native Language Identification (NLI). The oracle outperforms state-of-the-art systems by over 10% and results indicate that for many misclassified texts the correct class label receives a significant portion of the ensemble votes, often being the runner-up. We also present a pilot study of human performance for NLI, the first such experiment. While some participants achieve modest results on our simplified setup with 5 L1s, they did not outperform our NLI system, and this performance gap is likely to widen on the standard NLI setup.",

author = "Shervin Malmasi and Joel Tetreault and Mark Dras",

year = "2015",

language = "English",

isbn = "9781941643358",

pages = "172--178",

booktitle = "NAACL HLT 2015",

publisher = "The Association for Computational Linguistics",

note = "Workshop on Innovative Use of NLP for Building Educational Applications (10th : 2015) ; Conference date: 04-06-2015 Through 04-06-2015",

}

Malmasi, S, Tetreault, J & Dras, M 2015, Oracle and human baselines for native language identification. in NAACL HLT 2015: The Tenth Workshop on Innovative Use of NLP for Building Educational Applications : proceedings of the workshop. The Association for Computational Linguistics, Red Hook, New York, pp. 172-178, Workshop on Innovative Use of NLP for Building Educational Applications (10th : 2015), Denver, CO, 4/06/15. <http://www.aclweb.org/anthology/W15-0620>

Oracle and human baselines for native language identification. / Malmasi, Shervin; Tetreault, Joel; Dras, Mark.
NAACL HLT 2015: The Tenth Workshop on Innovative Use of NLP for Building Educational Applications : proceedings of the workshop. Red Hook, New York: The Association for Computational Linguistics, 2015. p. 172-178.

Research output: Chapter in Book/Report/Conference proceeding › Conference proceeding contribution › peer-review

TY - GEN

T1 - Oracle and human baselines for native language identification

AU - Malmasi, Shervin

AU - Tetreault, Joel

AU - Dras, Mark

PY - 2015

Y1 - 2015

N2 - We examine different ensemble methods, including an oracle, to estimate the upper-limit of classification accuracy for Native Language Identification (NLI). The oracle outperforms state-of-the-art systems by over 10% and results indicate that for many misclassified texts the correct class label receives a significant portion of the ensemble votes, often being the runner-up. We also present a pilot study of human performance for NLI, the first such experiment. While some participants achieve modest results on our simplified setup with 5 L1s, they did not outperform our NLI system, and this performance gap is likely to widen on the standard NLI setup.

AB - We examine different ensemble methods, including an oracle, to estimate the upper-limit of classification accuracy for Native Language Identification (NLI). The oracle outperforms state-of-the-art systems by over 10% and results indicate that for many misclassified texts the correct class label receives a significant portion of the ensemble votes, often being the runner-up. We also present a pilot study of human performance for NLI, the first such experiment. While some participants achieve modest results on our simplified setup with 5 L1s, they did not outperform our NLI system, and this performance gap is likely to widen on the standard NLI setup.

M3 - Conference proceeding contribution

SN - 9781941643358

SP - 172

EP - 178

BT - NAACL HLT 2015

PB - The Association for Computational Linguistics

CY - Red Hook, New York

T2 - Workshop on Innovative Use of NLP for Building Educational Applications (10th : 2015)

Y2 - 4 June 2015 through 4 June 2015

ER -

Oracle and human baselines for native language identification

Abstract

Workshop

Access to Document

Fingerprint

Cite this