Automatic Language Identification for Persian and Dari texts

Shervin Malmasi; Mark Dras

Automatic Language Identification for Persian and Dari texts

School of Computing

Research output: Contribution to conference › Paper

Abstract

We present the first empirical study of distinguishing Persian and Dari texts at the sentence level, using discriminative models. As Dari is a low-resourced language, we developed a corpus of 28k sentences (14k per-language) for this task, and using character and word n-grams, we discriminate them with 96% accuracy using a classifier ensemble. Outof-domain cross-corpus evaluation was conducted to test the discriminative models’ generalizability, achieving 87% accuracy in classifying 79k sentences from the Uppsala Persian Corpus. A feature analysis revealed lexical, morphological and orthographic differences between the two classes. A number of directions for future work are discussed.

Original language	English
Pages	53-58
Number of pages	6
Publication status	Published - 2015
Event	International Conference of the Pacific Association for Computational Linguistics (14th : 2015) - Bali, Indonesia Duration: 19 May 2015 → 21 May 2015

Conference

Conference	International Conference of the Pacific Association for Computational Linguistics (14th : 2015)
City	Bali, Indonesia
Period	19/05/15 → 21/05/15

Keywords

Language Identification
Dialect Identification
Persian
Farsi
Dari

Cite this

@conference{31d4ce5f17dc4c85a881a17ff0a820f8,

title = "Automatic Language Identification for Persian and Dari texts",

abstract = "We present the first empirical study of distinguishing Persian and Dari texts at the sentence level, using discriminative models. As Dari is a low-resourced language, we developed a corpus of 28k sentences (14k per-language) for this task, and using character and word n-grams, we discriminate them with 96\% accuracy using a classifier ensemble. Outof-domain cross-corpus evaluation was conducted to test the discriminative models{\textquoteright} generalizability, achieving 87\% accuracy in classifying 79k sentences from the Uppsala Persian Corpus. A feature analysis revealed lexical, morphological and orthographic differences between the two classes. A number of directions for future work are discussed.",

keywords = "Language Identification, Dialect Identification, Persian, Farsi, Dari",

author = "Shervin Malmasi and Mark Dras",

year = "2015",

language = "English",

pages = "53--58",

note = "International Conference of the Pacific Association for Computational Linguistics (14th : 2015) ; Conference date: 19-05-2015 Through 21-05-2015",

}

TY - CONF

T1 - Automatic Language Identification for Persian and Dari texts

AU - Malmasi, Shervin

AU - Dras, Mark

PY - 2015

Y1 - 2015

N2 - We present the first empirical study of distinguishing Persian and Dari texts at the sentence level, using discriminative models. As Dari is a low-resourced language, we developed a corpus of 28k sentences (14k per-language) for this task, and using character and word n-grams, we discriminate them with 96% accuracy using a classifier ensemble. Outof-domain cross-corpus evaluation was conducted to test the discriminative models’ generalizability, achieving 87% accuracy in classifying 79k sentences from the Uppsala Persian Corpus. A feature analysis revealed lexical, morphological and orthographic differences between the two classes. A number of directions for future work are discussed.

AB - We present the first empirical study of distinguishing Persian and Dari texts at the sentence level, using discriminative models. As Dari is a low-resourced language, we developed a corpus of 28k sentences (14k per-language) for this task, and using character and word n-grams, we discriminate them with 96% accuracy using a classifier ensemble. Outof-domain cross-corpus evaluation was conducted to test the discriminative models’ generalizability, achieving 87% accuracy in classifying 79k sentences from the Uppsala Persian Corpus. A feature analysis revealed lexical, morphological and orthographic differences between the two classes. A number of directions for future work are discussed.

KW - Language Identification

KW - Dialect Identification

KW - Persian

KW - Farsi

KW - Dari

M3 - Paper

SP - 53

EP - 58

T2 - International Conference of the Pacific Association for Computational Linguistics (14th : 2015)

Y2 - 19 May 2015 through 21 May 2015

ER -

Automatic Language Identification for Persian and Dari texts

Abstract

Conference

Keywords

Fingerprint

Cite this