Feature hashing for language and dialect identification

Shervin Malmasi; Mark Dras

doi:10.18653/v1/P17-2063

Feature hashing for language and dialect identification

Shervin Malmasi, Mark Dras

School of Computing

Research output: Chapter in Book/Report/Conference proceeding › Conference proceeding contribution › peer-review

3 Citations (Scopus)

35 Downloads (Pure)

Abstract

We evaluate feature hashing for language identification (LID), a method not previously used for this task. Using a standard dataset, we first show that while feature performance is high, LID data is highly dimensional and mostly sparse (>99.5%) as it includes large vocabularies for many languages; memory requirements grow as languages are added. Next we apply hashing using various hash sizes, demonstrating that there is no performance loss with dimensionality reductions of up to 86%. We also show that using an ensemble of low-dimension hash-based classifiers further boosts performance. Feature hashing is highly useful for LID and holds great promise for future work in this area.

Original language	English
Title of host publication	ACL 2017
Subtitle of host publication	Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Short Papers)
Editors	Regina Barzilay, Min-Yen Kan
Place of Publication	Stroudsburg, PA
Publisher	Association for Computational Linguistics (ACL)
Pages	399-403
Number of pages	5
Volume	2
ISBN (Electronic)	9781945626760
DOIs	https://doi.org/10.18653/v1/P17-2063
Publication status	Published - 2017
Event	55th Annual Meeting of the Association for Computational Linguistics, ACL 2017 - Vancouver, Canada Duration: 30 Jul 2017 → 4 Aug 2017

Conference

Conference	55th Annual Meeting of the Association for Computational Linguistics, ACL 2017
Country/Territory	Canada
City	Vancouver
Period	30/07/17 → 4/08/17

Bibliographical note

Copyright the Publisher. Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.

Access to Document

10.18653/v1/P17-2063

Publisher version (open access)Final published version, 240 KBLicence: CC BY

Cite this

@inproceedings{f82d79176f9347daac5f442974f37a85,

title = "Feature hashing for language and dialect identification",

abstract = "We evaluate feature hashing for language identification (LID), a method not previously used for this task. Using a standard dataset, we first show that while feature performance is high, LID data is highly dimensional and mostly sparse (>99.5%) as it includes large vocabularies for many languages; memory requirements grow as languages are added. Next we apply hashing using various hash sizes, demonstrating that there is no performance loss with dimensionality reductions of up to 86%. We also show that using an ensemble of low-dimension hash-based classifiers further boosts performance. Feature hashing is highly useful for LID and holds great promise for future work in this area.",

author = "Shervin Malmasi and Mark Dras",

note = "Copyright the Publisher. Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.; 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017 ; Conference date: 30-07-2017 Through 04-08-2017",

year = "2017",

doi = "10.18653/v1/P17-2063",

language = "English",

volume = "2",

pages = "399--403",

editor = "Regina Barzilay and Min-Yen Kan",

booktitle = "ACL 2017",

publisher = "Association for Computational Linguistics (ACL)",

}

Malmasi, S & Dras, M 2017, Feature hashing for language and dialect identification. in R Barzilay & M-Y Kan (eds), ACL 2017: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Short Papers). vol. 2, Association for Computational Linguistics (ACL), Stroudsburg, PA, pp. 399-403, 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, 30/07/17. https://doi.org/10.18653/v1/P17-2063

Feature hashing for language and dialect identification. / Malmasi, Shervin ; Dras, Mark.
ACL 2017: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Short Papers). ed. / Regina Barzilay; Min-Yen Kan. Vol. 2 Stroudsburg, PA: Association for Computational Linguistics (ACL), 2017. p. 399-403.

Research output: Chapter in Book/Report/Conference proceeding › Conference proceeding contribution › peer-review

TY - GEN

T1 - Feature hashing for language and dialect identification

AU - Malmasi, Shervin

AU - Dras, Mark

N1 - Copyright the Publisher. Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.

PY - 2017

Y1 - 2017

N2 - We evaluate feature hashing for language identification (LID), a method not previously used for this task. Using a standard dataset, we first show that while feature performance is high, LID data is highly dimensional and mostly sparse (>99.5%) as it includes large vocabularies for many languages; memory requirements grow as languages are added. Next we apply hashing using various hash sizes, demonstrating that there is no performance loss with dimensionality reductions of up to 86%. We also show that using an ensemble of low-dimension hash-based classifiers further boosts performance. Feature hashing is highly useful for LID and holds great promise for future work in this area.

AB - We evaluate feature hashing for language identification (LID), a method not previously used for this task. Using a standard dataset, we first show that while feature performance is high, LID data is highly dimensional and mostly sparse (>99.5%) as it includes large vocabularies for many languages; memory requirements grow as languages are added. Next we apply hashing using various hash sizes, demonstrating that there is no performance loss with dimensionality reductions of up to 86%. We also show that using an ensemble of low-dimension hash-based classifiers further boosts performance. Feature hashing is highly useful for LID and holds great promise for future work in this area.

UR - http://www.scopus.com/inward/record.url?scp=85040605553&partnerID=8YFLogxK

U2 - 10.18653/v1/P17-2063

DO - 10.18653/v1/P17-2063

M3 - Conference proceeding contribution

AN - SCOPUS:85040605553

VL - 2

SP - 399

EP - 403

BT - ACL 2017

A2 - Barzilay, Regina

A2 - Kan, Min-Yen

PB - Association for Computational Linguistics (ACL)

CY - Stroudsburg, PA

T2 - 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017

Y2 - 30 July 2017 through 4 August 2017

ER -

Feature hashing for language and dialect identification

Abstract

Conference

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this