Abstract
Because obtaining training data is often the most difficult part of an NLP or ML project, we develop methods for predicting how much data is required to achieve a desired test accuracy by extrapolating results from systems trained on a small pilot training dataset. We model how accuracy varies as a function of training size on subsets of the pilot data, and use that model to predict how much training data would be required to achieve the desired accuracy. We introduce a new performance extrapolation task to evaluate how well different extrapolations predict system accuracy on larger training sets. We show that details of hyperparameter optimisation and the extrapolation models can have dramatic effects in a document classification task. We believe this is an important first step in developing methods for estimating the resources required to meet specific engineering performance targets.
Original language | English |
---|---|
Title of host publication | ACL 2018 The 56th Annual Meeting of the Association for Computational Linguistics |
Subtitle of host publication | Proceedings of the Conference, Vol. 2 (Short Papers) |
Editors | Iryna Gurevych, Yusuke Miyao |
Publisher | Association for Computational Linguistics (ACL) |
Pages | 450-455 |
Number of pages | 6 |
Volume | 2 |
ISBN (Electronic) | 9781948087346 |
DOIs | |
Publication status | Published - 1 Jan 2018 |
Event | 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018 - Melbourne, Australia Duration: 15 Jul 2018 → 20 Jul 2018 |
Conference
Conference | 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018 |
---|---|
Country/Territory | Australia |
City | Melbourne |
Period | 15/07/18 → 20/07/18 |