Predicting accuracy on large datasets from smaller pilot data

Mark Johnson, Peter Anderson, Mark Dras, Mark Steedman

Research output: Chapter in Book/Report/Conference proceedingConference proceeding contributionpeer-review

35 Citations (Scopus)
93 Downloads (Pure)

Abstract

Because obtaining training data is often the most difficult part of an NLP or ML project, we develop methods for predicting how much data is required to achieve a desired test accuracy by extrapolating results from systems trained on a small pilot training dataset. We model how accuracy varies as a function of training size on subsets of the pilot data, and use that model to predict how much training data would be required to achieve the desired accuracy. We introduce a new performance extrapolation task to evaluate how well different extrapolations predict system accuracy on larger training sets. We show that details of hyperparameter optimisation and the extrapolation models can have dramatic effects in a document classification task. We believe this is an important first step in developing methods for estimating the resources required to meet specific engineering performance targets.

Original languageEnglish
Title of host publicationACL 2018 The 56th Annual Meeting of the Association for Computational Linguistics
Subtitle of host publicationProceedings of the Conference, Vol. 2 (Short Papers)
EditorsIryna Gurevych, Yusuke Miyao
PublisherAssociation for Computational Linguistics (ACL)
Pages450-455
Number of pages6
Volume2
ISBN (Electronic)9781948087346
DOIs
Publication statusPublished - 1 Jan 2018
Event56th Annual Meeting of the Association for Computational Linguistics, ACL 2018 - Melbourne, Australia
Duration: 15 Jul 201820 Jul 2018

Conference

Conference56th Annual Meeting of the Association for Computational Linguistics, ACL 2018
Country/TerritoryAustralia
CityMelbourne
Period15/07/1820/07/18

Bibliographical note

Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.

Fingerprint

Dive into the research topics of 'Predicting accuracy on large datasets from smaller pilot data'. Together they form a unique fingerprint.

Cite this