A framework for evaluating text correction

Robert Dale, George Narroway

Research output: Chapter in Book/Report/Conference proceedingConference proceeding contributionpeer-review

1 Citation (Scopus)

Abstract

Computer-based aids for writing assistance have been around since at least the early 1980s, focussing primarily on aspects such as spelling, grammar and style. The potential audience for such tools is very large indeed, and this is a clear case where we might expect to see language processing applications having a significant real-world impact. However, existing comparative evaluations of applications in this space are often no more than impressionistic and anecdotal reviews of commercial offerings as found in software magazines, making it hard to determine which approaches are superior. More rigorous evaluation in the scholarly literature has been held back in particular by the absence of shared datasets of texts marked-up with errors, and the lack of an agreed evaluation framework. Significant collections of publicly available data are now appearing; this paper describes a complementary evaluation framework, which has been piloted in the Helping Our Own shared task. The approach, which uses stand-off annotations for representing edits to text, can be used in a wide variety of text-correction tasks, and easily accommodates different error tagsets.
Original languageEnglish
Title of host publicationProceedings of the Eight International Conference on Language Resources and Evaluation (LREC'12)
EditorsNicoletta Calzolari, Khalid Choukri, Thierry Declerck, Mehmet Uğur Doğan, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis
PublisherEuropean Language Resources Association (ELRA)
Pages3015-3018
Number of pages4
ISBN (Print)9782951740877
Publication statusPublished - 2012
EventInternational Conference on Language Resources and Evaluation (8th : 2012) - Istanbul, Turkey
Duration: 23 May 201225 May 2012

Conference

ConferenceInternational Conference on Language Resources and Evaluation (8th : 2012)
CityIstanbul, Turkey
Period23/05/1225/05/12

Keywords

  • evaluation frameworks
  • error correction
  • replicability

Cite this