Intrinsic versus extrinsic evaluations of parsing systems

Diego Mollá, Ben Hutchinson

Research output: Chapter in Book/Report/Conference proceedingConference proceeding contribution

Abstract

A wide range of parser and/or grammar evaluation methods have been reported in the literature. However, in most cases these evaluations take the parsers independently (intrinsic evaluations), and only in a few cases has the effect of different parsers in real applications been measured (extrinsic evaluations). This paper compares two evaluations of the Link Grammar parser and the Conexor Functional Dependency Grammar parser. The parsing systems, despite both being dependency-based, return different types of dependencies, making a direct comparison impossible. In the intrinsic evaluation, the accuracy of the parsers is compared independently by converting the dependencies into grammatical relations and using the methodology of Carroll et al. (1998) for parser comparison. In the extrinsic evaluation, the parsers' impact in a practical application is compared within the context of answer extraction. The differences in the results are significant.
Original languageEnglish
Title of host publicationEACL 2003
Subtitle of host publication10th Conference of the European Chapter : proceedings of the conference, April 12-17th, 2003, Agro Hotel, Budapest, Hungary
Place of PublicationEast Stroudsburg, PA
PublisherAssociation for Computational Linguistics
Pages43-50
Number of pages8
ISBN (Print)9781932432008
Publication statusPublished - 2003
EventAssociation for Computational Linguistics European Chapter Conference (10th : 2003) - Budapest
Duration: 12 Apr 200317 Apr 2003

Conference

ConferenceAssociation for Computational Linguistics European Chapter Conference (10th : 2003)
CityBudapest
Period12/04/0317/04/03

Cite this

Mollá, D., & Hutchinson, B. (2003). Intrinsic versus extrinsic evaluations of parsing systems. In EACL 2003: 10th Conference of the European Chapter : proceedings of the conference, April 12-17th, 2003, Agro Hotel, Budapest, Hungary (pp. 43-50). East Stroudsburg, PA: Association for Computational Linguistics.