Validating the web-based evaluation of NLG systems

Alexander Koller*, Kristina Striegnitz, Donna Byron, Justine Cassell, Robert Dale, Sara Dalzel-Job, Jon Oberlander, Johanna Moore

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference proceeding contribution

7 Citations (Scopus)

Abstract

The GIVE Challenge is a recent shared task in which NLG systems are evaluated over the Internet. In this paper, we validate this novel NLG evaluation methodology by comparing the Internet-based results with results we collected in a lab experiment. We find that the results delivered by both methods are consistent, but the Internetbased approach offers the statistical power necessary for more fine-grained evaluations and is cheaper to carry out.

Original languageEnglish
Title of host publicationJoint Conference of the 47th Annual Meeting of the Association for Computational Linguistics and 4th International Joint Conference on Natural Language Processing of the AFNLP, ACL-IJCNLP 2009
Place of PublicationStroudsburg, PA
PublisherAssociation for Computational Linguistics (ACL)
Pages301-304
Number of pages4
ISBN (Print)9781617382581
Publication statusPublished - 2009
EventJoint Conference of the 47th Annual Meeting of the Association for Computational Linguistics and 4th International Joint Conference on Natural Language Processing of the AFNLP, ACL-IJCNLP 2009 - Suntec, Singapore
Duration: 2 Aug 20097 Aug 2009

Other

OtherJoint Conference of the 47th Annual Meeting of the Association for Computational Linguistics and 4th International Joint Conference on Natural Language Processing of the AFNLP, ACL-IJCNLP 2009
CountrySingapore
CitySuntec
Period2/08/097/08/09

Fingerprint Dive into the research topics of 'Validating the web-based evaluation of NLG systems'. Together they form a unique fingerprint.

  • Cite this

    Koller, A., Striegnitz, K., Byron, D., Cassell, J., Dale, R., Dalzel-Job, S., ... Moore, J. (2009). Validating the web-based evaluation of NLG systems. In Joint Conference of the 47th Annual Meeting of the Association for Computational Linguistics and 4th International Joint Conference on Natural Language Processing of the AFNLP, ACL-IJCNLP 2009 (pp. 301-304). Stroudsburg, PA: Association for Computational Linguistics (ACL).