Evaluating human pairwise preference judgments

Mark Dras*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

7 Citations (Scopus)
6 Downloads (Pure)


Human evaluation plays an important role in NLP, often in the form of preference judgments. Although there has been some use of classical non-parametric and bespoke approaches to evaluating these sorts of judgments, there is an entire body of work on this in the context of sensory discrimination testing and the human judgments that are central to it, backed by rigorous statistical theory and freely available software, that NLP can draw on. We investigate one approach, Log-Linear Bradley-Terry models, and apply it to sample NLP data.

Original languageEnglish
Pages (from-to)337-345
Number of pages9
JournalComputational Linguistics
Issue number2
Publication statusPublished - 19 Jun 2015

Bibliographical note

Copyright the Publisher 2015. Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.


Dive into the research topics of 'Evaluating human pairwise preference judgments'. Together they form a unique fingerprint.

Cite this