Document distance for the automated expansion of relevance judgements for information retrieval evaluation

Diego Mollá; Iman Amini; David Martinez

Document distance for the automated expansion of relevance judgements for information retrieval evaluation

Diego Mollá, Iman Amini, David Martinez

School of Computing

Research output: Chapter in Book/Report/Conference proceeding › Conference proceeding contribution › peer-review

Abstract

This paper reports the use of a document distance-based approach to automatically expand the number of available relevance judgements when these are limited and reduced to only positive judgements. This may happen, for example, when the only available judgements are extracted from a list of references in a published review paper. We compare the results on two document sets: OHSUMED, based on medical research publications, and TREC-8, based on news feeds. We show that evaluations based on these expanded relevance judgements are more reliable than those using only the initially available judgements, especially when the number of available judgements is very limited.

Original language	English
Title of host publication	Proceedings of the ACM SIGIR Workshop on Gathering Efficient Assessments of Relevance (GEAR)
Place of Publication	United States
Publisher	Cornell University Library
Pages	1-4
Number of pages	4
Publication status	Published - 2014
Event	ACM SIGIR 2014 Workshop on Gathering Efficient Assessments of Relevance - Gold Coast, Australia Duration: 11 Jul 2014 → 11 Jul 2014

Workshop

Workshop	ACM SIGIR 2014 Workshop on Gathering Efficient Assessments of Relevance
City	Gold Coast, Australia
Period	11/07/14 → 11/07/14

Keywords

Information Retrieval
Evaluation
Relevance Judgements Expansion

Access to Document

http://arxiv.org/abs/1501.06380

Cite this

@inproceedings{6d2a515e2240486db797ff5cb2ac134e,

title = "Document distance for the automated expansion of relevance judgements for information retrieval evaluation",

abstract = "This paper reports the use of a document distance-based approach to automatically expand the number of available relevance judgements when these are limited and reduced to only positive judgements. This may happen, for example, when the only available judgements are extracted from a list of references in a published review paper. We compare the results on two document sets: OHSUMED, based on medical research publications, and TREC-8, based on news feeds. We show that evaluations based on these expanded relevance judgements are more reliable than those using only the initially available judgements, especially when the number of available judgements is very limited.",

keywords = "Information Retrieval, Evaluation, Relevance Judgements Expansion",

author = "Diego Moll{\'a} and Iman Amini and David Martinez",

year = "2014",

language = "English",

pages = "1--4",

booktitle = "Proceedings of the ACM SIGIR Workshop on Gathering Efficient Assessments of Relevance (GEAR)",

publisher = "Cornell University Library",

note = "ACM SIGIR 2014 Workshop on Gathering Efficient Assessments of Relevance ; Conference date: 11-07-2014 Through 11-07-2014",

}

Mollá, D, Amini, I & Martinez, D 2014, Document distance for the automated expansion of relevance judgements for information retrieval evaluation. in Proceedings of the ACM SIGIR Workshop on Gathering Efficient Assessments of Relevance (GEAR). Cornell University Library, United States, pp. 1-4, ACM SIGIR 2014 Workshop on Gathering Efficient Assessments of Relevance, Gold Coast, Australia, 11/07/14. <http://arxiv.org/abs/1501.06380>

Document distance for the automated expansion of relevance judgements for information retrieval evaluation. / Mollá, Diego; Amini, Iman; Martinez, David.
Proceedings of the ACM SIGIR Workshop on Gathering Efficient Assessments of Relevance (GEAR). United States: Cornell University Library, 2014. p. 1-4.

Research output: Chapter in Book/Report/Conference proceeding › Conference proceeding contribution › peer-review

TY - GEN

T1 - Document distance for the automated expansion of relevance judgements for information retrieval evaluation

AU - Mollá, Diego

AU - Amini, Iman

AU - Martinez, David

PY - 2014

Y1 - 2014

N2 - This paper reports the use of a document distance-based approach to automatically expand the number of available relevance judgements when these are limited and reduced to only positive judgements. This may happen, for example, when the only available judgements are extracted from a list of references in a published review paper. We compare the results on two document sets: OHSUMED, based on medical research publications, and TREC-8, based on news feeds. We show that evaluations based on these expanded relevance judgements are more reliable than those using only the initially available judgements, especially when the number of available judgements is very limited.

AB - This paper reports the use of a document distance-based approach to automatically expand the number of available relevance judgements when these are limited and reduced to only positive judgements. This may happen, for example, when the only available judgements are extracted from a list of references in a published review paper. We compare the results on two document sets: OHSUMED, based on medical research publications, and TREC-8, based on news feeds. We show that evaluations based on these expanded relevance judgements are more reliable than those using only the initially available judgements, especially when the number of available judgements is very limited.

KW - Information Retrieval

KW - Evaluation

KW - Relevance Judgements Expansion

M3 - Conference proceeding contribution

SP - 1

EP - 4

BT - Proceedings of the ACM SIGIR Workshop on Gathering Efficient Assessments of Relevance (GEAR)

PB - Cornell University Library

CY - United States

T2 - ACM SIGIR 2014 Workshop on Gathering Efficient Assessments of Relevance

Y2 - 11 July 2014 through 11 July 2014

ER -

Document distance for the automated expansion of relevance judgements for information retrieval evaluation

Abstract

Workshop

Keywords

Access to Document

Fingerprint

Cite this