From appearance to essence

comparing truth discovery methods without using ground truth

Xiu Susie Fang, Quan Z. Sheng, Xianzhi Wang, Wei Emma Zhang, Anne H. H. Ngu, Jian Yang

Research output: Contribution to journalArticle

Abstract

Truth discovery has been widely studied in recent years as a fundamental means for resolving the conflicts in multi-source data. Although many truth discovery methods have been proposed based on different considerations and intuitions, investigations show that no single method consistently outperforms the others. To select the right truth discovery method for a specific application scenario, it becomes essential to evaluate and compare the performance of different methods. A drawback of current research efforts is that they commonly assume the availability of certain ground truth for the evaluation of methods. However, the ground truth may be very limited or even impossible to obtain, rendering the evaluation biased. In this article, we present CompTruthHyp, a generic approach for comparing the performance of truth discovery methods without using ground truth. In particular, our approach calculates the probability of observations in a dataset based on the output of different methods. The probability is then ranked to reflect the performance of these methods. We review and compare 12 representative truth discovery methods and consider both single-valued and multi-valued objects. The empirical studies on both real-world and synthetic datasets demonstrate the effectiveness of our approach for comparing truth discovery methods.

Original languageEnglish
Article number74
Pages (from-to)1-24
Number of pages24
JournalACM Transactions on Intelligent Systems and Technology
Volume11
Issue number6
DOIs
Publication statusPublished - Sep 2020

Keywords

  • multi-valued objects
  • performance evaluation
  • single-valued objects
  • sparse ground truth
  • truth discovery methods
  • Web search

Fingerprint Dive into the research topics of 'From appearance to essence: comparing truth discovery methods without using ground truth'. Together they form a unique fingerprint.

Cite this