GroExpert

a novel group-aware experts identification approach in crowdsourcing

Qianli Xing*, Weiliang Zhao, Jian Yang, Jia Wu, Qi Wang, Mei Wang

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference proceeding contribution

Abstract

Measuring workers’ abilities is a way to address the long standing problem of quality control in crowdsourcing. The approaches for measuring worker ability reported in recent work can be classified into two groups, i.e., upper bound-based approaches and lower bound-based approaches. Most of these works are based on two assumptions: (1) workers give their answers to a task independently and are not affected by other workers; (2) a worker’s ability for a task is a fixed value. However realistically, a worker’s ability should be evaluated as a relative value to those of others within a group. In this work, we propose an approach called GroExpert to identify experts based on their relative values in their working groups, which can be used as a basis for quality estimation in crowdsourcing. The proposed solution employs a fully connected neural network to implement the pairwise ranking method when identifying experts. Both workers’ features and groups’ features are considered in GroExpert. We conduct a set of experiments on three real-world datasets from the Amazon Mechanical Turk platform. The experimental results show that the proposed GroExpert approach outperforms the state-of-the-art in worker ability measurement.

Original languageEnglish
Title of host publicationProceedings of 20th International Conference of Web Information Systems Engineering - WISE 2019
EditorsReynold Cheng, Nikos Mamoulis, Yizhou Sun, Xin Huang
Place of PublicationCham, Switzerland
PublisherSpringer
Pages3-17
Number of pages15
ISBN (Electronic)9783030342234
ISBN (Print)9783030342227
DOIs
Publication statusPublished - 2019
Event20th International Conference on Web Information Systems Engineering, WISE 2019 - Hongkong, Hong Kong
Duration: 26 Nov 201930 Nov 2019

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11881 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference20th International Conference on Web Information Systems Engineering, WISE 2019
CountryHong Kong
CityHongkong
Period26/11/1930/11/19

Keywords

  • Crowdsourcing
  • Group-aware
  • Worker ability

Fingerprint Dive into the research topics of 'GroExpert: a novel group-aware experts identification approach in crowdsourcing'. Together they form a unique fingerprint.

  • Cite this

    Xing, Q., Zhao, W., Yang, J., Wu, J., Wang, Q., & Wang, M. (2019). GroExpert: a novel group-aware experts identification approach in crowdsourcing. In R. Cheng, N. Mamoulis, Y. Sun, & X. Huang (Eds.), Proceedings of 20th International Conference of Web Information Systems Engineering - WISE 2019 (pp. 3-17). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11881 LNCS). Cham, Switzerland: Springer. https://doi.org/10.1007/978-3-030-34223-4_1