GroExpert: a novel group-aware experts identification approach in crowdsourcing

Measuring workers’ abilities is a way to address the long standing problem of quality control in crowdsourcing. The approaches for measuring worker ability reported in recent work can be classified into two groups, i.e., upper bound-based approaches and lower bound-based approaches. Most of these works are based on two assumptions: (1) workers give their answers to a task independently and are not affected by other workers; (2) a worker’s ability for a task is a fixed value. However realistically, a worker’s ability should be evaluated as a relative value to those of others within a group. In this work, we propose an approach called GroExpert to identify experts based on their relative values in their working groups, which can be used as a basis for quality estimation in crowdsourcing. The proposed solution employs a fully connected neural network to implement the pairwise ranking method when identifying experts. Both workers’ features and groups’ features are considered in GroExpert. We conduct a set of experiments on three real-world datasets from the Amazon Mechanical Turk platform. The experimental results show that the proposed GroExpert approach outperforms the state-of-the-art in worker ability measurement.