| Crowdsourcing solicits an unknown group of people for solving human intrinsic problems that are inherently difficult for machines. Despite its great success in recent years, online crowdsourcing platforms face a biggest challenge over quality control of its problem solving process, which generally makes an open call" to the massive population. Workers lacking necessary expertise, having biased opinions, or driven by malicious incentives, may all contribute to the crowdsourced tasks, which in turn produce low-quality, erroneous results.To address this challenge, we derive a two-stage algorithm based on worker filtering and expectation maximization. In the first stage, the algorithm evaluates the quality of workers by the feedback of response time and answer conflict. Then it filters the spammers out based on the feedback ranking. In the second stage, we use the expectation maximization algorithm to iteratively estimate the quality of workers and true labels of tasks. A sampling-based algorithm is used to set the initial values for variables. This algorithm improves the state-of-the-art algorithms by many aspects, such as implicit feedback usage, spammers handling and initialization of estimation.Although many quality control techniques have been devised to obtain reliable results from workers’ answers, general evaluation criteria for quality control algorithms are still needed. To address this problem, we propose a generic evaluation framework for the algorithms. In this framework, we use both online and simulation experiments to evaluate the effectiveness and robust of the algorithm. Besides, we also point out that some factors in crowdsourcing platforms, such as task allocation, proportion of spammers, positional distribution of true labels, also have effect on algorithms.Finally, we implement the algorithm and several state-of-the-art algorithms on this framework. Experiments on both online and synthetic datasets have confirmed the proposed algorithm is effective and robust. |
PDF Full Text Request