Incentive Mechanism Improvement With Approval Voting In Crowdsourcing

As the demand of labeled data continues to grow in machine learning field,many researchers collect labeled data by crowdsourcing systems to save costs.However,the total quality of workers' answers is usually out of satisfaction due to the lack of expertise knowledge and personal motivation.To improve the quality of workers' answers,it is essential to design proper incentive mechanisms.In previous studies,researches mainly focus on the effects of workers' motivation and thus ignore the effects of the single-selection mechanism.Therefore,this thesis studies how to design incentive mechanisms with consideration of both goals.More concretely,this thesis focuses on designing incentive mechanisms in two typical cases with approval voting.Firstly,this thesis considers the design of incentive mechanisms with the existence of gold standard questions whose true answers are known apriori.Under the circumstances where questions in labeling tasks have latent true answers,task publishers are able to insert gold standard questions to evaluate the quality of workers' answers.To improve the quality,approval voting is applied to utilize workers' partial knowledge.Given a threshold,workers demand choosing one or more options with belief higher than the threshold for each question.Then this thesis proposes an incentive mechanism where workers should pay a fixed cost for every option selected,and gain the bonus only if the true answer is selected for each gold standard question.This thesis theoretically proves that it meets the constraint of improving the total quality of workers' answers.Then this thesis considers about the incentive mechanisms in peer prediction.Under the circumstances,it is difficult to design proper gold standard questions due to the subjectivity of questions in labeling tasks.To design proper incentive mechanisms,this thesis takes the correlation of options into consideration and proposes two mechanisms in different environments.In the first mechanism,workers pay a cost proportional to the posterior probability for every option selected,and gain the bonus only if the answers match the reference answers.In the second mechanism,workers increase payments for the agreement in each question and decrease payments for the agreement in irrelevant questions.This thesis theoretically proves that both mechanisms are able to incentivize workers to elicit truthful answers.Finally,this thesis proposes a modified aggregation method which utilizes the weighted frequencies as features to perform clustering algorithm and infer aggregated labels.Experiment results show that this method further improves the accuracy of labeling tasks.