Multi-factor Influencing Truth Inference In Crowdsourcing

Based on public wisdom,crowdsourcing has become an important field in computer applications,for it can effectively solve problems that are difficult to handle by computers.Existing crowdsourcing platforms such as AMT,publish these issues as microtasks to be addressed by the general public on the Internet.One of the key challenges of crowdsourcing is truth inference,which determines how to aggregate the answers of multiple workers to produce high-quality results.In the study of truth inference methods on crowdsourcing,the existing methods fail to comprehensively consider various influencing factors such as worker quality,similarity of candidate answers,task difficulty,and task domains.Therefore,there is a lack of a universal truth inference method with multiple influencing factors.In addition,when the number of answered tasks per worker is widely small,it is difficult to estimate worker quality well.Based on the research of existing truth inference methods,this paper proposes a truth inference method based on multiple influencing factors.The research work is mainly divided into the following parts:(1)We quantify task difficulty by analyzing the information provided by the task in depth.The difficulty of the task is determined by the similarity of candidate answers and the task-related domains,making the measurement of the task difficulty more accurate and objective.Among them,semantic similarity is considered in the calculation of the similarity of candidate answers.Finally,Entropy Weight method is used to calculate the weight of the diversity degree in the task domain and the similarity of candidate answers on the task difficulty.(2)We integrate multiple influencing factors to establish worker quality model and truth inference model.Firstly,worker quality of each worker is reasonably estimated by the task domain and task difficulty,and the worker quality model based on the domain-difficulty is established.Secondly,the relationship between multiple influencing factors and truth inference is established,and worker quality is dynamically updated once the truth is inferred.The comprehensive experimental results show that our truth inference method MFICrowd based on multiple influencing factors is effective,and it is better than the existing methods both in accuracy and time efficiency for the consideration of multiple influencing factors.(3)We optimize worker quality model to solve the problem of worker quality estimation when the number of answered tasks of each worker is generally small.Based on the research of different types of crowdsourcing tasks and long-tail phenomenon,two truth inference methods are proposed,which are suitable for different task scenarios.When the number of answered tasks per worker is small,the truth inference SCTI method based on small sample confidence interval is used for the truth inference for single selection tasks and numerical tasks with less task information;the optimization method SC-MFICrowd based on MFICrowd is used for truth inference for single selection tasks with rich task information.The comprehensive experimental results show that the above two truth inference methods can effectively solve the problem of long-tail phenomenon in different task scenarios,and the SC-MFICrowd method can take into consideration a variety of influencing factors.