Research On Data Quality Control Of Peer Grading In Crowdsourcing System

Crowdsourcing that uses the power and wisdom of the crowds to solve complex problems has witnessed great development recently.However,the uneven quality of data provided by large-scale participants makes quality control a huge challenge for crowdsourcing platforms.As a special application of crowdsourcing technology,peer grading/assessment can greatly improve the quality of participants' submissions and is an important way of quality control.The participants of peer grading often leads to inaccurate or random grades due to collusion,laziness,retaliation,dishonesty,and lack of experience,interest or time.Using MOOCs as a typical application environment,this thesis proposed and implemented peer grading schemes(data quality control method based on peer grading)that are suitable for more general tasks such as article writing or graphic design,and compared with some existing peer grding schemes.The main contributions of the thesis are as follows :(1)Firstly,proposed a peer grading schemes with teacher assistance,RankwithTA.Unlike the simple aggregation methods,RankwithTA performs a series of iterations,each iteration consisting of two update steps:(i)updating the reliability/accuracy of peer evaluator according to the difference between the grades given by the evaluator and the aggregated grades obtained by students;(ii)the quality of submission of a evaluator(i.e.,the grades of the evaluators)is updated in a weighted manner based on the evaluator's reliability.In addition,RankwithTA also uses external calibration,which is used to provide a basis for the accuracy of the gradings by allowing the teacher or TAs to give a part of the students' correct scores.The simulation results illustrate that RankwithTA performs better than the benchmark schemes.(2)RankwithTA method uses the grades received by participants to measure their grading ability which is not feasible in practice.To solove the problem,a semi-supervised peer assessment scheme(SSPA)was proposed.Semi-supervised means that the scheme incorporates a small number of teacher's gradings as ground truth,and uses them to externally calibrate the procedure of aggregating peer grades.SSPA explicitly distinguishes the participant's professional ability of soloving tasks and his ability to grading others.Then SSPA utilizes the weighted aggregation of peer grades to infer the final score of each student,in which grader students' evaluations on gradees are weighted with their grading abilities.The simulation results show that the performance of the SSPA scheme is better than the benchmark schemes and has also been improved compared to the RankwithTA scheme.(3)An effective aggregation algorithm can eliminate the unintended grading deviation in peer assessment,but there are still malicious and random attacks.According to theoretical analysis,appropriate incentives can effectively encourage participants to grade seriously and truthfully.Based on an important scenario of peer grading : according to students' mutual evaluations,a subset of students is selected to grant award.This thesis proposed a Truthful grading based Strategyproof Peer selection scheme for MOOCs,TSP.The scheme incorporates incentive mechanism to encourage students to strive to grading and report the grades truly,and to solve the problem that peers are not willing to invest effort in peer grading.The theoretical analysis and simulation results show that TSP can both stimulate students to truthfully report their gradings,and meanwhile select the best students in strategyproof way.