Research On The Bias In The Evaluation Of Automated Program Repair

Automated Program Repair techniques bring great convenience to tasks such as software debugging and software maintenance,which has aroused widespread concern of researchers.More and more approaches have been proposed to better solve this problem.However,there are many biases in the evaluation of these techniques,which makes it impossible for researchers to comprehensively and objectively analyze the repair capabilities of current techniques.This paper studies the three types of biases in the current repair tool evaluation process,whose contents cover two of the three links in the repair pipeline and the selection of the defect dataset,aiming to improve the cognitive level of the repair community and provide baselines for subsequent research.First of all,this paper selects the generated correct patches for analysis,summarizes the experience for reference and proposes a set of patch correctness assessment rules to solve the problem of patch correctness assessment.Secondly,this paper analyzes the repair capabilities of repair tools under different fault localization through a large-scale experiment,aiming at studying the effect of fault localization noise on program repair.Finally,this paper compares the repair capabilities of repair tools under different datasets from the perspective of single dataset and multiple datasets,aim to study the external validity threats faced by the automated program repair tools.This paper draws some conclusions that are of great value to the program repair community:(1)program repair techniques tend to generate patches that are different from the ground-truth for bugs with large patch sizes;(2)the accuracy of fault localization have a great impact on repair techniques from repair effectiveness and repair efficiency;(3)there is over-fitting between the existing tools and the selected test dataset.Based on these conclusions,this paper proposes a set of patch correctness assessment rules and two suggestions: the evaluation of the program repair techniques should be carried out under the same condition of defect localization and should consider more reasonable defect datasets,making an important contribution to achieve unbiased repair technique assessment.