Localizing Relevant Source Code Files For Bug Reports

During software development,developers must spend considerable time and effort to find the cause when a new bug report is received.To reduce the manual effort needed for localizing relevant source code files,researchers have proposed a large number of information retrieval(IR)based or machine learning(ML)based methods to automate the process.However,we find that the metrics they used for evaluation are all "list-position" related.In other words,their evaluation did not take into account the influence of the size of recommended files on the effectiveness in detecting bugs.In practice,developers will generally spend more code inspection effort to detect bugs if larger files are recommended.In this paper,we implement four typical bug localization methods,including two state-of-the-art methods(i.e.,LR and Buglocator)as well as two baselines(i.e.,VSM and US).In particular,we propose six effort-aware metrics to evaluate the effectiveness of the four methods.Based on six large-scale open-source Java projects,our experimental results show that:(1)in most systems,if top-k files are recommended,the LR method needs more inspection effort to localize relevant files for bug reports than BugLocator or VSM,meanwhile the LR method has a lower effort-precision value;(2)in most systems,no matter whether to localize all relevant files or only the first relevant file for bug reports,the LR method needs more inspection effort than the BugLocator and VSM methods,and it does not perform well enough in terms of effort-precision;(3)in the context of effort-aware bug localization,complex methods such as LR and BugLocator usually do not outperform the simple ones like VSM.As previous study has shown that the LR method outperforms BugLocator,VSM,and US under the evaluation of "list-position" related metrics,our experimental results suggest that traditional "list-position" related metrics cannot indicate the practical performance of IR-based or ML-based bug localization methods.Code inspection effort should be taken into account when evaluating those bug localization methods.