Using Effort-aware Performance Indicators To Compare The Ability Of Code Metrics, Process Metrics, And Historical Fault Metrics To Predict Fault-proneness

The prediction of fault-prone modules enables software project managers to know the distribution of faults in a software system. This information is very helpful for them to effectively allocate limited resources and improve the efficiency of activities in software testing and maintenance. In the literatures, software metrics are the most commonly used information to build fault-proneness prediction models. The existing software metrics can be roughly classified into the following three categories:code metrics, process metrics, and historical fault metrics. In this paper, based on five open-source object-oriented systems, we use effort-aware performance indictors to analyze and compare the ability of these three categories of metrics to predict fault-proneness.For each category of metrics, we collected as many metrics as possible. In order to make the comparison objectively, process metrics and historical faults metrics are collected on the data recorded within six months before the release time of each system. For actual faults in each system, we only collected faults which have been found and fixed within six months after release. We evaluated the ability of code metrics, process metrics, and historical fault metrics to predict fault-proneness under the following two typical scenarios:ranking and classification. In the ranking scenario, modules are ranked by their predicted fault-proneness. In the classification scenario, modules are classified into two categories according to their predicted fault-proneness: fault-prone and not fault-prone. In both scenarios, we take into account the effort to inspect/test classes when evaluating the prediction performance of a fault-proneness prediction model. The experimental results, from five open-source systems, show that:(1) in most systems, code metrics and process metrics have a good prediction performance, while historical fault metrics have a poor prediction performance;(2) there is no conclusion on which category of metrics has a better prediction performance and this indeed depends on the software systems used;(3) the combination of code metrics, process metrics, and historical fault metrics produces a better prediction performance than any individual category of metrics; and (4) when used for cross-version prediction, the combination of code metrics, process metrics and historical fault metrics is still better than any individual category of metrics but the prediction performance degrades considerably.In order to make the above-mentioned conclusions reliable, we analyzed the influence of the modeling techniques, feature selection methods, and imbalanced data distribution on the prediction performance of fault-proneness prediction models. This thesis provides an in-depth understanding on the fault-proneness prediction ability of software metrics in the context of effort-aware evaluation and hence is a beneficial complementary to existing work.