Software Testing-effort Allocation Methods For Finding Faults:An Experimental Comparison

It is a challenge for software development organizations to allocate test effort to software modules for finding defects effectively.Structural-complexity-guided allocation(SCA)methods and fault-prediction-guided allocation(FPA)methods have been proposed recently.The former leverages module size to allocate test effort and the latter leverages fault prediction models to allocate test effort.However,existing SCA methods only take into account module size,while existing FPA methods require that fault prediction must achieve a high prediction accuracy.In particular,the effectiveness of existing SCA and FPA methods are only evaluated at the "functional" granularity(one function is implemented by multiple modules).Consequently,it is unclear whether existing SCA and FPA methods are effectiveness at the module granularity(e.g.the class granularity).In order to address these problems,this paper investigates the effectiveness of test effort allocation methods at the module granularity.In summary,this paper makes the following contributions:(1)Extend SCA methods to take into account cyclomatic complexity:existing SCA methods lack diversity in structural complexity metrics,which only takes the sim-plest metric(module size)into consideration for test effort allocation.In order to con-duct a comprehensive investigation into SCA methods,we take into account not only code change metrics but also more complexity metrics(e.g.cyclomatic complexity).Our experimental results finally show that some code change metrics based SCA meth-ods are simple but effective and stable.In contrast,cyclomatic complexity may not be a good option although many previous studies have shown that it could be very useful in software fault prediction.(2)Validate the effectiveness of FPA under new conditions:existing FPA methods are evaluated under the condition that fault prediction is at "function" granularity.Fur-thermore,previous studies report that existing FPA methods are effective only when fault prediction has a high accuracy(Norm(Popt)>0.78).However,in practice,fault pre-diction is usually conducted at "module" granularity such as class or file.In particular,many studies have shown that it is difficult to achieve such a high prediction accuracy at "module" granularity.We investigate the effectiveness of existing FPA methods at"module" granularity in this paper and the experimental results show that FPA methods do not perform well.(3)Propose a new type of RGA methods:we apply software reliability growth model to software test effort allocation and the resulting methods are named as RGA(reliability-growth-model-guided allocation)methods.For a target release of a project,RGA methods first estimate fault distribution in this release by the fault distribution information in previous releases.Then,given the total test allocation,RGA methods generate the optimal test effort allocation scheme in theory.The experimental results show that RGA methods perform better than both SCA and FPA methods.On the one hand,given the same total test effort,RGA methods could discover more faults.On the other hand,RGA methods are more stable over releases.