Research On Test Suite Optimization Based On Attribute Analysis

Software testing is one of the most important ways to assure software quality.Software testing includes designing test cases,running test cases and verifying run-ning results.In order to decrease the cost of software testing and increase the efficien-cy of testing,a lot of automatic testing approaches including automatic test generation,test running and test results verification were proposed.Automatic test generation techniques often generate large amount of redundant test inputs.The large amount of inputs makes inspecting and analyzing test cases very costly.The aim of test suite opti-mization techniques is to reduce the amount of test cases that need to be run,inspected and analyzed.In this thesis,we studied two problems of test suite optimization techniques.When the standard outputs can not be collected and compared automatically,test suit-e optimization techniques based on program behavior clustering instrument programs and collect program behavior information.Then the program behavior information are clustered to optimize test suite and reduce the cost of verifying running results man-ually.The problem of such methods is that they treat all program elements equally lead to inappropriate clustering results.Another problem is that the program behavior information is not always available because programs cannot be instrumented.In the case study in Huawei we found that in the consideration of safety,testers cannot in-strument and collect program behaviors.In this case test suite optimization techniques can only be performed by analyzing inputs and the successful and failed information of test running.In such cases test suite optimization techniques based on program be-havior coverage cannot be used because they need to instrument programs and collect program running behaviors.In this thesis we propose corresponding methods to solve the above two problems.In order to overcome the problem of existing test suite optimization techniques based on program behavior clustering.we propose a weighted attributes based s-trategy(WAS).In WAS fault localization techniques use test cases execution pro-files and the information of successful and failed execution to calculate suspicious values for each program elements.WAS uses the suspicious values to weight ini-tial execution profiles and then the weighted profiles will be used for further clus-tering.In this thesis the most popular used and studied spectrum based fault local-ization techniques Crosstab,Jaccard,Ochiai,Tarantula,H3c and H3b are used for weighting execution profiles.We conducted experiments on 7 open source programs(make,ant,SED,flex,grep,gzip,and space).4 cluster filtering techniques are used for comparison(one per cluster n per cluster,adaptive sampling and ESBS).184 single fault and multiple fault version programs are generated for the experiment.The experimental results show that the proposed WAS has a better performance than the other 4 filtering techniques in Recall and Precision.In order to perform test suite optimization when the program cannot be instru-mented we have made an industrial case study in HUAWEI and propose CSBART(Category Selection Based Adaptive Random Testing)method.CSBART introduces the category selection method to find the categories that most related to failed exe-cutions to improve the efficiency of ability of LART(Linear-Order Algorithm for Adaptive Random Testing,LART)for selecting test cases that can reveal more faults.CSBART uses two category selection methods.The first method is Input Profile(IP).The intuition of IP is that if a choice occurs more often in failure test cases,the choice is a failure related choice.IP calculates the frequency of the choice occurring in failed test cases to identify categories that closed related to failed test cases.The second method is category selection method based on Mutual Information(MI).MI calculates the amount of information of the failed or successful executions that different cate-gories contains.The larger the amount of information,the more closely the category is related to the success or failure of execution.In this thesis the effectiveness of CS-BART is studied in the industry environment in Huawei.CSBART is compared with Random Testing,LART and methods based on clustering n per cluster sampling and adaptive sampling.The experiments results show that CSBART has a higher ability of selecting test cases that can reveal faults.In this thesis,these two methods are collectively called attribute analysis based test suite optimization methods.WAS analyzes the different attributes(statements)of program execution behaviors by using weighting technology to optimize the test suite.CSBART analyzes the attributes of test inputs(category)to optimize the test suite.In the last part of this thesis a prototype framework is proposed based on the two methods.A prototype tool was implemented in the case study in Huawei and evaluated by the professional testers.