A Method To Estimate Coincidnetal Correctness Probablity And Its Application In Software Testing

Testing is an important method to ensure the quality and reliability of software products.Code-coverage-based testing technologies are widely studied and used in the software development process.Coincidental correctness is the phenomenon that a fault is executed but no failure is detected.Research studies show that coincidental correctness is prevalent and a safety reducing factor for the coverage-based testing techniques.This phenomenon impacts the evaluation of test adequacy,and other test works like fault localization and test suite selection.In this thesis,a method based on the static and dynamic program analysis is proposed to estimate the probability that coincidental correctness(CCP for abbreviation)occurs during a test execution of the program.Based on the CCP estimation method,testing techniques including testing adequacy criterion,fault localization and test selection technique are studied in this thesis.The main works of this thesis includes the following four parts:1)A method is designed to estimate the probability that an error in the program is hidden because of coincidental correctness phenomenon.A program-under-test is first instrumented based on static code analysis to record runtime information about control-flow and data-flow of test executions.Based on these information,the method analyzes how the run-time data are generated,operated,and finally output to be checked by testers.During a test execution,different kinds of operations have different probabilities to produce correct results using wrong inputs,thus covers intermediate failures.For each statement,this method estimates the probability that if the statement contains a fault,the wrong immediate results generated by this statement is covered,instead of being transferred to the output,by the subsequent executions of operations.Comparing to the techniques estimating coincidental correctness based on similarity of test behaviors,this method gives more accurate estimation results.2)A test adequacy criterion based on CCP estimation is proposed.Intuitively speaking,a higher CCP of a statement means the statement is less tested,i.e.the potential error in this statement is more likely to be hidden.The proposed criterion requires that for each statement,the CCP of a statement must under a given number.After a program-under-test passes a test suite,the CCP of each statement is estimated and synthesized.This synthesized CCP reflects the adequacy that the statement is tested by the test suite.The empirical results shows that the test adequacy criterion based on CPP can more accurately quantizes the capability of test suite to find errors.3)A fault-localization technique based on the CCP is proposed.This method adjusts the formula to calculate suspiciousness values for statements using CCP estimations,thus alleviates the impact of coincidental correctness phenomenon.The empirical results show that our approach can improve the safety and precision of the fault localization technique to a certain degree.4)A regression test selection method using CCP estimation is proposed.This method considers the probability of coincidental correctness in addition to the code coverage.The selected tests not only cover the modified code,but have higher probability to transfer the intermediate results produced by the modified code to the program output.Such selection can reduce the impact of coincidental correctness.The empirical results shows that the test selection technique based CCP can improve the precision of selection and reduce size of the regression tests.