Automated Test Update Prediction By Exploting Production-test Co-evolution

Bugs are prevalent in software and may cause huge financial losses,or even endanger human lives.Normally,software developers write test programs to ensure that the production programs run as expected.However,the test code can also be buggy,though it is often considered defectless both in industry practice and scientific research.We observe a type of buggy test code: the developers fail to timely update the test code when the production code under test is changed.We call such tests outdated tests.Intuitively,the test code and the production code under test should co-evolve with each other.Otherwise,the outdated tests may trigger spurious failures when exercising the new business logic,which would cause confusions to developers.Previous research [21]has pointed out that such co-evolution is time and resources consuming.The main reason is that developers often need to spend more time reading and understanding test code [8],compared with writing production code.Existing studies have not systematically addressed the outdated test problem.There are two common techniques for automatically debugging or repairing programs: fault localization(FL)and automated program repair(APR).However,applying these techniques to debug or repair outdated tests yields two difficulties.For FL,especially for spectrum-based FL techniques,the faulty element is discovered by running multiple test cases,with the assumption that faults reside in the execution traces of failing test runs but do not reside in the execution traces of passing test runs.It is useful for locating faults in production code but cannot help find defects in test code since a test case or test method is executed exactly once and the test code often does not contain branches(all code elements will appear in the traces of failing/passing test runs and thus cannot be differentiated).As for APR,there are no sufficient patch ingredients,such as well-defined mutation operators,to support patching test code.Existing APR techniques focus on repairing the production code.Some just repair the conditional statements [57],which rarely occur in test code.To bridge the gap,in this work,we aim to study the practices of test maintenance in real world and propose an approach to automatically detecting outdated tests.We first conducted a large-scale empirical study on 975 open-source Java projects to understand how production and test code coevolve.We found that only about 25% of the production code changes are immediately followed by the corresponding test code changes.Also,we found that “edit test code along with production code” is the most common type of co-evolution.Based on our empirical findings,we further proposed an approach for predicting the test change status.It involves code-analysis based feature engineering and classification-based change prediction.It is a scalable end-to-end approach,and thus can be applied on large-scale subjects.We evaluated our approach on the collected projects under different sample configurations and model parameters,and found that the proposed approach can pinpoint the tests that should co-evolve,with a satisfactory performance.Our experiments reveal that random forest is a good classifier to solve this problem,and method-level changes are useful to make the “test-update” decision.