The Design And Implementation Of Testing Service System Based On Software Behavior Analysis

As the scale and complexity of software are continuously growing, Baidu Inc. is facing with a series of problems on software development and testing, such as huge amounts of test cases, long cycles of testing and high costs of locating faults. It is difficult for testers to design test cases that cover all codes, branches and paths under test. Uncovered codes may cause serve errors that will make the online service crash and further more bring economic losses. Thus, diversified test cases are required to ensure the suffiency of testing. The sizes of test suites for some products in Baidu Inc. are so large that executing all the test cases in these suites would be unacceptable in industrial practice. On the other hand, it usually takes developers a lot of time to locate a fault, due to the scale and complexity of software and complex interactions between modules. Traditional software debugging techniques require a lot of manual efforts and have a huge negative impact on the efficiency of software development.To solve the problems above, we developed a testing service system based on software behavior analysis, to automate the process of selecting test cases and locating faults. Our system consists of three modules to automatically perform test case execution and information collection, test case selection and fault localization respectively. The automated test case execution and information collection module is the basis of the system, which is responsible for building software under test, executing test cases, collecting and processing execution information. The test case selection module selects a small subset of test cases which can present the diversity and integrity of the whole suite in a certain degree. Automated fault localization module uses graph mining and other fault localization algorithms to process the information of execution sequences, and reveal locations where errors occur as well as the context information.In this thesis, we develop the automated test case execution and information collection module in C and Python, the test cases selection module in Python, and the automated fault localization module in C++. The modules are designed in a loose coupling style and interact through files and databases. In this way, this system is more flexible and scalable. Now the system has been applied to the development and testing process of the Natural Language Processing group at Baidu. The test case selection module effectively imporove testing effectiveness by reducing the amounts of test cases without damaging the diversity and integrity of the test suites. The fault localization module helps developers quickly locate faults and improve development efficiency. In addition, the system has high scalability, which means that developers can add new test cases selection or fault localization algorithms as well as funtional modules to fulfill other software development tasks very flexibly.