Automated Coverage Criteria-based Test Data Generation: Approaches And Implementations

Software testing is the most widely used software validation technique in practice, which aims to ensure software quality. It consists of three basic steps:(1) design effective test suites for the system under test; (2) execute these test suites; (3) compare the actual execution results with the expected results to do confirmation. In particular, test case design requires great human efforts, since testers have to be familiar with the specification, and even the actual source code. In order to fully test the system under test, ideally, the testers have to find all combinations of program input values, so that all parts of the system can be tested. However, it is very difficult to achieve in practice. As a more generic approach, test cases are selected according to some coverage criteria. The advantages of this approach lies in three aspects:(1) The coverage criteria can determine the way of selecting test cases, and guide testers to design effective ones. (2) Different coverage criteria can be adopted to meet different testing requirements, e.g., more advanced test cover-age criteria can enhance the effectiveness of test suites. (3) The coverage value can provide software developers, testers, and project managers with a referencable and traceable software quality metric, which reflects the degree of software reliability.However, in the face of coverage-based test generation, the existing methods still faces great challenges and more research efforts should be endeavored in this direction:(1) The existing methods cannot achieve the target coverage criteria quickly and efficiently, and usually only a specific target criterion is taken into consideration; (2) Most of existing methods focus on the most commonly used coverage criteria, e.g., statement or branch coverage. Little efforts has been put in more advanced criteria, e.g., data flow coverage and basic path coverage. Without effective testing methods, it is difficult for them to show their great potentials in detecting software bugs; (3) The existing testing methods mainly focus on CLI (Command Line Interface) software, and take their source code as input. But for more popular GUI (Graphical User Interface) software, few effective methods exists to achieve coverage-based test generation.To this end, this paper focuses on the problem of coverage-based test gexiera- tion, and investigates this topic from the following three aspects:1. Control flow coverage-based test generation. We focus on control flow coverage criteria, such as statement, branch, and MC/DC coverage, and build a unified testing framework. The framework adapts the dynamic symbolic execution technique to achieve efficient coverage-driven testing.2. Data flow coverage-based test generation. We propose a hybrid ap-proach to generate tests for data flow coverage, which combines symbolic execution and model checking, and shares their respective advantages. We provide an automated and efficient data flow testing framework, which out-puts tests for feasible test objectives, and also detects infeasible ones.3. Hybrid coverage-based test generation for GUI software. We focus on mobile apps (they are typical GUI software), and intend to enforce hybrid coverage criteria from the abstract model level and the actual implementation level. The test suites are iteratively generated and optimized to maximize the coverage values.In all, we believe these proposed automated testing methods can further improve the automation degree of test data generation, and benefit the early detection of software defects.