Optimizing effectiveness and efficiency of software testing: A hybrid approach

The overall goal of software testing is to disclose defects efficiently (i.e. as little time and cost as possible) and effectively (i.e. find as many faults as possible). It takes time to understand what to test, to generate test cases, and to execute the test suite. It also takes time to analyze the test results. In a situation where one can parameterize inputs and variables of interest, the cost of generating random operational profile conformant test cases may be acceptable. It is typically O (Mxp) where p is the number of parameters and M is the number of test cases where a test case is a vector of p values. However, random testing can result in very large test suites that may take long to execute. Unless an automated oracle is available, results also may take long to analyze. In contrast, systematic approaches tend to generate smaller test suites, thus reducing run-time and analysis costs, but may take much longer to generate since they may require a higher-level of initial expertise to develop. There may also be differences in the ability of various methods to detect faults in the software. A question of interest is the following: Is there a way to take advantage of the benefits of both statistical and systematic approaches in a hybrid scheme to optimize both efficiency and effectiveness?; Hybrid approaches combine one or more testing techniques. This work discusses the basics of the underlying theory and presents the results of the associated experiments and simulations. The specific parameter-based systematic technique is called N-wise testing. Test case vectors are selected without replacement in N-wise testing. The N-wise technique assumes that most faults of interest will be found when all or most of the involved parameter N-tuple values are covered by the tests. If faults involve no more than N parameter values (called N-way faults), then all faults will be detected. In contrast, random testing is usually driven by operational profile considerations. Test cases are sampled with replacement, and coverage of N-way faults is random. Thus it is not guaranteed that all N-way faults will be detected using random testing. However, a complete N-wise test suite of M test cases guarantees that all N-way defects will be found. Given M random test cases and M N-wise test cases, effectiveness of N-wise testing may be larger than that of random testing. On the other hand, random testing may be more cost-efficient. The efficiency and effectiveness of an approach that combines random testing with N-wise testing is explored.; In the case where all defects are N-way but one could only afford to perform k-wise testing (k < N), a hybrid of k-wise systematic testing and random testing may help maximize efficiency and increase effectiveness beyond that offered by either technique on its own. This hybrid approach seems most effective when the minimum number of interacting parameters, N, required to expose a defect is large and the average number of values associated with the p parameters is also large. A potential use of hybrid testing is in testing for faults from failures that may result from a complex combination of interacting parametric values, such as those found in security failures, and in testing highly complex network-based systems and workflows in general.