Constraints In Combinatorial Testing

Software is becoming more and more complicated,which makes testing a great challenge.Combinatorial testing is an efficient software testing technology which can detect software failures triggered by the value combinations of parameters.But when it is applied in real-world systems,constraints have great influence on it.Handling constraints well is the key to applying combinatorial testing.However,the study of constraints is far from enough.We do some work as follows:First,we discuss the influence of constraints on combinatorial testing and investigate constraints in papers and real-world systems.After that,we classify constraints according to their forms.We find these constraints are very common in systems and they can be classified into 4 groups:forbidden constraints,requiring constraints,numeric constraints and shielding constraints.We discuss the method to transform different kinds of constraints into each other,especially the transformation of shielding constraints.Namely,we unify the way to handle constraints in combinatorial testing.Second,we classify existing constraints handling strategies into handling before generation,handling during generation and handling after generation according to handling time and discuss them in details.In addition,we propose two constraints handling algorithms:test case modifying algorithm and test suite post-processing algorithm.Experimental result shows that constraints avoiding method is better both in covering array size and generation time.Third,in order to make it easier for testers to use combinatorial testing in practice,we develop a testing tool named CTCH(Combinatorial Testing with Constraint Handling)Which supports constraints handling.The tool adopts one-test-at-a-time generating strategy and uses constraints avoiding strategy to deal with constraints.Meanwhile it uses a simple test suite optimizing strategy Which makes use of flexible positions.Later,we survey existing tools that can deal with constraints and find that almost all tools use constraints avoiding strategy.Besides,we compare CTCH with popular combinatorial testing tools.The result indicates that CTCH is not only easy to use but also competitive.Finally,we use combinatorial testing insystems with plenty of constraints.We find out a lot of software failures in that system,which indicates that combinatorial testing is also effective when the system contains many constraints.