An Empirical Study Of Combinatorial Test Case Prioritization Technology

Software testing is an important way to ensure the quality of software,which can help to assess the robustness of the software and detect vulnerabilities as early as possible.As an effective software testing technology,combinatorial testing usually can detect software faults triggered by interactions between input parameters.Combinatorial test case prioritization is a technology,which prioritizes test cases in an existing combinatorial test suite to form an ordered test case sequence.This paper conducts an empirical study on the combinatorial test case prioritization technology.Now the robustness of the software based on deep neural networks(DNNs)in safety-critical areas has attracted widespread attention and due to lack of suitable structure,these softwares cannot directly adopt test adequacy criterion of traditional software.This paper regards combinatorial dense coverage for DNNs as coverage criterion to rank test cases by descending priority on DNNs testing and studies adversarial detection of test case sequence.This paper?s main research work are as follows:(1)Adopting combinatorial dense coverage criteria,MNIST dataset,three models and four sets of experimental examples to count the peak coverage and the corresponding test suite scale.This paper also researches the correlation between combinatorial coverage and adversarial detection.After lots of experiments,the results show that: There is correlation between combinatorial dense coverage and adversarial detection.(2)Selecting test cases by descending priority from original test suite with combinatorial dense coverage criteria and comparing the adversarial detection of combinatorial test case prioritization technology and random testing under the same test suite is implemented.The experimental results show that: Guiding test case sorting with combinatorial dense coverage criteria cannot improve adversarial detection.(3)Analyzing characteristics of the existing combinatorial test case generation tools and combined with the "combinatorial testing method" national standard in the approval stage to achieve the combinatorial testing tool based on the existing research results.