Differential Combination Testing For Deep Learning Systems

Deep learning(DL)systems are increasingly used in safety-related fields,including automatic driving and intelligent security,where the DL model prediction accuracy is very important.As with the traditional software development,confidence in the model behavior results from rigorous testing for every possible scenario.However,the logic of DL systems is learned through a training process.Therefore,DL model is difficult to be tested and the existing DL testing heavily relies on manually labeled data and often fails to expose erroneous behaviors for corner inputs.To solve these problems,this paper proposes Differential Combination Testing(DCT),an automated DL testing approach to systematically detect the erroneous behaviors about more corner cases without relying on manually labeled input data.Firstly,the DCT method based on random image transformations adds constraints to seed images and applies image combination transformations to automatically generate test cases that can increase neuron coverage and differential detection result of multiple similar DL models.The method utilizes multiple DL systems with similar functions as cross-references,so that input data can be labeled automatically and the correctness of output behavior in multiple DL systems can be checked automatically.Secondly,the DCT method based on priority image transformations adds priority strategy in image combination transformations and applies differential strategy of automatic detection results in system model detection.The method achieves a more efficient image processing and more accurate determine the output behavior of DL systems.Finally,extensive experiment results show that the DCT approach effectively finds thousands of erroneous corner behaviors in the most commonly used DL models,which can better detect the reliability and robustness of DL systems.In summary,The DCT approach can automatically annotate data and realize high coverage,and be capable of simultaneously detecting behavioral accuracy of multiple DL models,making a beneficial attempt in data labeling and DL systems testing.