Design And Implementation Of Whitebox Testing Framework For Deep Learning System Based On Adversarial Examples

With the great improvement of computing power of computer hardware and the continuous emergence of large-scale data sets,deep learning technology is constantly stepping into a new stage.The development of deep learning system not only brings opportunities,but also brings new challenges to its testing technology.Compared with traditional software,deep learning system has great differences in internal structure and external performance,so it is impossible to directly apply the white-box testing method in traditional software testing technology to deep learning system.At present,several white box test coverage criteria for deep learning systems have been proposed,but the effectiveness of the criteria in practical systems has yet to be tested.This paper mainly studies the white box test coverage criteria of deep learning system.Firstly,a group of measurement indicators of white box test coverage criteria are proposed to complete a comparative study.Secondly,based on the coverage criteria,a white box testing framework for deep learning systems is designed and implemented.Specific research contents are as follows:(1)The measurement indicators of white box test coverage criteria,including Efficiency,Consistency and Correlation,are proposed.Based on the Le Net model(Le Net1,Le Net4,and Le Net5),MNIST dataset and adversarial examples,the system compares each test coverage criterion.Through multiple experiments,the coverage and accuracy data of each coverage criterion are obtained,and the coverage criterion is classified according to its performance in the indicators.According to the experimental results,the following conclusions can be drawn in this paper.When testing the system with high robustness requirements,the coverage criterion with fine particle size can be selected,Including Neuron Bondary Coverage(NBC),Strong Neuron Activation Coverage(SNAC),Likelihood Based Surprise Coverage(LSA),For systems with general robustness requirements,K-multisection Neuron Coverage,Top-K Neuron Coverage and Importance?Driven Coverage can be selected for testing.For systems with low robustness requirements,Neuron Coverage ? Sign-Sign Coverage ? Distance-based Surprise Adequacy can be selected for testing.(2)A white box testing framework for deep learning systems is designed and implemented and verified on a classification system.The white box testing framework realizes the function of coverage calculation for the pre-trained deep neural network model and accuracy evaluation on the adversarial examples,and the model robustness can be analyzed by coverage results.Then,the framework is applied and verified in the system.In the verification process,NBC,SNAC and LSA are selected for coverage calculation according to the classification results of coverage criteria and the requirements of system robustness.The results show that the realized white box testing framework of deep learning system can be applied to the coverage calculation of classification model,and then the robustness of the system can be evaluated.