Optimization Research Of Neural Network Fuzz Testing Technique Based On Heuristic Search

Deep learning systems are rapidly being applied in safety critical fields,such as autonomous driving and medical image recognition,and thus the requirements for safety and reliability of deep neural networks have become increasingly strict.However,due to the black-box nature and uninterpretability of deep neural networks,how to ensure the safety and robustness of deep learning systems has become a difficult problem for academic circles and industry.Neural network testing techniques have emerged as a complement to verification techniques,the main idea is to test the system through a large number of test cases to find defects to provide assurance.Traditional software testing methods are not applicable to deep neural networks,so testing standards and testing techniques that specific to deep learning systems are needed.Inspired by the idea of traditional software testing,many coverage criteria and testing methods based on neuronal activation values have been proposed in recent years.These test methods include coverage based test input generation methods and adversarial example based generation methods,the purpose of these testing methods is to maximize coverage,find the best testing cases,detect model defects and find misbehavior inside neurons.This thesis propose a fuzz testing technique using heuristic search based on the coverage criteria of neural networks that proposed in recent years,and experimentally investigate the increase of coverage by test case generation and the detection ability to detect adversarial examples.The main works in this thesis are:(1)A coverage guided fuzz testing technique for deep learning systems is designed to improve the coverage of deep neural networks using heuristic search and find the best test cases.Experiments show that the coverage of neural networks can be improved under different granularity using heuristic search-guided fuzz testing technique,and the improvement effect of fuzz testing technique using clustered batch selector is more obvious and the computation time cost is less;(2)Based on adversarial example technique,the correlation between our testing technique and adversarial examples is explored,and the experimental results show that our fuzz testing technique can produce a certain amount of adeversarial examples;(3)Investigate the correlation of the test technique to improve the robustness of neural network model.Firstly,evaluate the naturalness of the generated cases to measure the practicality of the test technique.Adding the test cases generated by the fuzz testing to the dataset to study the improvement for the model accuracy.Finally,evaluate the improvement of the system's defect detection capability using Pearson's correlation coefficient and Spearman's correlation coefficient.It was found that the naturalness of the test-generated samples was good enough to meet the practical research significance.In addition,the model accuracy was not improved after adding the test input samples to the dataset for retraining,but the defect detection ability was improved to some extent.