Research On Testing Methods Of LSTM Model Based On Coverag Criteria

In recent years,the application of deep learning systems has become more and more extensive,the speech emotion recognition systems based on Recurrent Neural Networks(RNN)and its variants such as Long Short-Term Memory Neural Networks(LSTM)have also developed rapidly.It has performed well in safety-critical fields such as autonomous driving and medical diagnosis,therefore the attendant security issues cannot be ignored.The current testing research on deep learning systems mainly focuses on image recognition models such as Deep Neural Networks(DNNs).Scholars have also proposed many coverage criteria for DNN testing.However,the complex structures of RNNs cannot directly apply these methods.At present,there are few researches on the coverage criteria of LSTM models,and existing criteria has problems like coarse granularity and ignoring the role of other basic components except the hidden state.Therefore,the LSTM-based test coverage method still faces great challenges,requiring new test coverage criteria and new test case generation methods,aiming to measure test adequacy,generate richer test cases,and discover more wrong behaviors of the model,thereby improving the robustness of the model.In this paper,two coverage criteria and test case generation methods guided by coverage criteria are proposed for the unfolding structure and internal state transition of LSTM.Subsequently,an LSTM-based white-box testing framework was implemented,and the criteria mentioned above are experimentally evaluated.The main research contents of the paper are as follows:(1)Important-unit coverage(IUC)is proposed for the LSTM expansion structure,and the test case generation method based on IUC is given.This paper first proposes the concept of important hidden units,that is,the hidden units with the largest output value in the hidden state of the expanded LSTM model,which usually play an important role in decision-making.On this basis,this paper designs the important-unit coverage,and gives the corresponding test case generation method under the guidance of the criteria.(2)An improved state/transition coverage is proposed based on LSTM internal states and transitions,and a test case generation method is designed according to this criteria.This paper first spatially partitions and abstracts the states of multiple components of LSTMs(including hidden states,cell states,and forget gate states),then models them as discrete-time Markov chains(DTMC)to capture their statistical behavior.We proposes an improved state/transition coverage criteria that also takes into account the cell states and the role of gating components in the LSTM model baseed on basic state/transition coverage of Deepstellar.(3)Design and implement a white-box testing framework based on LSTM,and evaluate five criteria proposed above from four perspectives :consistency of changing trend,sensitivity,effectiveness and robustness improvement.Experiments show that,in terms of consistency,all the coverage criteria proposed in this paper show the same changing trend as the existing coverage criteria;in terms of sensitivity,IUC,transition coverage with parameters of(3,10),and state coverage with parameters of(6,10)performed well;in terms of effectiveness,all proposed coverage criteria can effectively perceive the abnormal cases;and in terms of robustness improvement,the test case generation methods guided by the proposed coverage criteria can effectively improve the classification accuracy of the model,and the generated cases guided by forget gate state coverage are the best in the ability to improve robustness.