Research On Improved Stochastic Stratified Average Gradient Algorithm In Convolutional Neural Network

Designing more effective algorithms to train neural network models is a common problem of multidisciplinary interest.Stochastic Stratified Average Gradient(SSAG)is a new method for training neural network models proposed in recent years.On the basis of further study of SSAG,this thesis proposes Stochastic Stratified Average Gradient with Momentum(SSAGM)that combines SSAG with momentum,and the new algorithm can converge faster and perform better than the original algorithm.In addition,Batch Normalization(BN)and stochastic stratified sampling method are combined to propose Stochastic Stratified Batch Normalization(SSBN),which can make SSAG and SSAGM more stable and efficient to train a deep neural network.Specifically,the work of this thesis mainly includes the following aspects:Firstly,on the LeNet-5 model,the performance of SSAG and Mini-Batch Stochastic Gradient Descent,Stochastic Gradient Descent with Momentum,Stochastic Average Gradient Descent on the three benchmark datasets MNIST,CIFAR-10 and STL-10 are compared,and it is found that the performance of SSAG is not weaker than other algorithms on datasets with small differences between classes(such as MNIST),and its performance is slightly better than that of other algorithms on datasets with large differences between classes(such as CIFAR-10 and STL-10).Secondly,the new algorithm SSAGM is proposed by combining SSAG with momentum,which not only solves the problem of unstable training of SSAG,but also further accelerates the convergence speed of the algorithm.The experimental results show that the F1-Score trained on the MNIST,CIFAR-10 and STL-10 datasets using SSAGM in the LeNet-5 model are 99.22%,67.40% and 49.21%,respectively,which are better than other gradient descent algorithms.Thirdly,aiming at the problem that SSAG and SSAGM cannot be trained for deep convolutional neural networks(DCNN),a DCNN training strategy SSBN that is compatible with SSAG and SSAGM is proposed by combining BN with stochastic stratified sampling method.It can enable SSAG and SSAGM to be trained on the VGG-19 model of DCNN,and the performance of the VGG-19 model is improved by about 10% compared with the LeNet-5 model,so that these two algorithms can be applied in more fields.