Research And Improvement Of Optimization Algorithms In Deep Learning

Deep learning is a recently proposed machine learning model,while comparing with the traditional shallow machine learning model,deep learning contains two or more hidden layers.In fact,deep learning is a multi-level network structure.The network structure can be represented by a complex composite function.The unknown arguments of this composite function are the weight parameters and bias terms of each layer in deep learning model.The values of these parameters directly determine the network model is good or bad.In order to improve the accuracy of the deep learning model output,we need to constantly optimize the parameters in the model.Therefore,the research of optimization algorithms in deep learning has become a hot topic.In this paper,five typical optimization algorithms in deep learning are improved,including Adam,RMSProp,AdaGrad,momentum and gradient methods.In order to improve the convergence speed of Adam and AdaGrad methods,we introduce the momentum idea to Adam and AdaGrad methods,and propose two new methods.The new method based on Adam is called AMM mehhod.The other new method based on AdaGrad is called improved AdaGrad method.Momentum method has initialization error,in order to correct the error,we propose an improved momentum method.In order to avoid setting the initial learning rate of the RMSProp algorithm,we introduce two typical BB steps into the RMSProp algorithm,and propose two new RMSProp algorithms with two kinds of BB steps.In order to overcome the shortcoming of gradient method slowing down near the minimum point,we introduce a typical pattern step into the gradient method and propose a gradient method with pattern step.We respectively show the effect of the five improved algorithms are better than the original ones by conducting numerical experiments.The improvement effect of Adam algorithm is most obvious compared with other four methods.We not only give the numerical experiment of AMM algorithm,but we also show the convergence of AMM algorithm.There is no convergence analysis about the other four algorithms so far.In addition,the gradient method with pattern step has not been applied to deep learning model.In the future,these work will be done.