Convergence Of Two Class Of RMSProp Algorithms Under Nonconvex Stochastic Setting

In recent years,thanks to the rapid development of computer programming language,graphics parallel processing and multithreading mechanism,the effectiveness of deep learning technology has been demonstrated by a large number of experiments.Deep learning is widely used in our life,such as speech recognition,computer vision and natural language processing.For a certain deep learning task,whether the optimization algorithm is effective or not is the decisive factor of the deep learning result.At present,adaptive gradient algorithm is the most popular optimization algorithm,among which adagrad,rmsprop and Adam are the most representative.In practical experiments,when the network becomes complex,the objective function cannot keep convexity.Therefore,it is of great value and significance to analyze the convergence of the adaptive gradient algorithm in the case of non convexity.An equivalent algorithm of RMSProp is given and its convergence is analyzed.In order to ensure its convergence,we introduce a sufficient condition which is easy to check,and this condition only depends on the linear combination of the parameters of the basic learning rate and the historical second-order moments.Next,the RMSProp algorithm is extended to a more general case,and a more general RMSProp algorithm is obtained,which is still an adaptive gradient algorithm.Then,the convergence of this algorithm is analyzed,and a sufficient condition for the convergence of more general algorithms is given by analogy.Finally,through numerical experiments,the accuracy and loss of general RMSProp and RMSProp under the same data set and network structure are compared,which shows that they have similar convergence properties.