Omadam: A Convergent Adaptive Learning Rate Stochastic Gradient Descent Method

In the field of statistics and machine learning,it is usually necessary to solve some stochastic nonconvex optimization problems.The data involved in these problems usually satisfy an unknown distribution.Adaptive learning rate stochastic gradient descent algorithm is a kind of main optimization method to solve these stochastic nonconvex optimization problems,which can adjust learning rate adaptively by gradient information.Adam and RMSprop methods are two popular adaptive learning rate stochastic gradient descent algorithms,which usually reach the optimal value rapidly.In some cases,however,they may not converge to the optimal value.Moreover,many improved algorithms usually require the objective function to be convex,which is difficult to be satisfied in practical training.The convergence analysis of the adaptive learning rate algorithm with non-convex function is usually very complicated.Based on the above problem,this thesis proposes OMadam Algorithm.In the iteration process,it improves the numerical performance by calculating the momentum of the gradient scaled by the second-order momentum,which is called the outer momentum technique.On the other hand,the algorithm adjusts the parameters in each iteration step to ensure the convergence.In the convergence analysis,this thesis assumes that the function is not convex,and gives the theoretical analysis process and the sufficient conditions for the convergence by introducing the approximate equivalent form of the algorithm and making theoretical proof.At the same time,the thesis uses the same proof method to give the theoretical analysis of another algorithm.This provides a new way to analyze the convergence of the adaptive learning rate stochastic gradient descent algorithms.Finally,experiments are carried out on both artificial data sets and real data sets.Experiments on artificial data sets verify the convergence of OMadam algorithm.On the real data set,OMadam method shows effectiveness of outer momentum technique,faster convergence speed,higher test precision and more stable decreasing trend of loss value.