In the field of machine learning,gradient descent algorithm is the most significant and fundamental method to solve optimization problems.With the continuous expansion of the scale of data,the traditional gradient descent algorithms can not effectively solve the problems of large-scale machine learning.Stochastic gradient descent algorithm selects one or several sample gradients randomly to represent the overall gradients in the iteration process,so as to reduce the computational complexity.In recent years,stochastic gradient descent algorithm has become the research focus of machine learning,especially deep learning.With the constant exploration of search directions and step sizes,numerous improved versions of the stochastic gradient descent algorithm have emerged.The improved strategies of stochastic gradient descent algorithm are roughly divided into four categories,including momentum,variance reduction,incremental gradient and adaptive learning rate.The first three categories mainly correct gradient or search direction and the fourth designs adaptively step sizes for different components of parameter variables.For the stochastic gradient descent algorithms under different strategies,the core ideas and principles are analyzed emphatically,and the difference and connection among different algorithms are investigated.Several main stochastic gradient descent algorithms are applied to machine learning tasks such as logistic regression and deep convolutional neural networks,and the actual performance of these algorithms is numerically compared.Aiming at two main shortcomings of variance reduction strategy: "gradient rigidity" and "ignoring historical gradient information",an improved strategy,named momentum aggregation correction,is proposed.This strategy can limit the gradient cumulative variance to a certain range and avoid falling into the local optimal solutiondue to insufficient gradient update.Using the momentum aggregation correction strategy to improve the stochastic variance reduction gradient descent algorithm,a stochastic gradient descent algorithm MAC-SVRG based on momentum aggregation correction is proposed.Nonnegative matrix factorization is an important application scenario of stochastic gradient descent algorithm.A nonnegative matrix factorization algorithm MAC-SVRMU based on momentum aggregation correction is proposed.Based on the random multiplication updating algorithm and the momentum aggregation correction strategy,this algorithm has better performance in both artificial and real data sets,and can make the objective parameters approach to the optimal solution quickly and effectively.At the end of the thesis,the main research work is summarized,and the future development direction of the stochastic gradient descent algorithms is prospected. |