Research On Three-step Accelerated Gradient Algorithm In Deep Learning

Gradient descent(GD)algorithm is the most widely used optimization method in training machine learning and deep learning models.More and more acceleration meth-ods were proposed to solve the slow convergence of GD,for example the momemtum algorithm.In this dissertation,based on GD,Polyak's momentum(PM)and Nesterov accelerated gradient(NAG),we gave the convergence of the algorithms from an initial value to the optimal value of an objective function in simple quadratic form.But their iteration steps of convergence were different.Based on the convergence property of the quadratic function,two sister sequences of NAG's iteration and parallel tangent methods in neural networks,the three-step accelerated gradient(TAG)algorithm was proposed,which has three sequences other than two sister sequences.The experiment results of quadratic function showed that the TAG algorithm had fewer iteration steps of conver-gence than GD,PM and NAG algorithms.Then we considered the extension of the objective function to high-dimensional quadratic functions to show the TAG algorithm was superior to other three algorithms.We also considered the extension of the objective function to nonquadratic function,which was FLETCHCR function came from the CUTE collection.The results showed that the TAG algorithm had fastest convergence,and had longer range of momentum parameter,which means the TAG algorithm was more robust than other two accelerated algorithmsThen we considered to combine the TAG algorithm to the backpropagation algorithm and the stochastic gradient descent algorithm in deep learning.We rewrote the R package neuralnet,named supneuralnet.All kinds of deep learning algorithms in this dissertation were included in supneuralnet package.Finally,we showed our algorithms were superior to other algorithms in four case studies.