Training Neural Network With Second-Order Algorithm

Since Hinton et al.proposed the BP back-propagation theory,the stochastic gra-dient descent method has become a common method for solving neural networks such as fully-connected neural networks,long-term and short-term memory neural networks,and convolutional neural networks.Although the Q-liner's stochastic gradient descent method can solve the ideal parameter value,the algorithm often needs more iterations to obtain the optimal parameter value.In order for the neural network's solution algo-rithm to converge within a small number of iterations,a faster convergence algorithm need to be adopted.In this paper,we first introduce the basic structure of a fully connected Neural Net-work and the approximation theory of neural networks.This provides a solid theoretical foundation for the following study in Neural Networks.In Chapter 2,we introduce the steepest gradient descent algorithm,Newton al-gorithm,conjugate gradient and SESOP algorithm,and compare their convergence speeds.In Chapter 3,similar to the local gradient in BP algorithm,the concept of second-order local Hessian matrix and second-order local partial derivatives are proposed for the specific structure of full-connected neural networks,and its back propagation formula is given.Finally,we propose the Damped-Gauss-Newton algorithm and the SESOP algorithm to solve parameters in Fully Connected Neural Network.In Chapter 4,we use the open-source MNIST dataset to verify that our two al-gorithms proposed in this paper are optimal to the BP algorithm in the convergence speed.In Chapter 5,we discuss the research prospects of two proposed algorithms.