Convergence Results Of Gradient Algorithms For Training Feedforward Neural Networks

In recent years,neural networks have been widely used in many fields.Feedfoward neural network is a simple neural model with wide application and often trained by gradient algorithm. Many techniques have been introduced to improve the performance of the gradient algorithm. For example,the generalization capability can be enhanced by adding a penalty term to the error function,the training process can be accelerated and may escape from the local minimum by adding a momentum to the weight changes,and complex-valued signals can be de processed by introducing the complex-valued neural networks and complex gradient algorithms.The theoretical analysis on the properties(especially the convergence) of these improved gradient algorithms is an important research domain of neural networks.This dissertation investigates some convergence properties of gradient algorithms used to train feedfoward neural networks. Moreover,a method to adaptively determine the momentum factor of the back-propagation(BP) algorithm with momentum is also presented.The main contents of this dissertation are listed as follows.1.A crucial condition for the convergence of the gradient method is the boundedness of the network weights in the learning process.Indeed,most of the convergence results in literatures explicitly or implicitly assume this condition holds.In practice,however,these boundedness conditions may be hard to check,and there is no theory to guarantee such conditions.Despite the fact that it can be replaced by other conditions,the boundedness condition remains important, due to the difficulty to check the new conditions.Adding a penalty term to the error function has become a common practice to make the network weights bounded.But there seems no theoretical proof of the weight boundedness for the gradient method with the penalty.To fill this theoretical gap in literature,the dissertation firstly rigorously prove the weight boundedness of the online gradient algorithm with penalty for training both the feedforward neural networks with sigmoid output and that with linear output(the convergence results for the corresponding algorithm are also obtained with the help of stochastic approximation theory),then rigorously prove the weight boundedness of the batch gradient algorithm with penalty for training feedforward neural networks.2.The convergence results for the batch gradient algorithm with or without momentum for training complex-valued neural networks are established.At the same time,an up-bound of the learning rate for the convergence of the batch gradient algorithm without momentum, and relationship between the learning rate and momentum factor for the convergence of the batch gradient algorithm with momentum,both for training complex-valued neural networks, are given.3.A method is developed to adaptively determine the momentum factor of BP algorithm to enhance the training speed of the neural networks.Taking the learning rate as constant, the algorithm adjusts the momentum factor according to the the gradient of the error function with respect to the weight vector.Numerical experiments show that the proposed algorithm is effective for both the batch and online training.Moreover,it is superior to the BP algorithm with constant momentum factor in respect of convergence rate and stability.