Some Gradient Learning Algorithms For Pi-Sigma Neural Networks

Recently, many neural network models have been proposed. The feedforward neural network is most widely used in applications. In the early years, the feedforward neural networks only use additive neurons which may lead to poor capability for solving nonlinear problems. Hence, product neurons are applied in the feedforward neural networks to enhance the nonlinear mapping capability and to improve the learning efficiency. This kind of feedforward neural network is called high-order neural networks. However, the product neurons are only composed of the product of inputs, and the number of weights required increases combinatorially with the dimension of the inputs. This phenomenon is called dimensional explosion. The pi-sigma network is proposed by Y. Shin in 1991, which uses the product neurons with polynomials of inputs. Then, lots of researchers, such as Y. Shin, A.J. Hussaina, C.K. Li et al, use pi-sigma network as a basic building block for other more complicated high-order neural networks. These networks have been used effectively for solving problems such as classification and approximation.The investigations have been made broadly on convergence and generalization of feedforward neural networks with additive neurons. But the investigations of feedforward neural networks with product neurons are focused only on experiments. There remains a lack of theoretical assurance. Therefore, it is necessary to study the convergence and generalization of high-order neural networks. This will make a great promotion for the theory and application of neural networks.Gradient algorithm is a simple and popular training algorithm for feedforward neural networks. There are two different ways to input the samples during the training process: online mode and batch mode. There are also two different ways for updating the weights: synchronous mode and asynchronous mode. The main work of this thesis is to study several learning gradient algorithms for Pi-Sigma networks in theory, including the de-terministically convergence of gradient algorithms and the efficiency of training. The structural optimization of the feedforward neural networks is also discussed. This thesis is organized as follows:Some background information about the feedforward neural networks is reviewed in Chapter 1. The second chapter points out a problem when the online gradient algorithm is used for pi-sigma networks, in that the update increment of the weights may become very small, especially in the early stage of the training, resulting in a very slow convergence. To overcome this difficulty, an adaptive penalty term is introduced into the error function, so as to increase the magnitude of the update increment of the weights when it is too small. This strategy brings about faster convergence as shown by the numerical experiments.The third chapter mainly deals with the convergence of the batch asynchronous gradient method with or without momentum. Corresponding convergence results are established.The convergence of an online BP algorithm is investigated in Chapter 4. The monotonicity of the error function and the convergence of the method are proved. A supporting numerical example is also given.In Chapter 5, a pruning algorithm based on grey incidence analysis for feedforward neural networks is proposed, which can optimize the topology of the neural networks, including both high-order neural networks and common feedforward neural networks, and can offer better generalization ability. In this algorithm, the redundant correlations are pruned according to the degree of grey incidence of each output sequence. The simulation results show the effectiveness of the proposed approach.