Convergence Of Online Gradient Algorithm With Stochastic Inputs For Pi-Sigma Neural Networks

BP neural network (BPNN) is one of feedforward neural networks and widely used inmany fields. But its most drawback is the poor convergence. High order neural networks ownsthe greater nonlinear capacity than BP neural networks, and can improve training efficient forthe same problem. Pi-Sigma Neural Network (PSNN) proposed by Y. Shin and J. Ghosh in1991 is a kind of new high order neural networks. The structure of PSNN is relatively simplesingle layer structure, with fast convergence character. Moreover, it also the powerful nonlinearmapping capacity owned by higher order neural networks. Due to its specific structure, thewell-known Dimension Disaster, which traps many other higher order neural networks, can beeffectively avoided by PSNN. So the networks constructed in terms of PSNN are widely usedto implement the classification and approximation tasks.The theoretic analysis of the convergence of neural networks is almost in the view ofprobability. Recently, many results about the determinacy convergence of BPNN have been gotby Prof. Wu et. al. However, the research of the determinacy convergence in the field ofhigh order neural networks (especially PSNN) is at the beginning. In the literatures [12] and [13]are the convergence results about training PSNN with Batch BP and online gradient method ina fixed order, respectively. In this paper, the results above are extended with the convergence oftraining PSNN with online gradient method in a special stochastic order proposed. Convergenceresults and the monotonicity of the error function after each batch are proved. Additionally, wefind the effect to the convergence made by the adoption to learning rate and initial weights. Thescheme is proposed to solve the problem by analysis to the phenomenon.