Learning Algorithm For Neural Network Based On Analytic Optimization Methods

Neural networks have been widely applied in life science, engineer science and many other fields. With the research of learning algorithms deepening in recent years, many good results have been achieved in intelligent control, pattern recognition, optimization computation, machine vision and biomedical. However, classical learning algorithms have some drawbacks as slow convergence speed and easily falling into local minimum. These methods usually are not applicable for large scale problems. So this thesis will discuss analytic optimization method based neural network learning algorithms.Firstly, we extended a linear-least-square based fast learning algorithm to multilayer neural networks by constructing the approximate value of the hidden layer. Numerical experiments showed that the new algorithm is suitable for large-scale learning samples. The new algorithm can reach a good MSE value in the first few epochs and converges faster than other algorithms. Secondly, we got a new self-scaling and a new collinear-scaling BFGS algorithm by constructing a new self-scaling and collinear gene. We discuss some properties of the given algorithms and their global convergence. The numerical experiments showed that the scaling method based network learning algorithms have better performance and stability. A Matlab algorithm toolbox was given in the end.