Research On Some Problems In Neural Networks

Stemming from the pioneer work of professor J. J. Hopfield, the research on artificial neural networks has been being a hot topic since its rebirth in the early 1980s. Up to now, a lot of achievements have been obtained and the theories of neural networks have been applied in many fields, including finance, military, engineering, medicine, insurance, entertainment, etc. The artificial neural networks imitate the human intelligent behaviors and have powerful parallel computational capability. Many distinguished scientists in some top institutes or universities, such as MIT, Harvard University and Boston University, have research interests on artificial neural networks. A lot of papers on neural networks have been published in top-ranking journals in the world, such as Science, Nature, IEEE Trans. Neural Networks. Clearly, the research of neural networks is quite important both in theory and applications.This dissertation is divided into three parts: 1) the learning algorithms for principal component analysis (PCA) neural networks; 2) support vector machine (SVM); 3) a class of recurrent neural networks.The first part focuses on the learning algorithms of PCA neural networks. PCA is basically a statistical learning method, which is widely used for signal processing, pattern recognition, digital image processing, etc. The classical methods for computing the principal components, such as QR or SVD, encounter some difficulties when computing the principal components for online data or huge data sets. PCA neural networks can overcome the problems. Especially, the networks are suitable for online computing the principal components. Thus, PCA neural networks can be widely used in many applications.PCA learning algorithms are important parts of PCA neural networks. Up to now, many PCA learning algorithms have been proposed. Among the algorithms, Oja's algorithm, Xu's LMSER algorithm and GHA are commonly used. Several other algorithms for PCA are related to these basic procedures. For these algorithms, the convergence is crucial in the practical applications. In the past years, the deterministic continuous time (DCT) method is an important tool for analyzing the convergence of PCA learning algorithms. Recently, it is reported that the DCT is not practical because of the constraint conditions and a more suitable method, the deterministic discrete time (DDT) method, was proposed. The DDT method is more effective for the convergence analysis of PCA algorithms, but more difficult. The part intensively investigates the convergence of the learning algorithms by using the DDT method. The contributions are:1. The convergence of Xu's LMSER PCA learning algorithm with a constant learning rate is studied firstly. An invariant set and an ultimate bound are obtained and local convergence are proven rigorously.2. A nonzero-approaching adaptive learning rate is proposed to guarantee the global convergence of Oja's PCA learning algorithm. Rigorous mathematical proofs for the global convergence are given.3. Nonzero-approaching adaptive learning rates are used to overcome the problems faced by GHA when using zero-approaching learning rate. Rigorous mathematical proofs are given to prove the global convergence of GHA with the nonzero-approaching adaptive learning rates.4. In practical applications, especially for online computation, the number of principal component directions must be determined before GHA is used. It is not practical. To overcome the problem, a single layer neural network model with lateral connections is proposed together with an improved generalized Hebbian algorithm (GHA) to adaptively approximate to the intrinsic dimensionality of a given data set.5. Stability and chaotic behaviors of a PCA learning algorithm and a MCA learning algorithm are analyzed. The conditions for chaos existence are obtained.6. An application of Oja PCA neural network to register the medical images is discussed.In the second part, SVMs are studied. SVMs have been widely used in applications because of their good generalization performance. However, when SVMs are used for handling huge data sets, SVMs suffer from a slow convergence rate because of the computational complexity. Two methods in this part are proposed to overcome the problem. The first is the improved SMO decomposition. The second is a reduced SVM method. In this method, a candidate subset of support vectors is selected by using pulse coupled neural networks. The size of the candidate set of support Vectors selected this way is smaller than that of the original training samples so that the computational cost in learning process for SVMs can be reduced and the convergence is accelerated. More interestedly, the candidate set of support vectors includes almost all support vectors, the performance of the SVMs based on this candidate set matches the performance level when the full training samples are used. In this part, a new fuzzy membership function for Fuzzy SVM is also proposed to reduce the effects of outliers when SVMs solving the classification problem.The third part focuses on the convergence of a class of recurrent neural networks (RNNs). Recurrent neural networks are potentially powerful because of the feedback connections and temporal behaviors. Many applications of RNN have been obtained, such as associative memory, image processing, pattern recognition. It is essential and necessary to understand these dynamical properties of RNNs before the RNNs are used in practical applications. This part investigates the global output convergence for a class of delayed recurrent neural networks with time varying inputs. Some sufficient conditions to guarantee output convergence of the networks are derived.