A Study On Some Algorithms For Improving The Generalization Capability Of Neural Networks

Abstract:As a powerful tool for pattern recognition, artificial neural networks have strong nonlinear approximation capability, but it can not guarantee well generalized performances. In this dissertation, we study the neural networks from both the network models and the learning algorithms, aiming at improving their generalization capabilities. The main contents are as follows.1. Learning two-dimensional feedforward neural networks (2D-FNNs) using gradient descent. Since the input to Traditional FNNs are usually vectors, and many data are usually presented in the form of matrices, the matrices have to be decomposed into vectors before FNNs are employed. A drawback to this approach is that important information regarding corre-lations of elements within the original matrices are lost. Unlike traditional vector input based FNNs, a new algorithm of extended FNN with matrix inputs, called two-dimensional back-propagation (2D-BP), is proposed in this paper to classify matrix data directly, which utilizes the technique of incremental gradient descent to fully train the extended FNNs. These kinds of FNNs help maintain the matrix structure of the 2D input features, which helps with image recognition. Promising experimental results of handwrit-ten digits and face-image classification are provided to demonstrate the effectiveness of the proposed method.2. Negative correlation learning (NCL) based on two-dimensional neu-ral network with random weights (2D-NNRW).2D-NNRW is a 2D-FNN with single hidden layer, it randomly assigns the left and right projecting weights and hidden biases, and only tunes the linear weights. As an ex-tension of NNRW, it simplifies the training of 2D-FNN, but it also need sufficient hidden units for guarantee, it may also lead to over-fitting. It is a weak classifier when the network is compact, but ensemble learning can cover its shortages. As one of the most efficient ensemble approach, NCL produces ensembles with sound generalization capability through control-ling the disagreement among base learners. So we combine NCL with 2D-NNRW, and come up with decorrelated 2D neural-net with random weights algorithm (DNNE 2D-NNRW). Further research shows that the proposed method can improve the generalization performance more efficiently than other methods for face recognition.3. Negative correlation learning (NCL) base on random local linear models (RLLMs). As an efficient ensemble learning framework, not only classical FNNs can be used for implementing such a learning scheme. Here, we can also adopts a class of FNNs called RLLMs as base components, and integrates with the NCL tactics for building new ensembles, named as DNNE RLLM. The main merits of the approach are that the basis functions of the base models are generated randomly and the analytical solution of free parameters can be determined by solving a linear equation system. This method is particularly designed for dynamic time series prediction tasks. And the experimental results show that our method is effective and competitive compared with some other methods.