Study On The Residence Error, Stability, And Generalization Capability Of Extreme Learning Machine

Extreme Learning Machine (ELM) is a recently proposed Single-Hidden Layer Feed-forward Neural Network to tackle some challenges. ELM considers the network as a linear system and acquires the optimal parameters by the system’s min-norm min-square solution, which not only has an extremely fast training speed but also solve the local minimum and over-fitting problems to some extent. A lot of recent references show that, due to the fast training, ELM has been applied to many real fields, especially in the recently popular Big Data analysis in which ELM’s good generalization is reported. This thesis is dedicated to study some key fundamental issues including representation of training residence error, robustness of ELM model, rank change of input data matrix, relations between outputted fuzziness and ELM’s generalization, and Divide-and-Conquer strategy of sample training. The research results are summarized in the following four parts.Firstly, we study the rank of data matrix inputted to an ELM and find a relation between the rank and ELM’s training error. The relation, which plays a an irreplaceable role to analyze ELM’s structure, to improve ELM’s generalization, and to optimize the ELM’s approximation error model, is a well acknowledged key issue in ELM theoretical study. An ELM approximation model based on the input data matrix rank analysis is developed. The model indicating the change of output-matrix rank with the increase of input-matrix dimension gives an estimation of training error and an evaluation of robustness.Secondly, we develop a genetic algorithm for L1-norm ELM. In the algorithm, considering the analyzing properties of L1/L2 space and the difference between L1 and L2 solutions, we propose to use L2-solution as the initial population of L1-problem. The algorithm’s advantages are experimentally demonstrated. In comparison with the randomly generated initial population of L1-problem, our proposed genetic algorithm shows an essential improvement regarding the convergence performance.Thirdly, ELM’s generalization ability is investigated from the viewpoint of uncertainty. It is well known that the, for any supervised learning model, the evaluation index of generalization (which means the model’s correct rate of prediction on unseen samples) is the most important. There are many factors having the impact on model’s generalization. These factors include sufficiency of samples, convergence of training algorithms, suitability of selected model, model’s robustness, and etc. Experimentally this part gives a statistical relation between an ELM’ generalization and the outputted uncertainty in a training set. When a well-trained feed-forward neural network is regarded as a stochastic function in which both inputs and outputs are random variables, and the variance of the output random variable can be considered as a type of stability of the neural network. Unfortunately for almost feed-forward neural networks the distribution function of outputs cannot be analytically derived even if the distribution function of inputs is very simple. Focusing on three feed-forward neural networks, this paper presents an experimental study on this type of stability through Mont Carlo simulations. A sorting of stability on the three neural networks is given, which provides some useful guidelines for users who select a feed-forward neural network as their model.Fourthly, we investigate a relation between the fuzziness of a group of samples outputted by a classifier and the misclassified rate of this group of samples. For a given trained classifier that outputs a membership vector, we demonstrate experimentally that samples with higher fuzziness outputted by the classifier mean a bigger risk of misclassification. We then propose a fuzziness category based divide-and-conquer strategy which separates the high-fuzziness samples from the low and middle fuzziness samples. A particular technique is used to handle the high-fuzziness samples for promoting the classifier performance. The reasonability of the approach is theoretically explained and its effectiveness is experimentally demonstrated.