A Study On The Generalization Ability Of Random Weight Network

Random weight network is a kind of neuron network that generates part of weights randomly in the training process of artificial neural network.It simplifies network training process by virtue of the reduction of unnecessary iteration.The most representative training algorithm of random weight network is Extreme Learning Machine(ELM).ELM is a kind of single hidden feed-forward neural networks using random generating weights mechanism,which have advantages like fast training speed,high generalization ability and easy realization.Because of those advantages,ELMs not only attract much attention from academic area,but also are widely applied to solve real life problems.In this paper,we do the research on the generalization ability of random weight network by means of the research on the generalization ability of ELMs.The main content of this paper consist of two parts:First one is the study on the relationship between input matrix rank and generalization ability of ELMs.Earlier studies show that output matrix,generated by activation function transformation in hidden layer,will be a full-rank matrix with probability.After having a deep study on training process of ELMs,we design some experiments to verify the relationship above.We generated needed data-sets according to certain rank change tendency,and then use those data-sets with different ranks do the regression of Benchmark functions on ELMs.Experimental results show that with the rank of data-sets increasing to be full,both training RMSE and Testing RMSE decrease from a certain point.Second one is a study on ELMs generalization ability from the aspect of supervised classification problem complexity.When we solve a classification problem using classifier,the accuracy of classification not only depends on the generalization ability of classifier but also on the complexity of the classification problem.Firstly,we do a research on classification problem complexity,trying to find out digital indexes that can measure classification problem complexity effectively.There are 12 indexes that have been summarized and we design experiments to verify the effectiveness of those 12 indexes.Experimental results show that there are two indexes have good effectiveness on measuring classification problem complexity and there is strong correlation between two indexes.We choose one of them to measure supervised classification problem complexity and then observe the performance of ELMs on classification data-sets with different complexity and do comparison with classifier SVM and C4.5.Experimental results show that ELMs have better performance on almost all data-sets and have lower sensitivity on classification problem complexity in the meantime.All of those results are direct-viewing manifestation of high generalization ability of ELMs.We find that ELMs have better generalization ability than classifier SVM and C4.5 from aspect of supervised classification problem complexity.